Xray Test plan execution status

Hello,
The issue type “Test Plan” is presenting the latest tests execution status and also make some calculations, it looks like that:


i would like to copy that kind of status and place it it eazyBI chart
i know that is possible to present Test Execution issues status, but i want the Test Plan issues status

Thanks, Adi

I’ve tried that command, but it didn’t work:
[Xray Test Plan].CurrentHierarchyMember.Get(‘Overall Execution Status’)

Hi @ValorHeart,

If you want to see the summary of all test executions for the Test plan, you can use the default measure “Xray Tests execution summary” in combination with Xray Test Plan dimension.

In case you want to see only the status of the current test execution, please copy the formula of the “Xray Tests execution summary” and alter it to create a new calculated measure, that has filter conditions only for the measure [Measures].[Xray Tests executing].

Kind regards,
Robert // eazyBI staff

Thanks @roberts.cacus for the answer,
Somewhy, the “Xray Test Plan” dimension shows no data in my eazyBI, even when i remove all filters and select “All test plans” in the dimension, Any idea?

Thanks, Adi

Hi @ValorHeart,

I recently encountered something quite similar. Although in my case no Xray dimension displayed any data with the Xray measures. I resolved this with selecting and deselecting the Xray import in the eazyBI import settings and performed a full import. Please try it and report back.

Kind regards,
Robert // eazyBI staff

Hi,

The “Overall Execution Status” in Jira has added value over the “Xray Tests execution summary”.
Suppose I have a test plan with 14 tests. In a first test execution, I execute the 14 tests and get 3 failed. I do some fixes, then re execute the 3 previous failed tests in another test execution. They become pass.

  • in the Overall Execution Status, it says 14 tests pass
  • in the Xray Test execution summary, it says 14 tests pass and 3 failed.

Only the Overall Execution Status gives the balance sheet of the test plan execution.
How to get the same result with EaszyBI ?

Regards

Hi,

eazyBI does not replicate Xray functionality as those are two different tools. But you may build custom reports and calculations similar to reports and calculations in Xray.

Calculated measure Xray Test execution summary would match overview in the Xray for each Test Execution. When measure Xray Test execution summary is used with a Test Plan, then it returns all execution results within the test plan. If the test is executed twice (has two Test Executions) within a test plan, then it will appear twice in summary.

To get tests only by the last execution status in the selected Test Plan, you may create a new calculated measure in Measures. The calculation would go through all tests within a test plan and for each test get the last execution status in that test plan. The formula might look like this:

CASE WHEN
  [Xray Test Execution Status].CurrentMember.Level.Name = "Status"
THEN --count tests by last execution status
  Cache(NonZero(Count(
    Filter(
      --iterate throug all Tests
      Descendants([Xray Test].CurrentMember, [Xray Test].[Test]),
      [Measures].[Xray Tests with executions] > 0 AND
      Order(
        --iterate through execution statuses for a selected Test
        Filter([Xray Test Execution Status].[Status].Members,
          [Measures].[Xray Tests with executions] > 0),
        --order statuses by execution date descending
        DateParse(
          Generate(
            Tail(
              --iterate through all dates when a Test was executed
              Filter([Time].[Day].Members,
                [Measures].[Xray Tests with executions] > 0
              )).Item(0),
            Format([Time].CurrentHierarchyMember.StartDate, 'yyyy-mm-dd'),
            ",")),
        BDESC
        --name of the last execution matches status in columns
      ).item(0) IS [Xray Test Execution Status].CurrentMember)
  )))  
ELSE --total count of tests with executions
  ([Measures].[Xray Tests with executions],
  [Xray Test Execution Status].DefaultMember)
END

This calculation is quite resourceful. I recommend selecting option Nonempty and running a report for a single selected Test Plan (see picture below).

Best,
Zane / support@eazyBI.com

2 Likes

Hello, I used this formula and it looks ok in most cases, but there are cases where a test is failed and also exists as To Do in a new test execution. In that case it is shown as failed in the “overall execution status” in the issue navigator, but it is shown as To Do in the above EazyBI formula.
To fix that I’ve added the following condition (marked as Bold Italics), and I’ve added a new calculated member for To Do:

[Measures].[Xray Test Count by Last Execution Status in Test Plan] is defined as:

CASE WHEN
[Xray Test Execution Status].CurrentMember.Level.Name = “Status”
THEN --count tests by last execution status
Cache(NonZero(Count(
Filter(
–iterate throug all Tests
Descendants([Xray Test].CurrentMember, [Xray Test].[Test]),
[Measures].[Xray Tests with executions] > 0 AND
Order(
–iterate through execution statuses for a selected Test
Filter([Xray Test Execution Status].[Status].Members,
[Measures].[Xray Tests with executions] > 0 and [Xray Test Execution Status].CurrentMember.name <> ‘TODO’),
–order statuses by execution date descending
DateParse(
Generate(
Tail(
–iterate through all dates when a Test was executed
Filter([Time].[Day].Members,
[Measures].[Xray Tests with executions] > 0
)).Item(0),
Format([Time].CurrentHierarchyMember.StartDate, ‘yyyy-mm-dd’),
“,”)),
BDESC
–name of the last execution matches status in columns
).item(0) IS [Xray Test Execution Status].CurrentMember)
)))
ELSE --total count of tests with executions
([Measures].[Xray Tests with executions],
[Xray Test Execution Status].DefaultMember)
END

[Measures].[Execution Status Pass] =
(
[Measures].[Xray Test Count by Last Execution Status in Test Plan],
[Xray Test Execution Status].[PASS]
)

[Measures].[Execution Status Fail] =
(
[Measures].[Xray Test Count by Last Execution Status in Test Plan],
[Xray Test Execution Status].[FAIL]
)

To Do is defined as follows:
[Measures].[Xray Tests created] -
[Measures].[Execution Status Pass] -
[Measures].[Execution Status Fail]

1 Like

Hi @zane.baranovska!

This is a great example, however, I don’t understand the term

and how it affects the ordering.

Can you explain, please!

Thx,
Johannes

Hi @zane.baranovska!

How to model this specific Xray configuration:

I.e., to model the precedence of final statuses over non-final statuses?
In our configuration PASS, FAIL, ABORTED and CONDITIONAL_PASS are final, while TODO, EXECUTING, BLOCKED, N/A are non-final.

I think it would be the general answer to the extension from @ran.lavi above.

Best regards,
Johannes

Hi Johannes,

The first question about the order
This statement helps to find tests with the latest status, which is the same status as selected in the rows.

The question about the status configuration
Currently, XRay status configuration (order, final/not final) is not taken into account in calculations.
Tests are counted to “XRay Tests executed”, “XRay Tests executing” and “XRay Tests scheduled” by the existence of dates “Started on” and “Finished on” (https://docs.eazybi.com/display/EAZYBIJIRA/Xray+Test+Management#XrayTestManagement-XrayMeasures).

Ilze, support@eazybi.com

Please see also the related topic on how to count Xray tests by the last execution status: