Automation V3: Provar Manager - Test Execution Reporting
Test Execution Reporting
When configuring a new Test Plan, the user must select an option to report test execution results to the Provar Manager.
Prerequisites
- A Connection to the Provar Manager must already be created.
- Configure the Test Plan and provide an option to report test execution results to the Provar Manager.
How to configure Test Plans for reporting in Provar Manager
You can configure Test Plans for reporting in Provar Manager in two cases, as explained below.
Create a new Test Plan. For more information on creating a Test Plan, please see Test Plan or click on the Test Plan if you already have a Test Plan created.
Case 1: When you are creating a new Test Plan.
When creating a new Test Plan, configure it to report to the Provar Manager.
In the New Provar Test Plan screen, select the Provar Manager (PM) check box in the Reporting section.
Note: The Connection Name (PM) is the name given to the Provar Manager connection by the user.
Case 2: If you already have a Test Plan created.
Click on Test Plan from the Plans view, then select the Provar Manager tab. Select the Enable Provar Manager reporting option.
Once this setting is enabled, the next time you run the Test Plan it will report to Provar Manager once fully completed.
Note: The Test Plan must first be uploaded to Provar Manager, and it must be run from the Plans view in order to report to Provar Manager.
Here is a current breakdown of the reporting currently supported by Provar:
- Test Execution Status – Provar Manager’s Test Execution object will be updated to reflect the status of each test case.
- Test Plan Folder Execution – On selecting a Test Plan Folder for execution, the Provar Test Execution will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the selected Test Plan Folders.
- Test Plan Instance Execution – On selecting a Test Plan Instance(s) for execution, the Provar Test Execution engine will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the select Test Plan Instances.
Run Mode
You can select any Test Plan, Test Plan Folder, or Test Plan Instance and select Run to execute it and report the results to Provar Manager.
You can also right-click and Test Plan, Test Plan Folder, or Test Plan Instance and select Run Plan Item to execute it and report the results to Provar Manager.
Above: Snapshot of Test Plan executed successfully.
A Test Cycle will be created that contains each test execution record.
Run with ANT
When exporting Run Under ANT for a Test Plan, you can configure reporting to Provar Manager so that test execution results are reported to Provar Manager.
Test Plans can be run via ANT in two ways:
Option 1: The user selects the test plan, folder, or instance they’d like to run and right-clicks it to open the context menu. Then you can select the Generate ANT build file option.
Option 2: The user selects the test plan, folder, or instance they’d like to run and clicks the ANT icon from the run menu at the top.
In either option the user is presented with the Export ANT build file dialog. In the Export ANT build file dialog the Enable Provar Manager (Test) reporting checkbox is selected by default if the Provar Manager connection is configured.
Reporting will happen in the same way as in Run mode. All results will be reported at the end of the ANT build.
Run with ProvarDX
In order to report ProvarDX test executions to Provar Manager, you can either add the necessary parameter to the ProvarDX property file or export the Test Plan as ProvarDX from the Test Plan item.
This must be enabled in Provar Automation V2.
Note: Exporting to ProvarDX is currently unsupported in Provar Automation V3. However, the V3 CLI will recognize Provar Manager reporting settings.
How Results are Mapped and Reported in Provar Manager
To understand how results are uploaded to Provar Manager, we first look at the object mapping between Provar Automation and Provar Manager when executing a test and uploading the test execution results to Provar Manager.
To get the reports in Provar Manager, you must configure the Test Plan. For example, which test cases have passed, this information should be in Provar Manager reports.
So, here are some pre-conditions :
- The Test Plan must be uploaded to Provar Manager.
- Enable the configuration in the Test Plan for reporting in Provar Manager.
All the test files of the Test Plan must be uploaded as you need to run the test case(s) , and if you have not uploaded the test case, then the test case report won’t be there in Provar Manager. At the end of the last test instance execution for the entire test run, results are then sent to Provar Manager.
When you run your test, the report generated in Provar Manager has two main parts:
- Test Cycle – These have the configuration information only. This includes information about the OS, tester, browser, and test environment, and overall status for the run.
- Test Execution – These include the name of the test case, execution time, and the status. You will also find detailed test step level information and status on the Test Execution record.
Test Cycles
From the Test Management application, click the Test Cycles tab.
In this list view you can see the Test Plan, environment, browser, resolution, OS name & version, testing tool, testing tool version, etc. You can create your own filters or view for which fields you’d like to display for the Test Cycles tab.
The Details tab shows a view of some of the high-level information, such as start & end time, Test Plan, test environment, and status.
The Test Execution tab displays all of the test execution records in the cycle. You can find the start time, status, OS, browser, tester, and # defects related to the record here. The hyperlink to the execution record will show you a deep dive on the test case results.
The Configuration tab displays basic configuration data on the Test Cycle, such as testing tool & version, test environment settings, and test plan schedule and job related records.
Test Executions
From the Test Management application, you can either select a previous Test Cycle, or access the Test Execution records directly from the Test Executions tab.
The Details tab shows the Test Cycle it’s related to, the Test Case, tester, status, test suite it belongs to (test plan folders in Provar Automation), and various test settings about the OS and browser.
The Results tab shows the full test execution timeline, with embedded screenshots, video playback (Provar Grid only), ability to raise defects from test failures, and associated defects.
You can see all the details for each test step when selected. It will report the status of each step, its name, execution time, and an output of the logs from the Provar CLI. The failure message can be seen in the Comments section if the test step fails.
In the Results section, you will also see the callable test case execution reports.
Known Issues:
- Currently, the user cannot upload more than 350 test cases in one Test Plan.
- On uploading all the plans in a project, the upload dialog gets stuck intermittently, and the user cannot proceed further in the dialog.