Test Execution Reporting
When configuring a new Test Plan, the user must select an option to report test execution results to the Provar Manager.
Prerequisites
- A Connection to the Provar Manager must already be created.
- Configure the Test Plan and provide an option to report test execution results to the Provar Manager.
Note: You need to create a Person record (or test record) that will be used as a tester in reporting execution to signify who has initiated the execution and who is the tester. This Person record must use the email address “automation@provartesting.com”
When the Provar reports execution to the Provar Manager, we fill the tester field for that execution. This tester field in Provar Manager accepts Person type objects. So, there has to be a Person object created before reporting execution. Otherwise, the reporting will fail.
How to configure Test Plans for reporting in Provar Manager
You can configure Test Plans for reporting in Provar Manager in two cases, as explained below.
Create a new Test Plan. For more information on creating a Test Plan, please see Test Plan or double-click on the Test Plan if you already have a Test Plan created.
Case 1: When you are creating a new Test Plan.
When creating a new Test Plan, configure it to report to the Provar Manager.
In the New Provar Test Plan screen, select the Provar Manager (PM) check box in the Reporting section.
Note: The Connection Name (PM) is the name given to the Provar Manager connection by the user.
Case 2: If you already have a Test Plan created.
Double-click on an existing Test Plan. The Test Plan Editor is displayed.
Click “+” (i.e., Add a new feature in the Test Plan). Select the Provar Manager(PM) reporting option.
Note: Once this PM reporting is configured, the next time you run or execute the Provar Manager, whichever option you have configured for the Test Plan will work, for example, PDF reporting, etc.
In the Test Plan Editor, in the Provar Manager section, select the Enable Provar Manager Reporting check box. When enabled, once a test execution is completed (passed/failed) for the Test Plan and the associated Test Files, the results are reported to the Provar Manager.
Note: You can clear the check box if you don’t want the reporting in Provar Manager.
Note: If you have enabled the reporting but you haven’t uploaded the Test Plan, then reporting will not be done. So, the user must first upload the Test Plan to the Provar Manager.
What is supported right now
As of date, what Provar supports is listed below:
- Test Plan Execution Failure – If the Test Plan execution fails, the Provar Manager’s Test Execution object will be updated to reflect the failure.
- Test Plan Folder Execution – On selecting a Test Plan Folder for execution, the Provar Test Execution will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the selected Test Plan Folders.
- Test Plan Instance Execution – On selecting a Test Plan Instance(s) for execution, the Provar Test Execution engine will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the select Test Plan Instances.
Note: Provar does not support running tests from the Test Builder and Debug modes.
Running tests from Provar
As of now, running tests from the Run option is only supported. Once you have created a Test Plan, you can execute it. Click on the Test Plan and click Run. The Test Plan is executed.
Note: The Test Plan is executed, and there is no reporting in Provar Manager at this stage.
Above: Snapshot of Test Plan executed successfully.
Run with ANT build.
When exporting Run Under ANT for a Test Plan, you can configure reporting to Provar Manager so that test execution results are reported to Provar Manager.
When the user selects the option to Run With ANT for a Test Plan, the user is presented with the Export ANT build file dialog. In the Export ANT build file dialog, on the Details tab; in the Features section, the Enable Provar Manager (Test) reporting checkbox is selected by default if the Provar Manager connection is configured. Otherwise, it isn’t selected.
How results are uploaded to Provar Manager
To understand how results are uploaded to Provar Manager, first, look at the object mapping between Provar Studio and Provar Manager when executing a test and uploading the test execution results to Provar Manager.
To get the reports in Provar Manager, you must configure the Test Plan. For example, which test cases have passed, this information should be in Provar Manager reports.
So, here are some pre-conditions :
- The Test Plan must be uploaded to Provar Manager.
- Enable the configuration in the Test Plan for reporting in Provar Manager.
Note: If you have only uploaded the Test plan in Provar Manager, it will not work. You will have to enable configuration for reporting in Provar Manager.
All the test files of the Test Plan must be uploaded as you need to run the test case(s) , and if you have not uploaded the test case, then the test case report won’t be there in Provar Manager. When you execute the test case, a report will be generated in Provar Manager.
Just before the test instance in green, we report it. See the screenshot below.
Above: Snapshot of Test Plan execution.
When you run your test, the report generated in Provar Manager has two main parts:
- Test Cycle – The test Cycle has the configuration information only. For example:
Which was the machine, what was the OS, who executed it, which Browser, and which Environment? All this information is in the Test Cycle.
- Test Execution – This section has the actual information like which test case you executed, how much time you executed, how much time it took for execution, and what’s the status (pass or fail).
In Provar Manager, the execution will look as shown below.
Click Test Projects > Test Cycles. If a new session starts, it will show all the details. Then, a new record would be created in the Test Cycle.
For example, here, we have clicked on AY Plan 1.
Test Cycle Number is auto-generated in Provar Manager. Provar doesn’t give this number. You can see the information, such as Plan, Environment, Browser, Resolution, OS Name, OS version, testing tool, testing tool version, etc. So, the Test Cycle will show just the configuration information, but nowhere does it mention which test case is executed and whether that test case has passed or failed; so for that, there is a different object that comes with Test Execution.
Let’s look at the complete cycle as an example: For example, a user executed 2 test cases, so for these 2 test cases, 2 reports are generated i.e., for these 2 test cases, 2 test execution objects are created.
For example, we have opened the first report.
Note: We have also covered the Test Step execution level i.e., every test step execution is reported.
As of now, we have showcased the complete test case execution in one go. So, if a test case will have multiple test steps, then each test step execution will be showcased.
Now that the test steps are uploaded in Provar Manager, the results are also uploaded, and the test step errors are mapped to the Comments field.
Click on the Related section to see that the execution records are created.
Click on the first execution, which is the latest execution, and you can see all these test step execution reports in the Test Step Executions section.
Now, suppose if you click and open one test step; you can see all the details, whether it passed or failed, its name, how much time it took for execution, and what information is in the logs. The failure message can be seen in the Comments section if the test step fails.
At the end of the Test Plan, we are reporting Test Step execution, and you will see a success message at the end of complete execution.
Note: You can also see the callable test case execution reports.
In the Test Step Executions section, you can also see the callable test case execution reports.
Above: Snapshot of callable test cases execution report in Provar Manager.
Support reporting to Provar Manager during ProvarDX execution
If the option to report to Provar Manager is enabled in the ProvarDX JSON settings, ProvarDX will report the test executions to Provar Manager as expected.
- Test executions (test cycles, test executions, etc.) are reported by ProvarDX to Provar Manager when the option is enabled or missing in the ProvarDX JSON settings.
- Test executions (test cycles, test executions, etc.) are NOT reported by ProvarDX to Provar Manager when the option is disabled in the ProvarDX JSON settings.
Known Limitation
- Currently, the user cannot upload more than 350 test cases in one Test Plan.
- On uploading all the plans in a project, the upload dialog gets stuck intermittently, and the user cannot proceed further in the dialog.
For more information, check out this course on University of Provar.