Integrating with Provar Manager

Provar Automation’s seamless integration with Provar Manager provides a great user experience. A brief overview of this integration is given below –

Provar Manager Connection – to create a connection to a Provar Manager instance.
Test Project Context Menu – to provide an option to upload the relevant Provar test project files to Provar Manager and associate Provar test files at the Test Plan level.
Test Plan configuration – to provide an option to report Test Plan execution to the Provar Manager.
Test Execution configuration – to provide an option to report test execution results to the Provar Manager.
Test Execution Results – The test execution results for Provar Manager reporting only.

Setting up a Connection to Provar Manager

The Provar Manager connection establishes a connection with a Provar Manager instance. It is configured by creating a Provar Manager connection via the Connection tab. The Provar Manager connection (leveraging a normal Salesforce Connection) only requires a user login via OAuth Web Flow.

Prerequisites

There are some prerequisites for setting up a connection to Provar Manager as listed below:

A Salesforce org with Provar Manager must be already installed.
A Salesforce user with a Provar Manager license assigned and the Provar Manager permission set assigned too.

How to set up an OAuth Web Flow connection to Provar Manager

Step 1: Create a Connected App.

Step 2: Create a Provar Manager Connection (OAuth WebFlow).

In the Test Settings, navigate to the Connections tab and click the plus (+) sign. The Add New Connection screen is displayed. Give a Connection Name and create an OAuth WebFlow Connection. In the Connection Type field, select Provar Manager then select Provar Manager Connection.

For more information, please see Salesforce OAuth (Web Flow) connection.

In the created OAuth WebFlow connection, you have to provide the details of the Consumer Key, Consumer Secret, and Callback URL that is a part of OAuth WebFlow connection.

Note: The above 3 fields are not the credentials of the user but they are the credentials of the Connected App.

Click Authorize, and it will ask for your credentials; provide your credentials, and when the authorization is successful, the Access Token and Refresh Token are populated.

Note:

Provar Manager is only available via the Salesforce Lightning Experience.
Only one Provar Manager connection can be set.

Above: Snapshot of creating a Provar Manager Connection.

When the Provar Manager connection is created, you can see it under the Connections tab.

Above: Screenshot of Provar Manager Connection in Provar.

Creating and linking the Test Project in Provar Manager

Once you have created the Provar Manager connection; a project is created in Provar Manager with a Test Project Name and a unique Test Project Key. For example, the project AY Test Project is created in Provar Manager which has a unique key.

If a user wants to link a Provar test project with an existing Provar Manager project then he has to provide that key (unique key) while creating the Provar Manager connection in Provar.

Note: The project was created in Provar Manager with the same name and unique key.

In Provar Manager –

Log on to the Provar Manager. Click Environments > Test Projects. You can see the AY Test Project created in Test Manager.

Above: Snapshot of the Project created in Provar Manager.

Click on the project (AY Test Project), and you can see the details of the project.

When you click on the project, you can see Test Plans inside it. When you click on Test Plans, you can see Test Suites (these are Test Plan Folders in Provar) and Test Cycles inside it.

Above: Details of the project created in Provar Manager.

Uploading screenshots

To upload screenshots, we have provided one option at the connection level. You can now select the Send Screenshots checkbox in the connection dialog box to upload screenshots.

This option is now available. If you have selected this checkbox then whatever screenshots are captured, they will all be uploaded. If it’s unchecked or de-selected, then no screenshot will be uploaded.

Note: The screenshot capture is allowed only in UI API’s. For example, in the UI Assert test case in the above test case example, you can see the options:

screenshot before
screenshot after

Note: In case of an error, a screenshot will always be created and uploaded. So, the maximum number of possible screenshots on a test step execution is 3 (before, after, and on-error). The screenshots can be identified as (before_ and after_).

Removing a Provar Manager Connection

Removing or deleting a Provar Manager connection removes all the Provar Manager associations. Right-click on Provar Manager connection to delete and the Provar Manager connection is deleted, and the context menu Upload to PM won’t be visible in Provar.

Object Mapping between Provar Automation and Provar Manager

The supported Provar test objects are mapped to the Provar Manager objects as shown below.

Above: Diagrammatic view of object mapping between Provar Automation and Provar Manager.

In Provar, the structure of Test Object(s) is organized as Test Plan > Test Plan Folder > Test Instance > Test Case. If you click on a Test Instance, there is a test case inside it which is Test Case 1.

Above: Screenshot of Test Objects in Provar.

In the screenshot above, you can see AY Plan 1 > AY Folder 1 > Test Case 1.testinstance > Test Case 1. Testcase. All these are linked in the hierarchy.

How to upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases in Provar Manager

Prerequisite:

To work with Provar Manager integration, users must have Test Plans (Test Plan Folders > Test Instance > Test Case) created before uploading or reporting execution to the Provar Manager.

In this section, you will learn how to upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases in Provar Manager, and you will also learn about the parent-child hierarchy.

To upload a Test Plan from Provar to Provar Manager, right-click on the Test Plan (i.e., the test object) and click Upload to PM.

Note: When you upload a Test Plan in Provar Manager, all the corresponding test objects in the Test Plan (i.e. Test Plan Folder, Test Instance, and Test Case) will also be uploaded to the Provar Manager. You can upload (create/update), link, and unlink these as required. For more information, see the below section Understanding the Parent-Child Hierarchy.

Understanding the Parent-Child Hierarchy

There is a parent-child hierarchy in the test objects. You can take advantage of the tree structure. For example, if you want to upload the Test Plan, then all the contents of the Test Plan are also uploaded. So, you just have to select the Test Plan, and Provar will automatically select all the contents of the Test Plan, you don’t need to select the contents individually.

Above: Image showing the tree structure hierarchy of a Test Plan.

Note: If you select any single test object, then Provar will automatically select its child as well as the parent. Provar automatically creates the hierarchy and reduces any chances of an error.

For example, if you just want to upload a single test case, then you can select it, and Provar will automatically select its parent as well as its child. So, even if you select AY Folder 2, Provar will automatically select AY Plan 1, and you can upload or update it in Provar Manager.

Above: Screenshot displaying the hierarchy. Selecting a test case and Provar automatically selects the parent and child test objects.

Note: Hierarchy is only for the Test Plans. Environments are independent of hierarchy as well as of each other. So, you have to select the environments individually as they are not connected to each other. For example, DEV and UAT environments are different and must be selected individually only.

Above: Selecting DEV and UAT environments separately.

The Upload to Provar Manager dialog provides the users with the ability to associate Provar Test Files to Provar Manager as listed below:

Creating a new object in Provar Manager from a Provar Test Object via upload.
Linking an existing Provar Manager object to a Provar Test Object via the “link” action.
Unlinking an associated Provar Test Object from Test Manager via the “unlink“ action.

Additional usability capabilities are also available to filter and identify when Provar Test Objects are matched, not matched, and could be inferred as a match based on the matching criteria.

Filters

Let’s first look at the filters (the checkboxes) on the top-right of the screen. You can use these as per your requirements. Although these filters are self-explanatory, we have provided brief explanations as mentioned below:

All Files – It will show all the files (that are in Provar).
Missing Files – The files i.e. the test objects unavailable in Provar Manager.
Changed Files – The files, i.e., the test objects that have been changed. (For example, renamed).
Failures – In case some failure has occurred. Failure can only happen when you have uploaded something, and it hasn’t been uploaded, and it failed.

We have given some examples below. We have selected All Files, and it is displayed as given below:

Above: Snapshot of filtering DEV environment that is renamed to DEV-Updated.

As a user, if you want to upload Provar Test Objects to the Provar Manager, right-click on the Test Plan (i.e., test object) and click Upload to PM. Please see the screenshot given below.

Above: Snapshot of Upload to PM context menu.

An Upload to Provar Manager dialog is displayed. You will also notice a message Fetching Test Objects from Provar Manager.

Above: View of Upload to PM dialog fetching test objects from Provar Manager.

Select All files check box at the top of the dialog box, and the fetched data is displayed in the dialog. Users can select and upload the relevant files to the Provar Manager.

You can see 5 fields on the Upload to Provar Manager dialog. These are explained below.

Provar Test Object – This field shows the test objects in Provar.
Test Manager Object – This field shows the test objects in Provar Manager.
Match Type –
- Inferred: It is an exact match of the Provar test object in Test Manager, but it isn’t linked. It has the same configuration in Provar Manager. You can link it. You need to explicitly link it. (There is no need to create the object, it is already created.)
- Blank: The Provar test object is not available in Provar Manager.
- Linked: The same test object is available in Provar and Provar Manager. And it is linked.
Status –
- Missing: The missing status is shown for test objects that are missing in Provar Manager. For example, you created a Test Plan but didn’t upload it in Provar Manager yet, so it will show the Status as Missing.
- “?”: Either something has changed in the test object, or the Provar cannot comprehend the change. This is an unknown status.

For example, you renamed a test object but didn’t upload it in Provar Manager (i.e., there is a mismatch in Provar and Provar Manager). You must update it accordingly. So, when you upload it, it will also be renamed in Provar Manager.

“-”: There is no change in the data. For example, AY Plan 1 is AY Plan 1.
Failed – If there is an error while uploading the test object, then the Status is displayed as Failed, and on hovering over it, you can see the message.

Actions –
- Link: Click the link to associate the Provar test object to Provar Manager.
- Unlink: Click unlink to dissociate the Provar test object from the Test. Manager (If it is linked by mistake, you linked Plan 1 to Plan 2.)

Note: If the data in Provar Test Object and Provar Manager Object field matches as one-to -then it will be displayed, otherwise it will be blank. The other columns contain additional information which is helpful to the user.

The Provar Test Object field shows the local Provar data, i.e., the test objects in Provar. The Provar Manager Object field shows the Provar Manager data, i.e., the test objects in Test Manager. If this data matches one-to-one, then Provar will show this data. Otherwise, it is blank. And, then we have some additional information (Match Type, Status, Actions) which are helpful for the user.

For example, suppose the user had earlier uploaded a plan (AY Plan1), which exists with the same name in Test Manager (AY Plan1). Then it is an exact match, and the Match Type is shown as Inferred; it is a potential match, and you can link it as displayed in the Actions column.

In the Actions field, click the link. A Test Manager Object selection dialog is displayed.

Note: It will display information depending on which test object the user clicked the link. For example, if a user clicks the link against the Test Plan (AY Plan1), he can see all the Plans only.

Above: The user clicks a link in the Actions field to link the Test Plan.

Users can see only the plans listed in the Provar Manager Object selection dialog.

Note: If the user clicked the link against the Test Plan Folder, then the Test Plan folder is Test Suite in Provar Manager; only test suite(s) would have been displayed.

Similarly, if the user had clicked the link against the test case, only test cases would have been displayed.

Note: The test objects (Test Case, Test Instance, Test Plan, Test Folder, except the Environment.) can be similarly uploaded in Provar Manager. The test object against which the user clicks the link is the only one linked; there isn’t any mismatch.

Now, create a new environment in Provar.

In the bottom section of the Provar, click Environments. Click ‘+’. An Add new Test Environment screen is displayed. Enter the Environment Name and Description. For more information, please see Provar Environment Management.

Note: Similarly, you can create environments as needed, like QA, UAT, etc.

Click Upload to Provar Manager. The Upload to Provar Manager dialog is displayed.

Select the All files checkbox. All the test objects are loaded into the environment. So, this fetches all the data. Here we have 5 types of objects – Test Case, Test Instance, Test Plan, Test Folder, and Environment. This triggers 5 different queries to the Provar Manager.

Note: Test Environments are always displayed at the top and have no hierarchy.

So, if the user wants to see the environments, it will trigger a query for the environment in Test Manager. In Test Manager, click Environments, and all the environments are displayed.

Above: Snapshot of Environments in Provar Manager.

Next, we have triggered a query on all the Test Plans, which are displayed.

Example 1: Explaining the DEV-Updated example.

If we look at the above screenshot for better understanding. Provar shows the environment as DEV – Updated. Test Manager shows the environment as DEV. So, if you see here, Match Type shows as Linked (this means that data was earlier linked in local at sometime (when it was in sync), but now it is renamed.), and Status shows “?” (and this is the reason it shows a question mark (?)).

Example 2: An example when there is no change.

Let’s see another example of AY Plan 1. It shows Match Type as Linked and Status as “- “, which means that there has been no change.

Now, you have to upload DEV-Updated. Click Upload. So, it will update it and not create it.

In Provar Manager –

Click Environments. You’ll see DEV-Updated in Provar Manager.

Above: Snapshot of Dev-Updated example in Provar Manager.

Uploading Callable Test Cases in Provar Manager

You can upload the callable test cases if a dependency of a test case is being uploaded to the Provar Manager. Select the callable test case checkbox to upload the callable test case.

You can see the Callable test cases at the bottom of the tree view under the Callable Test Cases section. You can expand/collapse the Callable Test Cases section. You can link/unlink callable test cases like any regular test cases. Related callable test cases are automatically selected when a test case is selected. There is no limit to the nesting of callable test steps (i.e., a callable test can call another callable test); however, the testing is limited up to 3 levels.

Steps to Upload Callable Test Cases

Right-click on a Test Plan and click Upload to Provar Manager.
When you select a Test Plan, all its dependent test cases and the Callable Test Cases are automatically selected. Click Upload.

Note: You can also select individual callable test cases, but you cannot deselect the dependent callable test cases from a plan.

3. You will see a message when the test cases are uploaded successfully.

In Provar Manager–

In the Test Steps section, open any uploaded Test case. Within the test steps, you will see callable test cases uploaded.

Open the callable test case and select the Test Case to Call field; it points to the dependent test case.

Open the test case to view its test steps.

Uploading Test Steps in Provar Manager

Now with the test case upload, we are also uploading the test steps. Now, suppose you want to upload a test case. For example, if you are uploading a “Break_with_while” test case.

Note: A warning is displayed ”Uploading a test case will overwrite all the test steps in Provar Manager”. If other users have already created the test steps manually in Provar Manager and if you are going to upload your test case, that’s where this warning message will be helpful.

Above: Snapshot of warning message at the top that test steps will be overwritten.

Previously, when you uploaded a test case, only the details (Details, System Information, etc., as shown below in the screenshot) were populated in Provar Manager.

The addition is in the Test Steps section. So, if you are uploading a test case now, you will also see the Test Steps uploaded. So, in the Test Steps section, you can see these test steps. So, whatever you upload is the latest data.

Note: Whatever configuration, whatever Test steps you have in Provar Manager (in the Test Steps section, it doesn’t matter) will get overwritten by the configuration or the information you are having in Provar Automation. So whatever information you have in your local Provar Studio is the source of truth. So, all this information will be uploaded to the Provar Manager.

Above: Snapshot of Test Steps uploaded in Provar Manager.

You also have the option to edit the test steps in Provar Manager; click Edit. You can manually edit, add a test step (click Add Step), delete the test step, can save it. Click Save. A message “Changes saved!” is displayed.

Limitation to uploading Test Steps in Provar Manager

Firstly, earlier, we used to send/ upload test objects in 200 batches like send plans 200, folders 200, instance 200, and also send 200 test cases. The only change now is that 200 test cases won’t be sent. Test cases will be sent one by one to upload in PM. So, now test cases would be sent one by one, first test case, then second test case, and third, and so on and so forth; because of this, the performance is somewhat slower compared to the earlier scenario when test cases were sent in batches.
Secondly, the API of Salesforce that we are using, that API has a limitation. So, the one call you send, you can make only 500 changes at maximum. That means, in your test case 500 changes can be done at maximum. For example, you made a very lengthy test case with 1000 steps that will not be uploaded; it will fail.

Here, when we say that the limit is 500, it doesn’t mean that to create it is 500, so when it is overwritten, it will also have to delete this data and then upload the new data. So, this collective information(deletion, insertion, and update) is 500.

Note: The user shouldn’t assume they can send 500 steps at a time. The collective information(creation, updation, insertion, and deletion ) is 500. That means they can send 500 test step changes. For example, we have a test case that has 500 test steps, and we try to upload it. So, the test case upload will fail and throw this error message, as shown in the screenshot below.

Provar Manager File Decoration

How will users know if a Test Plan structure is linked in Test Manager? Provar uses Provar Manager File Decoration to provide a visual hint to the user when the relevant Provar test objects/files are linked to the Provar Manager. So, just by a glance, the user can see which Test objects are already uploaded and linked in Provar Manager.

A reference is presented to the user to show the linked object in Provar Manager and provide visual feedback; the word Provar Manager is displayed next to the file in the format [AssociationType] e.g. [ProvarManager]. For an associated Test Plan, the Provar Manager File Decoration is displayed as given in the screenshot below.

Note: This visual reference [ProvarManager] is displayed for all test objects linked to the Provar Manager, i.e., Environments, Test Plans, Test Plan Folders, Test Instances, and Test cases.

Above: Provar Manager File Decoration is displayed for the Test Plans.

Test Execution Reporting

When configuring a new Test Plan, the user must select an option to report test execution results to the Provar Manager.

Prerequisites

A Connection to the Provar Manager must already be created.
Configure the Test Plan and provide an option to report test execution results to the Provar Manager.

Note: You need to create a Person record (or test record) that will be used as a tester in reporting execution to signify who has initiated the execution and who is the tester. This Person record must use the email address “automation@provartesting.com”

When the Provar reports execution to the Provar Manager, we fill the tester field for that execution. This tester field in Provar Manager accepts Person type objects. So, there has to be a Person object created before reporting execution. Otherwise, the reporting will fail.

How to configure Test Plans for reporting in Provar Manager

You can configure Test Plans for reporting in Provar Manager in two cases, as explained below.

Create a new Test Plan. For more information on creating a Test Plan, please see Test Plan or double-click on the Test Plan if you already have a Test Plan created.

Case 1: When you are creating a new Test Plan.

When creating a new Test Plan, configure it to report to the Provar Manager.

In the New Provar Test Plan screen, select the Provar Manager (PM) check box in the Reporting section.

Note: The Connection Name (PM) is the name given to the Provar Manager connection by the user.

Case 2: If you already have a Test Plan created.

Double-click on an existing Test Plan. The Test Plan Editor is displayed.

Click “+” (i.e., Add a new feature in the Test Plan). Select the Provar Manager(PM) reporting option.

Note: Once this PM reporting is configured, the next time you run or execute the Provar Manager, whichever option you have configured for the Test Plan will work, for example, PDF reporting, etc.

In the Test Plan Editor, in the Provar Manager section, select the Enable Provar Manager Reporting check box. When enabled, once a test execution is completed (passed/failed) for the Test Plan and the associated Test Files, the results are reported to the Provar Manager.

Note: You can clear the check box if you don’t want the reporting in Provar Manager.

Note: If you have enabled the reporting but you haven’t uploaded the Test Plan, then reporting will not be done. So, the user must first upload the Test Plan to the Provar Manager.

What is supported right now

As of date, what Provar supports is listed below:

Test Plan Execution Failure – If the Test Plan execution fails, the Provar Manager’s Test Execution object will be updated to reflect the failure.
Test Plan Folder Execution – On selecting a Test Plan Folder for execution, the Provar Test Execution will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the selected Test Plan Folders.
Test Plan Instance Execution – On selecting a Test Plan Instance(s) for execution, the Provar Test Execution engine will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the select Test Plan Instances.

Note: Provar does not support running tests from the Test Builder and Debug modes.

Running tests from Provar

As of now, running tests from the Run option is only supported. Once you have created a Test Plan, you can execute it. Click on the Test Plan and click Run. The Test Plan is executed.

Note: The Test Plan is executed, and there is no reporting in Provar Manager at this stage.

Above: Snapshot of Test Plan executed successfully.

Run with ANT build.

When exporting Run Under ANT for a Test Plan, you can configure reporting to Provar Manager so that test execution results are reported to Provar Manager.

When the user selects the option to Run With ANT for a Test Plan, the user is presented with the Export ANT build file dialog. In the Export ANT build file dialog, on the Details tab; in the Features section, the Enable Provar Manager (Test) reporting checkbox is selected by default if the Provar Manager connection is configured. Otherwise, it isn’t selected.

How results are uploaded to Provar Manager

To understand how results are uploaded to Provar Manager, first, look at the object mapping between Provar Studio and Provar Manager when executing a test and uploading the test execution results to Provar Manager.

To get the reports in Provar Manager, you must configure the Test Plan. For example, which test cases have passed, this information should be in Provar Manager reports.

So, here are some pre-conditions :

The Test Plan must be uploaded to Provar Manager.
Enable the configuration in the Test Plan for reporting in Provar Manager.

Note: If you have only uploaded the Test plan in Provar Manager, it will not work. You will have to enable configuration for reporting in Provar Manager.

All the test files of the Test Plan must be uploaded as you need to run the test case(s) , and if you have not uploaded the test case, then the test case report won’t be there in Provar Manager. When you execute the test case, a report will be generated in Provar Manager.

Just before the test instance in green, we report it. See the screenshot below.

Above: Snapshot of Test Plan execution.

When you run your test, the report generated in Provar Manager has two main parts:

Test Cycle – The test Cycle has the configuration information only. For example:

Which was the machine, what was the OS, who executed it, which Browser, and which Environment? All this information is in the Test Cycle.

Test Execution – This section has the actual information like which test case you executed, how much time you executed, how much time it took for execution, and what’s the status (pass or fail).

In Provar Manager, the execution will look as shown below.

Click Test Projects > Test Cycles. If a new session starts, it will show all the details. Then, a new record would be created in the Test Cycle.

For example, here, we have clicked on AY Plan 1.

Test Cycle Number is auto-generated in Provar Manager. Provar doesn’t give this number. You can see the information, such as Plan, Environment, Browser, Resolution, OS Name, OS version, testing tool, testing tool version, etc. So, the Test Cycle will show just the configuration information, but nowhere does it mention which test case is executed and whether that test case has passed or failed; so for that, there is a different object that comes with Test Execution.

Let’s look at the complete cycle as an example: For example, a user executed 2 test cases, so for these 2 test cases, 2 reports are generated i.e., for these 2 test cases, 2 test execution objects are created.

For example, we have opened the first report.

Note: We have also covered the Test Step execution level i.e., every test step execution is reported.

As of now, we have showcased the complete test case execution in one go. So, if a test case will have multiple test steps, then each test step execution will be showcased.

Now that the test steps are uploaded in Provar Manager, the results are also uploaded, and the test step errors are mapped to the Comments field.

Click on the Related section to see that the execution records are created.

Click on the first execution, which is the latest execution, and you can see all these test step execution reports in the Test Step Executions section.

Now, suppose if you click and open one test step; you can see all the details, whether it passed or failed, its name, how much time it took for execution, and what information is in the logs. The failure message can be seen in the Comments section if the test step fails.

At the end of the Test Plan, we are reporting Test Step execution, and you will see a success message at the end of complete execution.

Note: You can also see the callable test case execution reports.

In the Test Step Executions section, you can also see the callable test case execution reports.

Above: Snapshot of callable test cases execution report in Provar Manager.

Support reporting to Provar Manager during ProvarDX execution

If the option to report to Provar Manager is enabled in the ProvarDX JSON settings, ProvarDX will report the test executions to Provar Manager as expected.

Test executions (test cycles, test executions, etc.) are reported by ProvarDX to Provar Manager when the option is enabled or missing in the ProvarDX JSON settings.
Test executions (test cycles, test executions, etc.) are NOT reported by ProvarDX to Provar Manager when the option is disabled in the ProvarDX JSON settings.

Known Limitation

Currently, the user cannot upload more than 350 test cases in one Test Plan.
On uploading all the plans in a project, the upload dialog gets stuck intermittently, and the user cannot proceed further in the dialog.

Documentation library

Documentation

Looking for something in particular?