Integrating with TestRail

To efficiently manage your test cases and runs, you can execute test cases and record test results using TestRail. So, you can update the execution status of the test cases in TestRail from Automation when a test case gets Pass/Fail/Skipped during the execution. We integrate Provar with TestRail using TestRail REST endpoints, which is done by using the web services connection available in Automation.

In brief, the TestRail integration allows you to link a test case in Automation to one in TestRail. After execution, the test run results are shown in TestRail and give you the full and thorough clarity of your test results.

Briefly, the overall workflow of TestRail integration with Automation is as given below:

1) Create test cases in TestRail.

2) Create a TestRail connection in Automation.

3) Create a TestRun setup.

4) Execute the test cases in Automation and update the test case results in TestRail.

Above: Depicting overall steps and workflow for TestRail integration.

To begin with, we have first created three test cases in TestRail.

Create Account with ID as C1.
Create Opp with ID as C2.
Create Test with ID as C3.

These C1, C2 and C3 represent test case ID in TestRail.

Note: Users must pass the exact test case ID generated by TestRail (after a test case is created in TestRail) in Automation to get the status updated after the execution from Provar.

Above: Displaying test cases created in TestRail.

Steps for integrating TestRail with Automation

Step 1: Add a TestRail connection.

To add a TestRail connection, select Generic Web Service from the Connection Type drop-down menu.

Note: Go to Test Settings. On the Connections tab, click ‘+’ to add a new connection. For more information, please see Web Service connection.

You will see the Add New Connection window as displayed below. To add a TestRail connection, select Generic Web Service from the Connection Type drop-down menu.

Above: Adding a TestRail connection in Automation.

Step 2: Add an authentication.

In the URL field, enter the TestRail URL. Select Authentication Type from the drop-down menu and enter the relevant authentication details.

Note: TestRail supports two authentication types:

Username and Password
Username and API Key

In this example, we have selected Basic Authentication with username and password. For more information, please see authentication types.

Above: Adding an Authentication Type.

Step 3: Test the connection.

Click Test Connection to check if the connection is successful to ensure the communication between Automation and TestRail is created properly.

Above: View of the Test Connection dialog box.

Step 4: Use the TestRail connection in your test case.

To use the TestRail connection in your test case, drag and drop Web Connect and Web Request (REST) APIs in your test case from the Test Palette. For more information, please see web services.

Above: Using Web Connect and Web Request (REST) API’s in the test case.

Step 5: Select the TestRail connection.

When you drag and drop the Web Connect API in your test case from the Test Palette, a Test Step parameters screen is displayed on the right-hand side.

Select the TestRail web services connection from the drop-down in the Connection field.

Above: Passing the TestRail connection using the Web Connect API.

Step 6: Configure the Web Request (REST) step and set up the test case.

Now, perform a REST request. Drag and drop the Web Request (REST) API in your test case from the Test Palette.

Above: Set up the Test case and perform the Web (REST) request.

Click the TestRail connection you passed in the above Web connect step in the Connection Name list.

Create a Test Run in TestRail through Automation in the Resource URL field with the endpoint: “/index.php?/api/v2/add_run/:project_id“ where :project_id is the id of the project created in TestRail. For more info, please refer to Projects.

In the example above, we have used the Resource URL (“/index.php?/api/v2/add_run/1“ ) where 1 is the id of your project in TestRail.

In the REST Type field, select Generic REST Service.
In the Request Header field, enter the optional parameter for HTTP headers, as Content-Type:application/json.
In the Method field, select Post(create).
In the Body Style field, select JSON.

Note: The Body parameters used are:

⦁ Name: Name of the test run (Provar run (NOW)) created in TestRail.

⦁ Case_ids: There is an ID associated with each test case in the TestRail. We pass these ID(s) in Provar and when the test case execution is completed; the results of these test case(s) are updated in TestRail. These ID(s) are 1, 2, and 3. Please see TestRail Case and Status for more information.

Also, please refer to Runs for more information on Test Runs.

Step 7: Update test case results on TestRail.

We will now update the test execution results in TestRail of the test cases (Create Account, Create Opp, Create Test) we created in Automation. Here, we are taking an example of the Create Account test case.

Above: Updating test case results in TestRail.

To update the results in TestRail, first Get TestRail ID.

Above: Updating test case results based on the value in TestRail.

The TestRail Execution Status is defined in the Value Name field and updates the status of the test case in TestRail based on the numeric value in the Value field. i.e. (with the variable based on the value). The Status ‘1’ signifies ‘Pass’ and ‘5’ signifies ‘Fail’.

So, based on this value, the test case status is updated in TestRail. For more information, please see Test status in TestRail for the list of all TestRail status.

Above: Updating the test execution result in TestRail.

When the Test case execution is successful in Automation, the test result is updated in the TestRail or if the test case execution fails then the test result is updated in the TestRail with an error message.

Note: In the Test Step parameters screen, the status of the test case is updated in the Resource URL field with the TestRail ID “/index.php?/api/v2/add_result_for_case/(RestResponse_AddRun.id)/TestRail_id”, as displayed in the screenshot given above.

In the Body section, note the ‘status_id’ is populated with ‘TestRail_Execution_Status’, and an error message is passed if the test case fails and an error occurs.

Step 8: View the test results as updated and displayed in TestRail.

The view of the test case results is updated and displayed in TestRail, as shown below with the screenshots.