Integrating with TestRail
To efficiently manage your test cases and runs, you can execute test cases and record test results using TestRail. So, you can update the execution status of the test cases in TestRail from Automation when a test case gets Pass/Fail/Skipped during the execution. We integrate Provar with TestRail using TestRail REST endpoints, which is done by using the web services connection available in Automation.
In brief, the TestRail integration allows you to link a test case in Automation to one in TestRail. After execution, the test run results are shown in TestRail and give you the full and thorough clarity of your test results.
Briefly, the overall workflow of TestRail integration with Automation is as given below:
1) Create test cases in TestRail.
2) Create a TestRail connection in Automation.
3) Create a TestRun setup.
4) Execute the test cases in Automation and update the test case results in TestRail.
Above: Depicting overall steps and workflow for TestRail integration.
To begin with, we have first created three test cases in TestRail.
- Create Account with ID as C1.
- Create Opp with ID as C2.
- Create Test with ID as C3.
These C1, C2 and C3 represent test case ID in TestRail.
Note: Users must pass the exact test case ID generated by TestRail (after a test case is created in TestRail) in Automation to get the status updated after the execution from Provar.
Above: Displaying test cases created in TestRail.
Steps for integrating TestRail with Automation
Step 1: Add a TestRail connection.
To add a TestRail connection, select Generic Web Service from the Connection Type drop-down menu.
Note: Go to Test Settings. On the Connections tab, click ‘+’ to add a new connection. For more information, please see Web Service connection.
You will see the Add New Connection window as displayed below. To add a TestRail connection, select Generic Web Service from the Connection Type drop-down menu.
Above: Adding a TestRail connection in Automation.
Step 2: Add an authentication.
In the URL field, enter the TestRail URL. Select Authentication Type from the drop-down menu and enter the relevant authentication details.
Note: TestRail supports two authentication types:
- Username and Password
- Username and API Key
In this example, we have selected Basic Authentication with username and password. For more information, please see authentication types.
Above: Adding an Authentication Type.
Step 3: Test the connection.
Click Test Connection to check if the connection is successful to ensure the communication between Automation and TestRail is created properly.
Above: View of the Test Connection dialog box.
Step 4: Use the TestRail connection in your test case.
To use the TestRail connection in your test case, drag and drop Web Connect and Web Request (REST) APIs in your test case from the Test Palette. For more information, please see web services.
Above: Using Web Connect and Web Request (REST) API’s in the test case.
Step 5: Select the TestRail connection.
When you drag and drop the Web Connect API in your test case from the Test Palette, a Test Step parameters screen is displayed on the right-hand side.
Select the TestRail web services connection from the drop-down in the Connection field.
Above: Passing the TestRail connection using the Web Connect API.
Step 6: Configure the Web Request (REST) step and set up the test case.
Now, perform a REST request. Drag and drop the Web Request (REST) API in your test case from the Test Palette.
Above: Set up the Test case and perform the Web (REST) request.
Click the TestRail connection you passed in the above Web connect step in the Connection Name list.
Create a Test Run in TestRail through Automation in the Resource URL field with the endpoint: “/index.php?/api/v2/add_run/:project_id“ where :project_id is the id of the project created in TestRail. For more info, please refer to Projects.
In the example above, we have used the Resource URL (“/index.php?/api/v2/add_run/1“ ) where 1 is the id of your project in TestRail.
- In the REST Type field, select Generic REST Service.
- In the Request Header field, enter the optional parameter for HTTP headers, as Content-Type:application/json.
- In the Method field, select Post(create).
- In the Body Style field, select JSON.
Note: The Body parameters used are:
⦁ Name: Name of the test run (Provar run (NOW)) created in TestRail.
⦁ Case_ids: There is an ID associated with each test case in the TestRail. We pass these ID(s) in Provar and when the test case execution is completed; the results of these test case(s) are updated in TestRail. These ID(s) are 1, 2, and 3. Please see TestRail Case and Status for more information.
Also, please refer to Runs for more information on Test Runs.
Step 7: Update test case results on TestRail.
We will now update the test execution results in TestRail of the test cases (Create Account, Create Opp, Create Test) we created in Automation. Here, we are taking an example of the Create Account test case.
Above: Updating test case results in TestRail.
To update the results in TestRail, first Get TestRail ID.
Above: Updating test case results based on the value in TestRail.
The TestRail Execution Status is defined in the Value Name field and updates the status of the test case in TestRail based on the numeric value in the Value field. i.e. (with the variable based on the value). The Status ‘1’ signifies ‘Pass’ and ‘5’ signifies ‘Fail’.
So, based on this value, the test case status is updated in TestRail. For more information, please see Test status in TestRail for the list of all TestRail status.
Above: Updating the test execution result in TestRail.
When the Test case execution is successful in Automation, the test result is updated in the TestRail or if the test case execution fails then the test result is updated in the TestRail with an error message.
Note: In the Test Step parameters screen, the status of the test case is updated in the Resource URL field with the TestRail ID “/index.php?/api/v2/add_result_for_case/(RestResponse_AddRun.id)/TestRail_id”, as displayed in the screenshot given above.
In the Body section, note the ‘status_id’ is populated with ‘TestRail_Execution_Status’, and an error message is passed if the test case fails and an error occurs.
Step 8: View the test results as updated and displayed in TestRail.
The view of the test case results is updated and displayed in TestRail, as shown below with the screenshots.
Above: Test results as updated and displayed in TestRail.
Above: A graphical representation of the test results updated in TestRail.
- Provar Automation
- Installing Provar Automation
- Updating Provar Automation
- Using Provar Automation
- API Testing
- Behavior-Driven Development
- Creating and Importing Projects
- Creating Test Cases
- Custom Table Mapping
- Debugging Tests
- Defining a Namespace Prefix on a Connection
- Defining Proxy Settings
- Environment Management
- Exporting Test Cases into a PDF
- Exporting Test Projects
- Override Auto-Retry for Test Step
- Managing Test Steps
- Namespace Org Testing
- Provar Desktop
- Provar Test Builder
- Refresh and Recompile
- Reintroduction of CLI License Check
- Reload Org Cache
- Running Tests
- Searching Provar with Find Usages
- Secrets Management and Encryption
- Setup and Teardown Test Cases
- Tags and Service Level Agreements (SLAs)
- Test Cycles
- Test Plans
- Testing Browser Options
- Tooltip Testing
- Using the Test Palette
- Test Palette Introduction
- Control Test Steps
- Generate Test Case
- List Compare
- Page Object Cleaner
- Read Test Step
- String Test Steps
- UI Test Steps
- Using Custom APIs
- Callable Tests
- Data-Driven Testing
- Page Objects
- Block Locator Strategies
- Introduction to XPaths
- Creating an XPath
- Label Locator Strategies
- Maintaining Page Objects
- Mapping Non-Salesforce fields
- Page Object Operations
- Refresh and Reselect Field Locators in Test Builder
- Using Java Method Annotations for Custom Objects
- Applications Testing
- Database Testing
- Document Testing
- Email Testing
- Mobile Testing
- OrchestraCMS Testing
- Guide in Salesforce CPQ Testing in Automation
- Guide in ServiceMax Testing
- Skuid Testing
- Vlocity API Testing
- Webservices Testing
- Provar Manager
- Introduction to Test Scheduling
- Apache Ant
- Configuration for Sending Emails via the Automation Command Line Interface
- Continuous Integration
- AutoRABIT Salesforce DevOps in Provar Test
- Azure DevOps
- Running a Provar CI Task in Azure DevOps Pipelines
- Configuring the Automation secrets password in Microsoft Azure Pipelines
- Parallel Execution in Microsoft Azure Pipelines using Multiple build.xml Files
- Parallel Execution in Microsoft Azure Pipelines using Targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- Gearset DevOps CI/CD
- GitHub Actions
- Integrating GitHub Actions CI to Run Automation CI Task
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using Multiple build.xml Files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab Continuous Integration
- Travis CI
- Execution Environment Security Configuration
- Provar Jenkins Plugin
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team Foundation Server
- Version Control
- Salesforce Testing
- Adding a Salesforce Connection
- Assert Page Error Messages on Add/Edit Product
- Internationalization Support
- List and Table Testing
- Salesforce Console Testing
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) Locator Support
- Salesforce Release Updates
- Visualforce Testing
- Best Practices
- Configurations and Permissions
- Error Messages
- Licensing, Installation and Firewalls
- Test Builder and Test Cases
- Release Notes