Automation V3: Running a Provar CI Task in Azure DevOps Pipelines
This guide will walk you through setting up a Microsoft Azure DevOps Pipeline that can be run on a scheduled basis (for daily regression) or as a continuous Integration process when changes are made to your Version Control System (VCS).
Similar guides have been available for Jenkins, TFS, and CircleCI; see the Further Reading section for more information.
Prerequisites in Azure DevOps Pipelines
The following steps can be completed using a free Azure DevOps Pipelines account. Signup is available here, or you can use an existing Microsoft account. This guide assumes you’ve already integrated your Provar project with a git repository. Details are provided below regarding which artifacts to commit to your VCS repository.
Overview
You can adapt this for your environment by hosting your VCS repository within Azure DevOps Pipelines, which may provide a more streamlined solution than the one listed below. The key steps include: setting up your project repository, customizing your Provar build.xml file, and configuring the ANT task. We’ll cover the rest of these steps below in more detail.
VCS Repository Configuration
We’ve made public a working project that you can use on GitHub. It is pre-configured for Azure DevOps.: https://github.com/rclark-provar/cidemo.
You may wish to test the configuration with this project and create your branch before you attempt to configure your project. The key project structure is as follows. You can have more files though this is the minimum required:
Repo root |_ DemoProject |_ Results |_ .secrets |_ src |_ pageobjects |_ tests |_ <files & folders> |_ .testproject |_ build.xml |_ Provar |_ .licenses // Provar Project you wish to test, using our Git plugin// You can move this, but note the path for test results // Do not store your .secrets file in a public repository! // Pageobjects locator source files // Parent folder for running all tests if required // Test cases you wish to execute // Provar project configuration file // ANT build.xml file to be executed in the pipeline // Folder for your license files, you can have multiple// Ensure your license file isn’t public // From our Provar ANT Download // From our Provar ANT Download |_ ProvarHome |_ ant |_ lib |_ urlicense.properties
For instructions on integrating your Provar project into Git, please refer to our support articles about Git.
Build.xml Configuration
Provar helps you create a default build.xml which is excellent for running locally, but you will most likely need to adapt this for your CI server environment. Below is an example from the demo project with some values called out for your attention.
<project default="runtests">
<property environment="env"/>
<property name="provar.home" value="${env.PROVAR_HOME}"/>
<property name="testproject.home" value="DemoProject"/>
<property name="testproject.results" value="DemoProject/Results"/>
<property name="license.path" value="../Provar/.license"/>
<taskdef name="Provar-Compile"
classname="com.provar.testrunner.ant.CompileTask"
classpath="${provar.home}/ant/ant-provar.jar"/>
<taskdef name="Run-Test-Case"
classname="com.provar.testrunner.ant.RunnerTask"
classpath="${provar.home}/ant/ant-
provar.jar;${provar.home}/ant/ant-provar-bundled.jar;${provar.home}/ant/ant-
provar-sf.jar"/>
<taskdef name="Test-Cycle-Report"
classname="com.provar.testrunner.ant.TestCycleReportTask"
classpath="${provar.home}/ant/ant-
provar.jar;${provar.home}/ant/ant-provar-bundled.jar;${provar.home}/ant/ant-
provar-project.jar;${provar.home}/ant/antprovar-sf.jar"/>
<target name="runtestsVSTS">
<Provar-Compile
provarHome="${provar.home}"
projectPath="${testproject.home}"/>
<Run-Test-Case
provarHome="${provar.home}"
projectPath="${testproject.home}"
resultsPath="${testproject.results}"
resultsPathDisposition="Replace"
testEnvironment="UAT"
webBrowser="Chrome"
webBrowserConfiguration="Full Screen"
excludeCallableTestCases="true"
salesforceMetadataCache="Reload"
projectCachePath=".provarCaches"
testOutputlevel="WARNING"
licensePath="${license.path}"
pluginOutputlevel="WARNING"
stopTestRunOnError="false"
invokeTestRunMonitor="true">
<fileset dir="../DemoProject/tests">
<include name="Test Case 1.testcase"/>
</fileset>
</Run-Test-Case>
</target>
Azure DevOps Pipelines
Follow the steps below to set up your Azure DevOps Pipelines. You may want to create separate pipelines for different environments. You can also parameterize the pipeline to control the different environments and VCS branches from which you want to run tests.
Step 1: Create a new project. Complete the details as per your specific requirements.
Step 2: Import the project in Azure Git (optional).
If you don’t have a repository, you can let Azure DevOps create one. You’ll need to go back and configure this in Provar and commit your test cases, build.xml, and Provar ANT files as specified in the previous section.
Also, you can import your repository from any other GIT or TFS repository.
Step 3: Create a new pipeline.
If you don’t want to push your project into Azure git (mentioned in Step 2), you can directly select the Pipelines option and click on Create Pipeline.
Step 4: Select the type of pipeline.
Azure DevOps provides us with two options here. First, create a pipeline using a YAML file. Second, use the classic Editor (without YAML). As many options are available, you can choose any of them according to your requirements.
Step 5: Using the Classic Editor.
Click on the link. Use the classic editor, and you will be redirected to select your repository source.
Step 6: Select the repository source.
We’ve chosen to use a GitHub repo. You can make whatever selection fits your environment and the tools available. Note that the choice here may affect the screens afterward, but the necessary items for any Git host are the repository to be used and the default branch.
Step 7: Select the repository and default branch.
Clicking Save and Queue will control the branch used to build and run your tests. You will probably be following a VCS and DevOps strategy where Master is not the branch you build from.
Step 8: Choose the task template.
There are numerous options for how to run Provar test executions. The simplest option is to use the ANT task, which mimics how you can test your regression or CI targets locally in Provar Desktop.
Step 9: Configure the ANT task.
It’s essential to use a build agent that includes browser support. VS2017-win2016 is what we have used, or you can build your custom agent.
Step 10: Configure pipeline source.
Check that the Repository and Branch are correct. You may have to reselect these. You may have multiple tasks in your pipeline for which you access different branches, such as Dev, UAT, Staging, etc.
Step 11: Download Provar binaries.
If provar binaries are not checked in in the GIT repository, you need to download them before executing the ANT task.
Since we have used a windows agent, we can include a Powershell Script task and download Provar binaries from the Provar_ANT.zip file and extract them in the ProvarHome directory.
Commands are mentioned below.
$webclient = New-Object System.Net.WebClient $url = "https://download.provartesting.com/latest/Provar_ANT_latest.zip" $file = "Provar_ANT_latest.zip" $webclient.DownloadFile($url,$file) Expand-Archive -LiteralPath Provar_ANT_latest.zip -DestinationPath ProvarHome
Step 12: Configure the ANT task.
These values will be specific to your Provar project. Integrating your Junit results will allow you to graph the test result history.
To use JDK 8, go to advanced settings and select JDK 8 in the JDK version dropdown.
Step 13: Configure environment variables.
Click the Variables sub-tab and add an entry for PROVAR_HOME based on your VCS repository structure. This is a good place for other environment variables you may want to edit frequently, such as which ANT target to execute, the environment, or the test path.
If you’re using a public repository and have authenticated connections, we recommend storing your connection usernames and passwords as either pipelines or release variables. Provar looks for two exceptional (case-sensitive) environment variables when it cannot find a .secrets file for a connection.
PROVAR_sf_<connection-name>
PROVAR_sf_<connection-name>_password
In the example below, a Salesforce connection called DemoAdmin has been secured outside of using Github.
Use the lock to secure the credential, so it’s not in plain text. The environment variable must be named. This password-masking demo video shows how we do this for Jenkins.
Step 14: Save and queue to test your pipeline.
Before we trigger the pipeline automatically, troubleshoot any issues with the setup manually.
Step 15: If all goes well…
You should get a successful job result. Likely, you won’t on the first attempt, so click into the error if you get one and review the issue. The most common causes are incorrect license file location or type, inconsistent settings in the build.xml target, or missing files from the VCS repository.
You can also view your test results history under the test tab and drill down into the test results to a high level.
Step 16: Provar test results in Azure DevOps.
To get Provar test results (PDF or HTML files), you must add a task – Publish Artifacts. This will require you to provide the path of the results directory and the artifact name. We have used some predefined parameters available in Azure DevOps. See more information in this link for Use Predefined Variables.
Step 17: Caching in Azure DevOps.
To reduce the execution time between the jobs, we can include a caching task in the job. Two parameters are required in caching tasks – Key and Path. Key can be any value, and Path would be the relative path of the .provarCaches folder defined in build.xml.
Values in build.xml should be:
| salesforceMetadataCache=”Reuse” projectCachePath=”.provarCaches” |
Configure CI Job
To enable continuous integration, edit your pipeline job and select the Triggers tab. You can check the Continuous Integration flag to make the job run whenever your VCS branch is updated. We also recommend checking the Batch changes option if you want to optimize your pipeline so that it’s not running for each separate commit.
You can also use the Pull Request option to invoke your CI process when a Pull Request is made. This is useful for validating when a developer has changes pending for review, and you want to run a set of Provar unit or smoke tests.
Schedule the Regression Job
In addition to running a CI process, it’s usually best practice to configure a nightly build to execute a full regression pack. It would be best if you did this on a separate pipeline to the CI job, as you’ll most likely execute a test pack that takes hours instead of minutes. You will also almost certainly wish to execute this on a different branch of your repository to the CI job.
Executing Provar Tests using YAML file on Linux
If you try configuring a job using a YAML file, the steps will differ slightly, but the concepts remain the same.
Below is a sample YAML file that includes the following commands:
Step 1: Trigger.
Continuous integration (CI) triggers a pipeline to run whenever you push an update to the specified branches or push selected tags.
Step 2: Pool.
You need at least one agent to build your code or deploy your software using Azure Pipelines. Azure Pipelines provides a predefined agent pool named Azure Pipelines with Microsoft-hosted agents.
Step 3: Cache.
Pipeline caching can help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost of recreating or redownloading the duplicate files again.
- Caching .provarCaches: The execution time required to download metadata on every run can be reduced by caching it.
- Caching ProvarHome: Cachiof the Provar Home directory is done in this step.
Step 4: Bash.
The bash script downloads Provar Binaries if unavailable through the cache.
Step 5: ANT Task Execution.
Executing the Ant task using xvfb-run command using the bash task. This task will execute provar test cases and generate test results.
Step 6: Publish your test results.
We must explicitly use the publish test results task to publish JUNIT test results in GUI.
Step 7: Publish artifacts.
To get Provar Test Results (PDF or HTML files), you must add a task, Publish Artifacts.
Note: To execute on the Linux environment, we must change the Browser to Chrome_Headless in the build.xml.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Cache@2
inputs:
key: 'provarcache'
path: '$(System.DefaultWorkingDirectory)/.provarCaches'
- task: Cache@2
inputs:
key: 'provarhome'
path: '$(System.DefaultWorkingDirectory)/Provar_Home'
- task: Bash@3
inputs:
targetType: 'inline'
script: |
if [ ! -d Provar_Home ]; then
curl -O https://download.provartesting.com/latest/Provar_ANT_latest.zip
unzip Provar_ANT_latest.zip -d ProvarHome;
else
echo "Provar_Home is already available.";
fi
- task: Bash@3
inputs:
targetType: 'inline'
script: 'xvfb-run ant -Dprovar.home=$(PROVARHOME) -f DemoProjects/build.xml'
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'DemoProjects/Results/*.xml'
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: 'DemoProjects/ANT/Results'
ArtifactName: 'ResultsDirectory_$(Build.BuildNumber)'
Troubleshooting
After configuring the steps above, your pipeline process should work 100%. If you’ve deviated from the steps above, please check carefully that the changes you have made are consistent and in sync (for example, the location of the build.xml, Provar Home configuration, and the Provar license files and path).
The following is a summary of common symptoms and how to correct them:
Further Reading
The following articles may also be of interest:
Introduction to testing scheduling
Introduction to Git integration