Salesforce DX Integration
Provar and Salesforce DX can be used together to generate a scratch org, push the latest Salesforce customizations, and run Provar tests inside the scratch org to get functional testing insights. This allows immediate feedback during a development cycle or when a scratch org is used as a system integration test environment.
As part of our ongoing commitment as the only Salesforce-first test automation tool, we support the integration of Provar with the Salesforce CLI using ProvarDX™. For information about how you can run Provar tests for Salesforce DX scratch orgs using ANT and ConfigureDX, please see this related article.
ProvarDX™ (Provar Developer eXperience) is a free, open-source extension that uses your local Provar installation licenses. ProvarDX™ is designed to integrate Provar seamlessly into your Salesforce developer environments so tests can be executed supporting local development using scratch orgs. This provides a richer set of tools than ConfigureDX, designed purely for customers running Provar tests on scratch orgs under ANT.
ProvarDX™ consists of the following elements
- A Salesforce CLI plugin that allows you to execute Provar test cases from a command line without requiring the installation or use of ANT. ProvarDX™ can be used instead of ANT as part of your CI/CD pipeline to run Provar tests.
- An optional Visual Studio Code Extension that provides access to the most useful ProvarDX™ CLI commands using the Visual Studio Code command autocomplete menu.
- A Provar Desktop right-click option (Export as ProvarDX™) to allow you to export one or more tests (or test plans) as a property file for execution using ProvarDX™.
Both ANT (via ConfigureDX) and ProvarDX™ support using scratch orgs for test execution. However, ProvarDX™ has several advantages, including:
- The ability to override Provar Salesforce connections using the DevHub org alias as the connection name instead of requiring you to use the original Salesforce username as the alias. This subtle change lets you be very clear in your scratch org names and use meaningful aliases (such as Sales User, Service Admin, Metadata Admin, etc.) instead of email@example.com.
- ProvarDX™ works on scratch orgs and sandboxes, allowing you to use a single solution regardless of whether you follow a source-driven, hybrid, or org-driven Salesforce application lifecycle.
- ProvarDX™ does not require you to generate passwords for your scratch orgs. ProvarDX™ does this for you when it needs one.
- With ProvarDX™, you can override property settings on command line execution using parameters.
- ProvarDX™ is consistent with the Salesforce CLI, requiring a single set of skills to maintain and operate. Updates are also synchronized with the Salesforce CLI.
Everything in this guide applies to teams following source-driven development using scratch orgs and those following org-driven development using sandbox orgs.
Provar is designed to help our customers quickly create robust, reusable tests that are intelligent regarding Salesforce org metadata and reusable across multiple user types, environments, and user interfaces.
Provar can uniquely run across multiple environments in a typical software development lifecycle. To align with Salesforce’s source-driven development approach using scratch orgs, and traditional org-driven development, we’ve delivered ProvarDX™ to make this integration seamless for our users.
- Admins and developers use scratch orgs and/or developer sandboxes to make atomic changes, which they want to verify before pushing to continuous integration.
- CI tools hooked into a version control system (e.g., Git) trigger publication and run Provar regression tests to validate the changes. Issues notified to the development and QA team for correction or update of test cases to match changes in behavior.
- CD pipeline invoked as needed to deploy application changes to each environment. Provar regression and smoke test plans executed as appropriate to validate correct deployment package deployment, plus check for legacy issues or primary business scenarios that are working as expected.
As you can see, with ProvarDX™, we’re unlocking those test cases already developed using Provar so that Salesforce administrators and developers can run a suite of tests to verify their changes much earlier in the development process. We’ve chosen to do this using the same tooling that Salesforce provides, Visual Source Code and the Salesforce Command Line Interface (CLI).
Note: This does not eliminate the need for developers to write unit tests for their apex code and Lightning components.
To utilize ProvarDX™, you must first install the Salesforce CLI (following Salesforce’s installation recommendations) in the environment where ProvarDX™ will be executed.
After installing the Salesforce CLI, the ProvarDX™ package can be installed, which extends the Salesforce CLI command set with four simple commands using the Provar namespace:
Runs a specific set of tests using environment overrides just as you would using the ANT build.xml file. Like ANT, we support the same muti-browser and test reporting options. Behavior is controlled by the provardx property file. We recommend having multiple provardx properties to reflect each environment you want to test and the appropriate test plan.
The optional command allows you to quickly validate changes you make to your ProvarDX™ property files rather than wait until you try running runtests to execute them.
Downloads the latest metadata for the named Salesforce connections and stores this locally. This optional command allows you to preprocess this step rather than wait during your runtests command for any changes to download. It is very useful for CI/CD pipelines where you want to execute post-deployment steps before starting a full test run.
Optional command to allow you to precompile any Provar page objects rather than wait until your runtests command is executed. Useful for ensuring your Provar PageObjects, Custom APIs, and PageControls before starting a test run. Typically used in a CI/CD pipeline after pulling the latest Provar project from your version control system.
Each of these commands has several flags and options you can set to control their operation. Each flag has a longform (e.g. –targetusername) and short form (e.g. -u) for convenience. Please see the ProvarDX™ NPM listing to get the up-to-date list of flags and options.
Note: If you’re struggling to remember the correct command syntax, just type
sfdx provar --help to see a list of the options and parameters. You can do this at any level of the command, e.g.
sfdx provar:runtests --help tells you the parameters of the
In addition to the Salesforce CLI extension, we have also made available a Visual Studio Code Extension for those who prefer to use the Visual Studio Code command palette window. These commands can be accessed like any other Visual Studio Code command using either Command + P (Mac OS) or Ctrl + P (Windows).
We currently provide three commands via the Visual Studio Code extension
ProvarDX: Create Properties File
Creates a boilerplate provardx-properties.json file so you have to edit the values you need.
ProvarDX: Run Testcases
sfdx:provar:runtests using a specified provardx property file.
ProvarDX: Validate Properties File
sfdx:provar:validate for the named provardx property file.
We plan to add more commands, so please bookmark this page and watch out for upcoming Provar release notes for announcements on ProvarDX™ enhancements.
Step 1: Using Provar Desktop, first open your Provar project.
Step 1: Within the Navigator, right-click on the test plan, test case folder, or the individual test cases you want to run under ProvarDX™.
Step 3: Select the Export as ProvarDX option.
Step 4: Update the filename and any parameters you want to change. This does not impact your ANT execution settings or start running any tests.
Note: You can either create a file from here or use the clipboard option to make a copy and paste it into your property file in Visual Studio Code as you prefer.
Step 5: Using Visual Studio Code, you can open this property file in your Provar projects’ ProvarDX™ subfolder. If you use a VCS integration, you may want to add this folder to your repository to share it with others.
Creating this file from Provar Desktop is recommended as it prepopulates the required environment settings and parameters and is ready to run. There are some properties we want to briefly focus on here as they are extremely useful:
environmentlets you select a predefined Provar environment. This is important if you want to leverage the QA team’s work to predefine each environment, the screen resolution, the default browser, and, most importantly, a suite of connection overrides.
connectionOverridelets you provide name-value pairs of a Provar Salesforce connection and the username you want to use instead during metadata or
runtestscommand execution. This is currently how you change your tests to run against a new temporary scratch org. Please refer to the Overriding Salesforce connections and tests section for an example of this in more detail.
testCaselets you change the list of tests (including wildcards) that are executed using runtests. This is currently how you can change the tests to be run.
testPlanlets you do the same as testCase but allows you to name one or more Provar Test Plans and is more powerful in test case reuse. This requires your QA team to already have predefined Test Plans that you can use.
Step 6: Validate the property file, especially if you have made any changes. You can do this using the Command Palette option, ProvarDX: Validate Properties File, or from the terminal window type.
sfdx provar:validate -p provardx-properties.json
Note: The provardx-properties.json filename is a default, like build.xml is with ANT. If you do not specify the property file, this special file will be selected by default if it is available in the local directory.
Step 7: You should now see the Property file is valid and displayed in the output.
Whenever you want to execute these tests, you have two options.
From the terminal command line type:
sfdx provar:runtests -p provardx-properties.json
Or, from the Visual Studio Code command palette, select the ProvarDX: Run Testcases option. You will be prompted for your property file; you can accept the default of provardx-properties.json.
Overriding Salesforce connections and tests
We can use the ProvarDX™ property file to pass a user substitution to run this same set of tests against any org with the same or similar configuration. We can do this using the overrideConnections list in the provardx-properties.json file.
Step 1: Check the Salesforce orgs.
Step 2: The example above is a scratch org, (username firstname.lastname@example.org). This reflects configuration changes or developments you may have made to deliver a new feature to your Salesforce users.
If we wanted to run our tests precisely as defined by our Provar project, we could simply use
sfdx provar:runtests -p provardx-properties.json
This is great if you want to run the same tests that your QA team already built in Provar, but it is not recommended for running those tests within your scratch org that you just made changes in.
To run the tests against your scratch org, you need to edit the provardx-properties.json file and add some lines to tell provardx to override the connections in the provardx project with our scratch org user. You can do this using the connectionOverride.
This takes the name of the connection in your Provar project. e.g., Admin, in the case below, reassigns all tests using that connection to use the scratch org instead.
Step 3: Now you can take those tests that you were working on in your admin connection and run them against this scratch org by re-running this command:
sfdx provar:runtests -p provardx-properties.json
This time, you will see the original connection username (email@example.com in my case) replaced with our scratch org user. You can do this for multiple user connections; ensure you have added them to your DevHub first.
Step 4: To change the test cases that are executed, you can edit your provardx-properties to make the following changes:
- Amend the testCase JSON entry to change the test cases to execute. You can use a wildcard character of * to select multiple (e.g. “testCase” : [“/Sales/Acct Management 01.testcase”, “Service/*.testcase”]. This would run all of your tests in the service folder after running the Sales Acct management 01 test case.
- Add a plan entry to execute one or more Provar Test Plans: e.g.
This will execute both the DevSmoketest plan and the additional test cases.
ProvarDX™ is available as a beta feature and is fully supported. However, there are still some known limitations with its usage, and we’re planning further development to optimize the feature and address these issues. We reserve the right to change or remove features as needed to resolve these limitations and extend the product capability.
For a full list of up-to-date commands and parameters, look at the ProvarDX™ npm project listing.
If you’re new to SalesforceDX and the Salesforce CLI in general, take a look at the following resources from Salesforce:
- SalesforceDX Integration using ANT (ConfigureDX)
- Trailhead: Quick Start SalesforceDX
- Trailhead: Developer Tools Installation
- Trailhead: App Development with SalesforceDX
- Trailhead Application Lifecycle Model
- ProvarDX™ npm listing and additional help can be accessed here.
- The ProvarDX™ Visual Studio Code extension can be accessed here
- Provar Automation
- Installing Provar Automation
- Updating Provar Automation
- Using Provar Automation
- API Testing
- Behavior-Driven Development
- Creating and Importing Projects
- Creating Test Cases
- Custom Table Mapping
- Debugging Tests
- Defining a Namespace Prefix on a Connection
- Defining Proxy Settings
- Environment Management
- Exporting Test Cases into a PDF
- Exporting Test Projects
- Override Auto-Retry for Test Step
- Managing Test Steps
- Namespace Org Testing
- Provar Automation
- Provar Test Builder
- Refresh and Recompile
- Reintroduction of CLI License Check
- Reload Org Cache
- Running Tests
- Searching Provar with Find Usages
- Secrets Management and Encryption
- Setup and Teardown Test Cases
- Tags and Service Level Agreements (SLAs)
- Test Cycles
- Test Plans
- Testing Browser Options
- Tooltip Testing
- Using the Test Palette
- Test Palette Introduction
- Control Test Steps
- Generate Test Case
- List Compare
- Page Object Cleaner
- Read Test Step
- String Test Steps
- UI Test Steps
- Using Custom APIs
- Callable Tests
- Data-Driven Testing
- Page Objects
- Block Locator Strategies
- Introduction to XPaths
- Creating an XPath
- Label Locator Strategies
- Maintaining Page Objects
- Mapping Non-Salesforce fields
- Page Object Operations
- Refresh and Reselect Field Locators in Test Builder
- Using Java Method Annotations for Custom Objects
- Applications Testing
- Database Testing
- Document Testing
- Email Testing
- Mobile Testing
- OrchestraCMS Testing
- Salesforce CPQ Testing
- ServiceMax Testing
- Skuid Testing
- Vlocity API Testing
- Webservices Testing
- Provar Manager
- Provar Manager Setup and User Guide
- Installing Provar Manager
- Configuring Provar Manager
- How to Use Provar Manager
- Managing Your Testing Life Cycle
- Provar Manager Test Execution
- Test Executions and Defect Management
- Provar Manager Test Coverage
- How to Integrate Provar Manager
- Setting Up a Connection to Provar Manager
- Object Mapping Between Automation and Manager
- How to Upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases
- Provar Manager Filters
- Uploading Callable Test Cases in Provar Manager
- Uploading Test Steps in Provar Manager
- How to Know if a File in Automation is Linked in Test Manager
- Test Execution Reporting
- Provar Manager Plugins
- Uploading Existing Manual Test Cases to Provar Manager with DataLoader.Io
- Provar Grid
- Introduction to Test Scheduling
- Apache Ant
- Configuration for Sending Emails via the Automation Command Line Interface
- Continuous Integration
- AutoRABIT Salesforce DevOps in Provar Test
- Azure DevOps
- Running a Provar CI Task in Azure DevOps Pipelines
- Configuring the Automation secrets password in Microsoft Azure Pipelines
- Parallel Execution in Microsoft Azure Pipelines using Multiple build.xml Files
- Parallel Execution in Microsoft Azure Pipelines using Targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- Gearset DevOps CI/CD
- GitHub Actions
- Integrating GitHub Actions CI to Run Automation CI Task
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using Multiple build.xml Files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab Continuous Integration
- Travis CI
- Execution Environment Security Configuration
- Provar Jenkins Plugin
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team Foundation Server
- Version Control
- Salesforce Testing
- Adding a Salesforce Connection
- Assert Page Error Messages on Add/Edit Product
- Internationalization Support
- List and Table Testing
- Salesforce Console Testing
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) Locator Support
- Salesforce Release Updates
- Visualforce Testing
- Best Practices
- Configurations and Permissions
- Error Messages
- Licensing, Installation and Firewalls
- Test Builder and Test Cases
- Release Notes