Overview of CircleCI Orbs
CircleCI orbs are open-source and shareable packages of configuration elements, including jobs, commands, and executors. Using orbs makes customizing CircleCI configuration simple.
The key steps to set up the CircleCI Orb integration are as follows –
- Set up the project repository.
- Customize the Automation build.xml file.
- Configure the CircleCI config file.
- How to build a CircleCI Pipeline.
Prerequisites in CircleCI Orbs Git Repository
This support article assumes you’ve already integrated your Automation project with a Git repository, as documented here. The details provided in this support article include which artifacts to commit to your Git repository.
The steps in this support article can be completed using a free CircleCI account. Sign-up is available here, or you can use an existing account.
Note: Currently, the CircleCI can only be used with GitHub or Bitbucket repositories.
The code repositories supported by CircleCI can be found on their integrations page.
Steps to Set up the CircleCI Orb Integration
The detailed steps to set up the CircleCI integration are given below.
In the example, we have used the following tools to create a Continuous Integration (CI) pipeline as in the screenshot given below. You can adapt this to your environment.
Step 1: Set up the project repository.
We will configure the Version Control System (VCS) repository for this step. This public repository contains an example of a CircleCI configuration file and an Automation project. The minimum requirements are as follows (looking at the repository home).
-ProvarProject (containing .testproject, tests folder, and build files) -.settings -ANT -build_circleci.xml // can be named anything, just use this one as an example. -META-INFS -MANIFEST.MF -plans // not required, but our main example uses a build file with a test plan. -Any test plan folders or test plans -tests -Any tests or test folders -templates -Any Test data -src -pageobjects -customapis -.secrets // Make sure your project is encrypted prior to checking in this file. -.testproject -.classpath -build.properties -.project -.circleci -config.yml
Step 2: Customize the Automation build.xml file.
Example build_circleci.xml file (using test plans):
Example build_circleci.xml file (without test plans):
Step 3: Configure the CircleCI config file.
Example of config.yml file:
We suggest that you begin with our build.xml and config.yml files as templates. However, you don’t need to match our example build.xml and config.yml files exactly.
There are some common environment variables and parameters referenced to make things easy. You can modify the following parameters according to your project:
-test_plan: The name of the test plan in the plans/ folder of your Automation project. You can leave this parameter blank if you are not using a test plan. For information on test plans, please refer to this page.
-email_target: The email address of the person to send test results/reports.
-build_file: The name of the build file in the ANT/ folder of your Automation Project you want to execute.
-environment: The name of the test environment you want to override your connections/variables within Automation.
-project_name: The name of the folder in your repository home that contains the Automation project.
-test_folder: The name of the test folder in your project that you want to execute (This can be specific or general as you prefer). If you use a test plan, you can leave this parameter blank.
Make sure your project is checked into the VCS of your choice before you continue.
Step 4: How to build a CircleCI Pipeline.
Please refer to this Orb link to build a CircleCI pipeline. For this article, refer to the run_provar_command example as a base. The steps to develop your CircleCI pipeline to run Automation tests are below.
Step 4a: Set up the CircleCI Project.
Navigate to the CircleCI project page to set up your project. CircleCI will prompt you to log in or create a new account if you haven’t already. Then, select your organization through GitHub or BitBucket to continue. Find the repository that contains your test project (as configured in the above section) and choose Set Up Project.
Step 4b: Build the CircleCI Pipeline.
Select Use Existing Config and click Start Building to initialize the pipeline. If you don’t have a config.yml file in the “.circleci” folder of the root of your repository, you’ll need to add one before you can begin building your pipeline.
Note: This will kick off an initial build immediately if you already have a config.yml file in your repository. Make sure to add the example config.yml listed above if you do not already have it checked in. The build may fail on the first run, but we still have a few more steps to configure in the pipeline.
Step 4c: Add the Environment Variables.
Navigate to Project Settings > Environment Variables. You can add project or job-specific environment variables here. In this case, we created an environment variable for the Automation Secrets password we used to encrypt the project. We can access environment variables created here in the build file of our Automation project. In the example repository above, the build_circleci.xml refers to an environment variable, as given below.
We have created a CircleCI Project Environment Variable named ProvarSecretsPassword. We can set the encryption secrets password here. You can also make other optional environment variables for –
- Test plan
- Project name
- Build file
- Test environment
- Email target
Step 4d: Store the Test results.
The last step, store_test_results, will store the test results in the pipeline details. If your job succeeds, you’ll see the results below.
And if all the tests pass, you will see the results below.
All these files are displayed as hyperlinks and can be opened directly in CircleCI. For example, you can examine the PDF report by clicking on ProvarProject/ANT/Results/Test_Run_Report.pdf.
Additionally, with the CircleCI insights, you can get the historical data on your test suite results, as shown below.
CircleCI Common Build Errors
Please see CircleCI Common Build Errors for troubleshooting if you encounter some build errors.
- Provar Automation
- Installing Provar Automation
- Updating Provar Automation
- Using Provar Automation
- API Testing
- Behavior-Driven Development
- Creating and Importing Projects
- Creating Test Cases
- Custom Table Mapping
- Debugging Tests
- Defining a Namespace Prefix on a Connection
- Defining Proxy Settings
- Environment Management
- Exporting Test Cases into a PDF
- Exporting Test Projects
- Override Auto-Retry for Test Step
- Managing Test Steps
- Namespace Org Testing
- Provar Automation
- Provar Test Builder
- Refresh and Recompile
- Reintroduction of CLI License Check
- Reload Org Cache
- Running Tests
- Searching Provar with Find Usages
- Secrets Management and Encryption
- Setup and Teardown Test Cases
- Tags and Service Level Agreements (SLAs)
- Test Cycles
- Test Plans
- Testing Browser Options
- Tooltip Testing
- Using the Test Palette
- Test Palette Introduction
- Control Test Steps
- Generate Test Case
- List Compare
- Page Object Cleaner
- Read Test Step
- String Test Steps
- UI Test Steps
- Using Custom APIs
- Callable Tests
- Data-Driven Testing
- Page Objects
- Block Locator Strategies
- Introduction to XPaths
- Creating an XPath
- Label Locator Strategies
- Maintaining Page Objects
- Mapping Non-Salesforce fields
- Page Object Operations
- Refresh and Reselect Field Locators in Test Builder
- Using Java Method Annotations for Custom Objects
- Applications Testing
- Database Testing
- Document Testing
- Email Testing
- Mobile Testing
- OrchestraCMS Testing
- Salesforce CPQ Testing
- ServiceMax Testing
- Skuid Testing
- Vlocity API Testing
- Webservices Testing
- Provar Manager
- How to Use Provar Manager
- Provar Manager Setup
- Provar Manager Integrations
- Release Management
- Test Management
- Test Operations
- Provar Manager and Provar Automation
- Setting Up a Connection to Provar Manager
- Object Mapping Between Automation and Manager
- How to Upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases
- Provar Manager Filters
- Uploading Callable Test Cases in Provar Manager
- Uploading Test Steps in Provar Manager
- How to Know if a File in Automation is Linked in Test Manager
- Test Execution Reporting
- Metadata Coverage with Manager
- Provar Grid
- Introduction to Provar DevOps
- Introduction to Test Scheduling
- Apache Ant
- Configuration for Sending Emails via the Automation Command Line Interface
- Continuous Integration
- AutoRABIT Salesforce DevOps in Provar Test
- Azure DevOps
- Running a Provar CI Task in Azure DevOps Pipelines
- Configuring the Automation secrets password in Microsoft Azure Pipelines
- Parallel Execution in Microsoft Azure Pipelines using Multiple build.xml Files
- Parallel Execution in Microsoft Azure Pipelines using Targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- GitHub Actions
- Integrating GitHub Actions CI to Run Automation CI Task
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using Multiple build.xml Files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab Continuous Integration
- Travis CI
- Execution Environment Security Configuration
- Provar Jenkins Plugin
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team Foundation Server
- Version Control
- Masking Provar Credentials on CI
- Salesforce Testing
- Adding a Salesforce Connection
- Assert Page Error Messages on Add/Edit Product
- Internationalization Support
- List and Table Testing
- Salesforce Console Testing
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) Locator Support
- Salesforce Release Updates
- Visualforce Testing
- Best Practices
- Salesforce Connection Best Practices
- Improve Your Metadata Performance
- Testing Best Practices
- Automation Planning
- Supported Testing Phases
- Provar Naming Standards
- Test Case Design
- Create records via API
- Avoid using static values
- Abort Unused Test Sessions/Runs
- Avoid Metadata performance issues
- Increase auto-retry waits for steps using a global variable
- Create different page objects for different pages
- The Best Ways to Change Callable Test Case Locations
- Working with the .testProject file and .secrets file
- Best practices for the .provarCaches folder
- Best practices for .pageObject files
- Configurations and Permissions
- Error Messages
- Provar Manager 3.0 Install Error Resolution
- Provar Manager Test Case Upload Resolution
- Administrator has Blocked Access to Client
- macOS Big Sur Upgrade
- Resolving Failed to Create ChromeDriver Error
- Resolving Jenkins License Missing Error
- Resolving Metadata Timeout Errors
- Test Execution Fails – Firefox Not Installed
- Selenium 4 Upgrade
- Licensing, Installation and Firewalls
- Test Builder and Test Cases
- Release Notes