Automation V3: Test Plans
What are test plans?
Test plans are designed to help users streamline the testing processes associated with supporting new software releases in a structured and easily repeatable manner. Test plans give you the tools you need to specify which (and how) test cases need to be run to test a new release or milestone.
You can build multiple test plans and run a repeatable collection of tests per each release cycle and make global changes to the environment settings such as browser settings, build number and build server. Plus, you can get consolidated reports of your results.
How are test plans different than test cycles?
Test cycles can help you group tests together and execute basic functions such as rerunning failed tests. Test plans provide more advanced configuration options and are designed specifically to help support your CI/CD business objectives.
Why should I use test plans?
Test plans are designed to help teams support an accelerated software release cycle in two primary ways.
- Test plans help you reuse existing test cases and make it easier to find and select existing test cases and add them to your test plan using tags. (This will be part of a later release.)
- Test plans help you make global changes to the test cases that are within each test plan – eliminating the need to open and edit each test case individually.
Using test plans at a glance
Here is a quick overview of the key steps you’ll need to take.
- Planning: Identifying how you want to use test plans to support your business objectives.
- Creating a test plan (this is basically the umbrella where you will add/configure test cases).
- Creating and naming folders to organize individual test cases.
- Configuring your settings.
- Running test plans and reports.
Planning
Most organizations leverage their own unique approach to software development, testing and deployment. That’s why we designed test plans to give you the flexibility you need to create and manage tests to support your organization’s individual business objectives.
Test plans are designed to help testers simplify the process of supporting CI/CD needs – and you can customize the application as needed.
While it’s not required, we highly recommend that you first document how you want to use test plans. By planning your use cases in advance, you can build more meaningful tests that provide actionable insights.
- At what milestones in the development process do you want to test?
- What are the most common testing scenarios?
- What are the most critical workflows that you need to monitor?
- What tests and test components are most often used and why?
- What are the configuration settings you would like to change on a regular basis to test different scenarios?
Configurations: What can I change?
Provar has several built-in configuration settings and it also allows custom settings to be added. The following includes the most commonly used configuration options.
Test environment
This includes the servers, web addresses and endpoints that the tests should be executed against, e.g. DEV, UAT, PROD. Test environments are defined in Provar’s Settings view.
Web browser
Web browsers that tests will be run against are defined in the browser providers tab (inside Provar’s settings view).
- Local desktop browsers can be configured with different screen resolutions. These run on the same machine as Provar.
- Desktop browsers can be provided through Selenium Grid or through cloud-based services such as BrowserStack and SauceLabs.
- Mobile devices can be connected through local Appium servers or through cloud-based services such as Perfecto and TestObject (part of SauceLabs).
- When a web browser is selected, further settings are shown as appropriate. For example, if Desktop: Full Screen is selected, a drop-down is shown offering Chrome, Firefox, etc.
Salesforce user experience (UX)
Identifies whether tests should be run under the Salesforce Classic or Lightning mode.
Custom settings
The test plan editor allows custom settings to be defined.
- These are made available to the test cases as variables when they are executed.
- Can be supplied via environment variables in build scripts.
How can I create a test plan?
Step 1: Creating a test plan.
Test plans are created via the + menu within the Plans section.
Step 2: Open the test plan editor.
Double-clicking a test plan opens the following editor. You can use the test plan editor to make global changes to test cases within your test plan.
Step 3: Adding plan folders.
Plan folders can be added to a test plan via the New tool item in Provar’s main toolbar or via context menus on the test plan. Test folders are optional. You can use these to organize test cases. E.g. You might want to create a folder to organize test cases associated with various parts of your development cycle.
Step 4: Adding tests.
You can add tests by copying the desired tests and pasting them into the test plan or test plan folder. You can also drag and drop test cases as well.
How do I configure test plans?
The test plan editor allows various features to be applied to a test plan.
- The test plan editor allows high-level details to be configured for each feature.
- The plan folder editor allows the settings for each added feature to be overridden where applicable.
How do I run my test plan?
Once the test plan has been set up, the tests can be run in various ways:
- The Run in Provar context menu executes the tests in Provar Desktop’s Test Runner.
- The Run under TrailheadDx context menu executes the tests using Provar’s TrailheadDx integration.
- The Export as ANT Script context menu creates the appropriate ANT build file.
How can I configure output reports?
Provar is able to format test results into various file formats:
- These are stored in the run’s artifacts folder under the name supplied via the Artifact Name field. This makes them accessible to tools like Jenkins.
- They can then be included in Email and Slack notifications using the TestArtifact() function.
CSV
This produces a comma-separated file with the following values:
- Test case path
- Outcome
- Start and end times
Provar can be configured to produce PDF reports for test executions. This is used in several ways:
- A PDF can be produced for an entire run. This report can be stored as an artifact and/or emailed to a list of recipients at the end of the run.
- PDFs can be produced for individual test executions and attached to the execution in the TMT.
Pie chart
Produces a pie chart showing success, failure and skipped counts.
xUnit
Provar can produce an xml file which can be post-processed by tools like JUnit and Allure.
How can I receive test results?
Test results can publish run results through various channels.
The body of the messages can include the following substitutions:
- Test plan settings, e.g. {buildNumber}
- Outcome counters, e.g. <TODO: expand>
- Outputs via the TestArtifacts function, e.g. {TestArtifact(“pieChart.png”)}.
Reporting integration
Provar has built-in integration with many test management tools allowing it to record test executions and their outcomes to these tools. Provar also supports custom reporting which allows it to be extended to support any reporting solution.
Reporting connections
Provar’s test settings tab allows reporting destinations to be configured in its Reporting sub-tab.
Typically only the high-level information needed to connect to the server is stored in the reporting connection:
- This includes information like the server address, username, password and project/solution.
- Secret information such as passwords and API keys are stored in Provar’s .secrets file so they can be encrypted.
Reporting settings
Provar’s plan folder and test instance editors allow reporting settings to be configured for reporting connections that have been defined in the settings tab.
For Micro Focus ALM, for example, the test set is selectable via a drop-down.
Test case definitions
Some test management tools store the test suite and test case definitions alongside the test plan. Micro Focus ALM, Zephyr and TestRail are examples of this.
Provar provides the ability to Download (or import) and Upload (or export) the test case definitions from and to these tools.