Using Provar with Amazon Web Services (AWS) Device Farm

If you develop new mobile-optimized applications, you know that the testing process throughout the development lifecycle is exponentially more challenging, given the wide range of mobile devices that your end-users may be operating to access your app.

That’s why more leading organizations are turning to the Amazon Web Services Device Farm to help streamline the testing process by helping you test your application on a shared fleet of 2500+ devices remotely in the cloud. (We get it. The cube next to you probably isn’t big enough to charge that many phones.)

With Provar, you can create a wide range of testing scenarios and rapidly reapply those tests by integrating those tests with the Amazon Web Services Device Farm. (Side note, we also support testing via several remote web and mobile testing platforms such as Sauce Labs, Perfecto, and BrowserStack.)

If you’d like to use Provar in conjunction with the AWS Device Farm, you can use the following information to run Provar test cases on various Android devices (that use a Chrome browser) within the AWS Device Farm. You can also check out the AWS Device Farm documentation on the Salesforce website to learn more.

Prerequisites in Amazon Web Service (a.k.a. Things you need)

An AWS console account.
A Provar ANT zip file.
A Provar project with at least one test case.

Setting up your Folder Structure to Execute Test Cases in the Device Farm

Since the AWS Device Farm is designed to conceal the web driver, you will need to upload the project in an AWS Device Farm in a zip format.

For that, you will need to create a folder structure using the steps below:

(1) Create a folder (e.g. ProvarProject).

(2) Add the project folder, the Provar_ANT folder, and the license file in the ProvarProject folder. Rename the license file with the name ProvarLicense.properties.

Setting up your Folder Structure to Execute Test Cases in the Device Farm

(3) In the project folder, there are test cases in the tests folder, build.xml, ANT folder, and the .testproject file containing settings for Appium.

.testproject file containing settings for Appium is located in the Provar project folder

(4) Using ANT build.xml, execute your test cases.

Using ANT build.xml, execute your test cases in Provar for Amazon Web Service Device Farm

(5) The Provar_ANT folder contains all of your required jar files.

Provar_ANT folder contains all of your required jar files.

(6) Create a zip file of ProvarProject.

screenshot of ProvarProject zip file in Provar test builder

Configuring Your Settings

You can configure the settings in a couple of different ways. You can either use the Provar UI or update the build.xml file and the .testproject file.

(1) In the build.xml, a few changes are required.

Change provar.home where the Provar_ANT directory is available.

<property name="provar.home" value="../../Provar_ANT"/>

Change ProviderName (same as mentioned in .testproject) and DeviceName (any name) in the Run-Test-Case tag.

webBrowserProviderName="SampleProviderForDevice"
webBrowserDeviceName="Google Pixel 2"

(2) In the .testproject file, select the browserProvider (same as ProviderName in build.xml). The values mentioned in PINK are mandatory fields and should have the same value as shown below.

<browserProvider name="SampleProviderForDevice" type="Appium">
                  <devices>
         <browserProviderDevice name="Google Pixel 2" platform="ANDROID">
                         <properties>
  <browserPlatformProperty name="deviceName" type="DesiredCapability" value="Google Pixel 2"/>
          		<browserPlatformProperty name="platformName" type="DesiredCapability" value="ANDROID"/>
          		<browserPlatformProperty name="platformVersion" type="DesiredCapability" value="9"/>
          		<browserPlatformProperty name="udid" type="DesiredCapability" value="deviceUDID"/>
                        </properties>
                   </browserProviderDevice>
                </devices>
                       <properties>
      		<browserPlatformProperty name="gridUrl" value="http://127.0.0.1:4723/wd/hub"/>
      		<browserPlatformProperty name="newCommandTimeout" type="DesiredCapability" value="100"/>
    	          </properties>
          </browserProvider>

Note: deviceUDID parameter will be replaced by the actual device UDID at run time.

How to Execute Test Cases

(1) Sign in to your AWS console and navigate the AWS Device Farm.

(2) Create a project in the AWS Device Farm console.

screenshot of AWS Device Farm console

(3) In automated tests, select Create a new run button.

screenshot on how to create a new run in AWS Device Farm console

(4) Choose the web application, provide a name, and click Next Step.

screenshot of testing new run button in AWS Device Farm console

Choose ProvarTest in a new created run in AWS Device Farm console

(5) In Configure Steps, select the Appium Node.js test type.

how to configure Appium Node.js in Amazon device farm console

(6) Click on the Upload button and upload the ProvarProject.zip file.

screenshot on how to upload ProvarProject.zip in Amazon Device Farm Console

(7) Once the project is uploaded, specify the spec file. See the spec file below.

version: 0.1

# Phases are collection of commands that get executed on Device Farm.

phases:

  # The install phase includes commands that install dependencies that your tests use.

  # Default dependencies for testing frameworks supported on Device Farm are already installed.

  install:

    commands:

      # By default, Appium server version used is 1.7.2.

      # You can switch to an alternate supported version from 1.6.5, 1.7.1, 1.7.2, 1.8.0 , 1.8.1, 1.9.1 by using a command like "avm 1.7.1"

      # OR

      # To install a newer version of Appium use the following commands:

      - export APPIUM_VERSION=1.9.1

      - avm $APPIUM_VERSION

      - ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium  /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js   

      - java -version

      - ant -version

      - export PROVAR_HOME=$DEVICEFARM_TEST_PACKAGE_PATH

      - cd $DEVICEFARM_TEST_PACKAGE_PATH

      - mkdir -p /home/device-farm/Provar/.licenses

      - mv $DEVICEFARM_TEST_PACKAGE_PATH/ProvarProject/ProvarLicense.properties /home/device-farm/Provar/.licenses    

  # The pre-test phase includes commands that setup your test environment.

  pre_test:

    commands:

      # We recommend starting appium server process in the background using the command below.

      # Appium server log will go to $DEVICEFARM_LOG_DIR directory.

      # The environment variables below will be auto-populated during run time.

      - echo "Start appium server"

      - >-

        appium --log-timestamp --browser-name Chrome --device-name $DEVICEFARM_DEVICE_NAME --platform-name $DEVICEFARM_DEVICE_PLATFORM_NAME --udid $DEVICEFARM_DEVICE_UDID --chromedriver-executable $DEVICEFARM_CHROMEDRIVER_EXECUTABLE  >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &        

      - >-

        start_appium_timeout=0;

        while [ true ];

        do

            if [ $start_appium_timeout -gt 60 ];

            then

                echo "appium server never started in 60 seconds. Exiting";

                exit 1;

            fi;

            grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1;

            if [ $? -eq 0 ];

            then

                echo "Appium REST http interface listener started on 0.0.0.0:4723";

                break;

            else

                echo "Waiting for appium server to start. Sleeping for 1 second";

                sleep 1;

                start_appium_timeout=$((start_appium_timeout+1));

            fi;

        done; 



  # The test phase includes commands that run your test suite execution.

  test:

    commands:

      # Go into the root folder containing your source code

      - echo "Navigate to test source code"

      - cd $DEVICEFARM_TEST_PACKAGE_PATH

     

      - echo "Start Provar/Appium test"

      - sed -i "s/deviceUDID/$DEVICEFARM_DEVICE_UDID/g" ProvarProject/Project/.testproject

      # Enter the command below to start the tests . The comamnd should be similar to what you use to run the tests locally.

      - cd ProvarProject/Project/ANT

      - ant

 # The post test phase includes are commands that are run after your tests are executed.

  post_test:

    commands:

# The artifacts phase lets you specify the location where your tests logs, device logs will be stored.

# And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm.

# These logs and artifacts will be available through ListArtifacts API in Device Farm.

artifacts:

  # By default, Device Farm will collect your artifacts from following directories

  - $DEVICEFARM_LOG_DIR

  - $DEVICEFARM_TEST_PACKAGE_PATH/ProvarProject/Project/ANT/

8) Create a new device pool and add devices to it.

Creating a new device pool and add devices in Amazon Web Service Device Farm

(9) After selecting Device Pool, click the Next Step button.

selective a device under Provar to test compatibilty

(10) Click the Confirm and Start Run button.

screenshot of confirming and testing in Amazon Web Service Device Farm

(11) Check the status of the test execution using the links shown below.

screenshot of Checking the status of the test execution in Amazon Web Service Device Farm

screenshot of output after testing a run

Getting those Reports You Need

The AWS Device Farm provides many options (like Video, Test Specification log, Files, and Screenshots) to verify your test execution status. And it stores your reports for up to 400 days. These reports include metadata, logs, screenshots, and performance data.

screenshot of the AWS Device Farm with many options for reporting