CircleCI

Overview

The official Endtest CircleCI orb allows you to run Endtest web and mobile tests directly from a CircleCI workflow.

The orb can:

Start one or more Endtest test executions.
Wait for all executions to finish.
Display the result of each execution in CircleCI.
Display aggregate results when multiple test suites are triggered.
Fail the CircleCI job when any execution reports failed assertions or errors.
Block a deployment when an important end-to-end test fails.

You can view the orb in the CircleCI Orb Registry.

The source code is available in the Endtest CircleCI Orb repository.

Requirements

Before using the orb, you need:

An Endtest account.
Your Endtest App ID.
Your Endtest App Code.
An Endtest API Request for starting the desired test execution.
A CircleCI project connected to your source-code repository.

Your App ID and App Code can be found on the Endtest Settings page.

You can generate an API Request by following the instructions from the Endtest API chapter.

The API Request added to the CircleCI configuration must not contain the appId or appCode parameters. The orb adds those values securely at runtime.

Add the environment variables

Open your project in CircleCI and navigate to:

Project Settings → Environment Variables

Add these environment variables:

ENDTEST_APP_ID
ENDTEST_APP_CODE

Their values should be:

ENDTEST_APP_ID: The App ID for your Endtest account.
ENDTEST_APP_CODE: The App Code for your Endtest account.

CircleCI hides the values of environment variables after they are saved.

The Endtest API Request itself is added visibly to .circleci/config.yml, without credentials. This allows team members to see which test suite, label, browser, operating system, resolution, and other settings are being used.

Add the Endtest orb

Create or update the .circleci/config.yml file in your repository:

version: 2.1

orbs:
  endtest: endtest/endtest@0.0.7

workflows:
  endtest-tests:
    jobs:
      - endtest/run:
          api_request: 'https://app.endtest.io/api.php?action=runWeb&suite=YOUR_SUITE_ID&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes='
          number_of_loops: 10

Replace YOUR_SUITE_ID with the ID of the Endtest test suite you want to execute.

Do not add appId or appCode to the API Request. The orb reads them from the CircleCI environment variables and adds them securely.

Always check the Endtest page in the CircleCI Orb Registry for the latest published version.

Trigger multiple test suites

An Endtest API Request can trigger multiple test suites by using a label instead of a suite ID.

For example:

version: 2.1

orbs:
  endtest: endtest/endtest@0.0.7

workflows:
  critical-tests:
    jobs:
      - endtest/run:
          api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes='
          number_of_loops: 10

This request starts every test suite associated with the Critical label.

The Endtest API may return multiple execution hashes:

EXECUTION_HASH_1,EXECUTION_HASH_2

The orb automatically:

Detects every execution hash.
Waits for all test executions to finish.
Displays the result of each test suite separately.
Calculates aggregate totals.
Fails the CircleCI job if any execution reports failures or errors.

No additional CircleCI configuration is required for multiple test executions.

Using the default environment-variable names

The default credential environment-variable names are:

ENDTEST_APP_ID
ENDTEST_APP_CODE

When you use these names, you do not need to pass the app_id or app_code parameters:

version: 2.1

orbs:
  endtest: endtest/endtest@0.0.7

workflows:
  endtest-tests:
    jobs:
      - endtest/run:
          api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes='
          number_of_loops: 10

Using custom credential variable names

You can use different names for the credential environment variables:

version: 2.1

orbs:
  endtest: endtest/endtest@0.0.7

workflows:
  endtest-tests:
    jobs:
      - endtest/run:
          app_id: MY_ENDTEST_APP_ID
          app_code: MY_ENDTEST_APP_CODE
          api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes='
          number_of_loops: 10

In that case, add these variables to the CircleCI project:

MY_ENDTEST_APP_ID
MY_ENDTEST_APP_CODE

The values passed to app_id and app_code are the names of CircleCI environment variables, not the actual credentials.

Legacy environment-variable configuration

Older versions of the Endtest orb expected the entire API Request to be stored in an environment variable named:

ENDTEST_API_REQUEST

Version 0.0.7 continues to support this configuration for compatibility.

When the visible api_request parameter is not provided, the orb reads the complete request from ENDTEST_API_REQUEST.

The recommended configuration is to keep only the App ID and App Code in environment variables and place the API Request without credentials in .circleci/config.yml.

Parameters

Parameter	Type	Default	Description
`app_id`	Environment-variable name	`ENDTEST_APP_ID`	The variable containing your Endtest App ID.
`app_code`	Environment-variable name	`ENDTEST_APP_CODE`	The variable containing your Endtest App Code.
`api_request`	String	Empty	A visible Endtest API Request without `appId` or `appCode`.
`api_request_env_var`	Environment-variable name	`ENDTEST_API_REQUEST`	Legacy fallback variable containing the complete API Request.
`number_of_loops`	Integer	`10`	Maximum number of result checks. Each check occurs after a 30-second wait.

The api_request parameter takes priority over api_request_env_var.

For security, the visible API Request must use:

https://app.endtest.io/api.php

The orb rejects visible requests that contain appId or appCode.

Choosing the number of loops

The orb checks the Endtest execution once every 30 seconds.

For example, when an execution normally takes approximately three minutes, you can use:

number_of_loops: 7

This allows up to 210 seconds:

7 × 30 seconds = 210 seconds

Choose a value slightly higher than the normal duration of the test execution.

When a request triggers multiple test suites, set the value according to the longest-running execution.

For tests that normally finish within five minutes, you can use:

number_of_loops: 10

Pipeline behavior

The CircleCI job passes when every completed Endtest execution reports:

Failed: 0
Errors: 0

The job fails when:

Any execution reports one or more failed assertions.
Any execution reports one or more errors.
An execution enters an error state.
The API returns an invalid response.
The number of returned results does not match the number of started executions.
The executions do not finish within the configured polling period.

Use Endtest as a deployment gate

The endtest/run job can be used as a deployment gate:

version: 2.1

orbs:
  endtest: endtest/endtest@0.0.7

jobs:
  deploy:
    docker:
      - image: cimg/base:stable
    steps:
      - run:
          name: Deploy application
          command: ./deploy.sh

workflows:
  test-and-deploy:
    jobs:
      - endtest/run:
          name: endtest-tests
          api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes='
          number_of_loops: 10

      - deploy:
          requires:
            - endtest-tests

The deploy job only starts when all Endtest executions pass.

Results displayed in CircleCI

For every Endtest execution, the orb displays:

Test suite name.
Test configuration.
Number of test cases.
Number of passed assertions.
Number of failed assertions.
Number of errors.
Start time.
End time.
Execution hash.
Link to the Endtest Results page.

When multiple test suites are triggered, the orb also displays an aggregate summary:

Endtest summary
Executions: 2
Test Cases: 4
Passed: 12
Failed: 0
Errors: 0

The Results page for an execution has the following format:

https://app.endtest.io/results?hash=EXECUTION_HASH

Manual API integration

The official orb is the recommended way to integrate Endtest with CircleCI.

For custom workflows that cannot use an orb, you can call the Endtest API directly from a shell script. The script must:

Send the API Request that starts one or more test executions.
Save the execution hash or comma-separated execution hashes returned by Endtest.
Call the getResults API endpoint using the App ID, App Code, and execution hashes.
Continue checking until every execution finishes.
Return a non-zero exit code when any execution fails.

More information about direct API usage is available in the Endtest API chapter.