JavaScript Frameworks

Testomat.io reporter is NodeJS package that can be applied to all popular JavaScript test runners.

CodeceptJS

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Add plugin to codecept conf:

plugins: {
  testomatio: {
    enabled: true,
    require: '@testomatio/reporter/lib/adapter/codecept',
  }
}

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx codeceptjs run

🖼 Screenshots of failed tests and videos (for Playwright helper) will be automatically uploaded as Artifacts

CodeceptJS Parallel Run

If tests run parallel, like workers in CodeceptJS use start-test-run command to get proper reports:

TESTOMATIO={API_KEY} npx start-test-run -c 'npx codeceptjs run-workers 2'

Specify a command to run with -c option in start-test-run

Use --env-file <envfile> option to load environment variables from .env file. Inside env file TESTOMATIO credentials like TESTOMATIO api key or S3 config for artifacts.

Command start-test-run is used to initiate a single run report before all workers are started. Each worker will report to the same Run, and after all workers and codeceptjs finishes, this will finish the run report.

📑 Example Project | 🗄 CodeceptJS API Example | 🥒 CodeceptJS Cucumber Example

📺 Video

Playwright

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Add a reporter to Playwright config:

reporter: [
  ['list'],
  [
    '@testomatio/reporter/lib/adapter/playwright.js',
    {
      apiKey: process.env.TESTOMATIO,
    },
  ],
];

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx playwright test

🖼 Screenshots of failed tests and videos will be automatically uploaded as Artifacts

📑 Example Project | 🥒 Playwright + Cucumber Example

Cypress

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

For Cypress < 10.0.0 (click to expand)

Register `cypress-plugin` in `cypress/plugins/index.js`:

const testomatioReporter = require('@testomatio/reporter/lib/adapter/cypress-plugin');

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  testomatioReporter(on, config);

  return config;
};

For Cypress >= 10.0.0 use setupNodeEvents in cypress.config.js(ts)

setupNodeEvents(on, config) {
  return require('@testomatio/reporter/lib/adapter/cypress-plugin')(on, config)
}

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx cypress run

🖼 Screenshots of failed tests and videos will be automatically uploaded as Artifacts

📑 Example Project | 🥒 Cypress + Cucumber Example

Mocha

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Run the following command from you project folder:

mocha --reporter ./node_modules/@testomatio/reporter/lib/adapter/mocha.js --reporter-options apiKey={API_KEY}

📑 Example Project

Jest

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Add the following line to jest.config.js:

reporters: ['default', ['@testomatio/reporter/lib/adapter/jest.js', { apiKey: process.env.TESTOMATIO }]],

Run tests:

TESTOMATIO={API_KEY} npx jest

Warning Do not use bail option in your jest config or testrun script. (It cause issues with updating testrun status).

📑 Example Project

📺 Video

WebdriverIO

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Add the following lines to wdio.conf.js:

const testomatio = require('@testomatio/reporter/lib/adapter/webdriver');

exports.config = {
  // ...
  reporters: [
    [testomatio, {
      apiKey: $ {
        process.env.TESTOMATIO
      }
    }]
  ]
}

For making screenshots on failed tests add the following hook to wdio.conf.js:

    afterTest: function (test, context, { error, result, duration, passed, retries }) {
        if (error) {
            browser.takeScreenshot()
        }
    },

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx start-test-run -c 'npx wdio wdio.conf.js'

📑 Example Project

📺 Video

Cucumber

📐 When used with Testomat.io Application it is recommended to import feature files via check-cucumber.

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx cucumber-js --format ./node_modules/@testomatio/reporter/lib/adapter/cucumber.js

Note If you use Cucumber with Playwright, Cypress, CodeceptJS, please refer to corresponding framework. This reference is required only if you run tests via cucumber-js CLI.

📑 Example Project

📺 Video

TestCafe

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx testcafe chrome -r testomatio

Newman

To report Newman tests a separate package is required:

npm i newman-reporter-testomatio --save-dev

Note > newman and newman-reporter-testomatio should be installed in the same directory. If you run your tests using globally installed newman (newman run ...), intall newman-reporter-testomatio globally too (npm i newman-reporter-testomatio -g). If you use locally installed newman (within the project) (npx newman run ...), install newman-reporter-testomatio locally (npm i newman-reporter-testomatio). You can verify installed packages via npm list or npm list -g.

Run collection and specify testomatio as reporter:

TESTOMATIO={API_KEY} npx newman run {collection_name.json} -r testomatio

📑 Example Project

Detox

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Run detox tests sptcifying configuration name:

TESTOMATIO={API_KEY} npx detox test -c {configuration_name}

Warning Do not use bail option in your jest config or testrun script. (It cause issues with updating testrun status).

Protractor

📐 When used with Testomat.io Application it is recommended to import automated tests first via check-tests. To create items on the fly set TESTOMATIO_CREATE=1 env variable.

Add the following lines to conf.js:

const JasmineReporter = require('@testomatio/reporter/lib/adapter/jasmine');

exports.config = {
  onPrepare: () => {
    jasmine.getEnv().addReporter(new JasmineReporter({ apiKey: process.env.TESTOMATIO }));
  },
};

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx start-test-run -c 'npx protractor conf.js'

📑 Example Project

Java Frameworks

📐 This section describes reporting into Testomat.io Application

Reporting from Java Frameworks is done via JUnit XML report. Install @testomatio/reporter NodeJS package to process reports:

npm init -y
npm install @testomatio/reporter --save-dev

JUnit XML report can be created by test framework you use:

JUnit

If you run JUnit tests via Maven you can use Surefire Report Plugin to generate JUnit XML report.

For example, with Surefire Report Plugin in Maven, you can add the following configuration to your project's pom.xml:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>3.0.0-M5</version>
        </plugin>
    </plugins>
</build>

In this case JUnit XML will be saved into target/surefire-reports/

So you can import reports to Testomat.io by running:

TESTOMATIO={API_KEY} npx report-xml "target/surefire-reports/**.xml" --java-tests

Note If your tests are located in a folder other other than src/test/java, specify a path to test files using --java-tests option: --java-tests="path/to/tests"

This will import test reports into Testomat.io. If a reporter can access source code of Java tests, the source code will also be imported into Testomat.io

Test IDS

It is possible to attach reported tests to the current tests in Testomat.io by their IDs. Copy Test ID of a test you want to match, and put it as a comment into a Java test you want to import.

In this example, we added ID as a comment to negativeNumbersCanBeSubtracted test:

  @Test
  public void negativeNumbersCanBeSubtracted() throws Exception {
      // @T8acca9eb
      assertThat(calc.Calculate(-1.0, -3.0, "-"), equalTo(2.0));
  }

To make this feature work, please ensure that source code of Java tests is accessible to npx report-xml command, use --java-tests option to specify the correct path. To check if source code of tests is available run reporter with DEBUG mode:

DEBUG=@testomatio/reporter:* TESTOMATIO={API_KEY} npx report-xml "target/surefire-reports/**.xml" --java-tests

Is a source code is not available, test IDs can be set from output. To set Test ID to a test, copy test ID and print it from a test:

System.out.println("tid://" + TID);

For example, if your test id is @T8acca9eb you can print it:

System.out.println("tid://@T8acca9eb");

Artifacts

Screenshots or videos from tests are uploaded if test contains output with a path to file of following format:

file://path/to/screenshot.png

Use System.out.println to print an absulute path to file that should be uploaded as a screenshot.

System.out.println("file://" + pathToScreenshot);

This will produce XML report which contains path to a file:

<testcase>
  <system-out><![CDATA[
    file://path/to/scrrenshot.png
  ]]></system-out>
</testcase>

When XML report is uploaded, all files from file:// will be uploaded to corresponding tests.

🖼 Read more how Artifacts work

Selenide

Please refer to JUnit if you use JUnit as a test runner for Selenide tests. However, it is important to note, that Selenide automatically adds artifacts into JUnit reports printing them as file:// into XML report. This means that no code changes should be made to publish artifacts to Testomat.io.

Cucumber Java

If you use Java version of Cucumber Java you should import your feature files first using check-cucumber. Provide a path to directory containing feature files by using -d option:

TESTOMATIO={API_KEY} npx check-cucumber -d features

It is recommended to set Test IDs for Cucumber scenarios to make reports match imported tests.

TESTOMATIO_TITLE_IDS=1 TESTOMATIO={API_KEY} npx check-cucumber -d features --update-ids

We use --update-ids option to write test IDs obtained from Testomat.io into source code Also we use TESTOMATIO_TITLE_IDS=1 to write test IDs into scenario titles instead of scenario tags. This is important so JUnit report would contain test IDs.

To generate JUnit reports, you can use the built-in Cucumber JUnit plugin. When you run your Cucumber tests the JUnit reports will be generated in the default directory (build/reports/tests/test) in XML format.

To submit report to Testomat.io use npx report-xml command from @testomatio/reporter NodeJS package:

TESTOMATIO={API_KEY} npx report-xml "build/reports/tests/**/*.xml"

If you want to have artifacts attached, use System.out.println to print an absulute path to file that should be uploaded as a screenshot.

System.out.println("file://" + pathToScreenshot);

PHP Frameworks

Taken from PHP Reporter Readme

Installation

composer require testomatio/reporter --dev

Codeception

Get API key from Testomatio application and set it as TESTOMATIO environment variable. Run your tests withTestomatio\Reporter\Codeception extension enabled:

On Linux/MacOS:

TESTOMATIO={apiKey} php vendor/bin/codecept run --ext "Testomatio\Reporter\Codeception"

On Windows

set TESTOMATIO={apiKey}&& php vendor/bin/codecept run  --ext "Testomatio\Reporter\Codeception"

Alternatively, you can add Testomatio\Reporter\Codeception extension to suite or global config. Reporter will be started only when TESTOMATIO environment variable is set:

extensions:
  enabled:
    - Testomatio\Reporter\Codeception

PHPUnit

PHPUnit v10+ is not implemented yet

Add Testomatio\Reporter\PHPUnit listener to phpunit.xml or phpunit.dist.xml

<listeners>
    <listener class="Testomatio\Reporter\PHPUnit" file="vendor/testomatio/php-reporter/src/PHPUnit.php" />
</listeners>

Run tests and provide TESTOMATIO API key as environment variable:

On Linux/MacOS:

TESTOMATIO={apiKey} php vendor/bin/phpunit

On Windows

set TESTOMATIO={apiKey}&& php vendor/bin/phpunit

Python Frameworks

Pytest

Taken from PHP Reporter Readme

uses testomat.io API:

• https://testomatio.github.io/check-tests/
• https://testomatio.github.io/reporter/

Installation

pip install git+https://github.com/tikolakin/pytestomatio.git@1.7#egg=pytestomatio

configuration

Create environment variable TESTOMATIO and set your testomat.io API key. Linux:

export TESTOMATIO=<key>

Windows (cmd):

set TESTOMATIO=<key>

Run groups parameter

There is environment variable TESTOMATIO_RUNGROUP_TITLE that can be used to specify run tests with specific group title.

pytest.ini

In case you are using private testomat.io service, create pytest.ini file in your project root directory. Specify testomat.io url in it

[pytest]
testomatio_url = https://app.testomat.io

Usage

Synchronize tests to testomat.io and get back test id. Tests will not be executed

pytest --testomatio sync

Remove all test ids from source code. Tests will not be executed

pytest --testomatio remove

Run pytest and send test results into testomat.io.
Test can be sent to testomat.io without ids in your test code. If testomat.io failed to match tests by title, it will create new tests for the run

pytest --testomatio report

Run pytest with debug parameter to get test data collected in metadata.json file

pytest --testomatio debug

Submitting Test Run Environment

to configure test environment, you can use additional option:

pytest --testomatio report --testRunEnv "windows11,chrome,1920x1080"

Environment values are comma separated, please use double quotation.

Submitting Test Artifacts

testomat.io does not store any screenshots, logs or other artifacts.

In order to manage them it is advised to use S3 Buckets (GCP Storage). https://docs.testomat.io/usage/test-artifacts/

Analyser needs to be aware of the cloud storage credentials. There are two options:

1. Enable Share credentials with testomat.io Reporter option in testomat.io Settings -> Artifacts.
2. Use environment variables ACCESS_KEY_ID, SECRET_ACCESS_KEY, ENDPOINT, BUCKET

You would need to decide when you want to upload your test artifacts to cloud storage

1. Upload page screenshot when test fails, using fixtures reference

import pytest
from typing import Dict
from pytest import StashKey, CollectReport
from playwright.sync_api import Page

phase_report_key = StashKey[Dict[str, CollectReport]]()

@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    rep = yield
    item.stash.setdefault(phase_report_key, {})[rep.when] = rep
    return rep


@pytest.fixture(scope="function")
def handle_artifacts(page: Page, request):
    yield
    report = request.node.stash[phase_report_key]
    if ("call" not in report) or report["setup"].failed or report["call"].failed:
        random_string = ''.join(random.choices(string.ascii_letters + string.digits, k=8))

        filename = f"{random_string}.png"
        screenshot_path = os.path.join(artifacts_dir, filename)
        page.screenshot(path=screenshot_path)
        # file_path - required, path to file to be uploaded
        # file_bytes - required, bytes of the file to be uploaded
        # key - required, file name in the s3 bucket
        # bucket_name - optional,name of the bucket to upload file to. Default value is taken from testomat.io
        artifact_url = pytest.testomatio.upload_file(screenshot_path, filename)
        # or
        # artifact_url = pytest.testomatio.upload_file_object(file_bytes, key, bucket_name)
        pytest.testomatio.add_artifacts(request.node, [artifact_url])
    page.close()

⚠️ Please take into account s3_connector available only after pytest_collection_modifyitems() hook is executed.

2. If you prefer to use pytest hooks - add pytest_runtest_makereport hook in your conftest.py file.

def pytest_runtest_makereport(item, call):
    artifact_url = pytest.testomatio.upload_file(screenshot_path, filename)
    pytest.testomatio.add_artifacts([artifact_url])

Clarifications

• tests can be synced even without @patest.mark.testomatio('@T96c700e6') decorator.
• test title in testomat.io == test name in pytest
• test suit title in testomat.io == test file name in pytest

Example of test

To make the experience more consistent, it uses standard pytest markers.
testomat.io test id is a string value that starts with @T and has 8 symbols after.

import pytest


@pytest.mark.testomatio('@T96c700e6')
def test_example():
    assert 2 + 2 == 4

Testomatio check-tests

Action	Compatibility	Method
Importing test into testomat.io	complete	pytest --testomatio sync
Exclude hook code of a test	N/A	N/A
Include line number code of a test	N/A	N/A
Import Parametrized Tests	complete	default behaviour
Disable Detached Tests	complete	pytest --testomatio sync --no-detached
Synchronous Import	complete	default behaviour
Auto-assign Test IDs in Source Code	complete	default behaviour
Keep Test IDs Between Projects	complete	pytest --testomatio sync --create
Clean Test IDs	complete	pytest --testomatio remove
Import Into a Branch	N/A	N/A
Keep Structure of Source Code	complete	pytest --testomatio sync --keep-structure
Delete Empty Suites	complete	pytest --testomatio sync --no-empty
Import Into a Specific Folder	complete	pytest --testomatio --directory "Windows\smoke"
Debugging	parity	pytest --testomatio debug