Suites

This topic was updated due to the 2.2.0 MFTF release.

Suites are essentially groups of tests that run in the specific conditions (preconditions and postconditions). They enable you including, excluding, and grouping tests for a customized test run when you need it. You can form suites using separate tests, groups, and modules.

Each suite must be defined in the <magento 2 root>/dev/tests/acceptance/tests/_suite/suite.xml file. The generated tests for each suite go into a separate directory under <magento 2 root>/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated. By default, all generated tests are stored in the default suite under …/Magento/FunctionalTest/_generated/default

If a test is generated into at least one custom suite, it will not appear in the default suite.

Format

The format of a suite:

<?xml version="1.0" encoding="UTF-8"?>

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="">
        <before>
        </before>
        <after>
        </after>
        <include>
            <test name=""/>
            <group name=""/>
            <module name="" file=""/>
        </include>
        <exclude>
            <test name=""/>
            <group name=""/>
            <module name="" file=""/>
        </exclude>
    </suite>
</suites>

Principles

A suite name:
- must not match any existing group value. For example, the suite <suite name="ExampleTest"> will fail during test run if any test contains in annotations <group value="ExampleTest">.
- must not be default or skip. Tests that are not in any suite are generated under the default suite. The suite name skip is synonymous to including a test in the <group value="skip"/>.
- can contain letters, numbers, and underscores.
- should be upper case.
A suite must contain at least one <include>, or one <exclude>, or both.
Using <before> in a suite, you must add the corresponding <after> to restore the initial state of your testing instance.

Conditions

Using suites enables test writers to consolidate conditions that are shared between tests. The code lives in one place and executes once per suite.

Set up preconditions and postconditions using actions in <before> and <after> correspondingly, just similar to use in a test.
Clean up after suites just like after tests. The MFTF enforces the presence of both <before> and <after> if either is present.
Do not reference in the subsequent tests to data that was persisted in the preconditions. Referencing to $stepKey.field$ of these actions is not valid.

Test writing

Since suites enable precondition consolidation, a common workflow for test writing is adding a new test to an existing suite. Such test is generated in context of the suite that contains it. You cannot isolate this test from preconditions of the suite; it cannot be used outside of the suite at the same time.

There are several ways to generate and execute your new test in the context of a suite:

Edit the appropriate suite.xml to include your test only and run:
```
vendor/bin/robo group <suiteName>
```
Temporarily add a group to your test like <group value="foo"> and run:
```
vendor/bin/robo group foo
```
To limit generation to your suite/test combination, run in conjunction with the above:
```
vendor/bin/robo generate:suite <suite>
```
To generate any combination of suites and tests, use generate:tests with the --tests flag.

Examples

Browser actions in suite conditions

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="WYSIWYG">
        <before>
            <actionGroup ref="LoginAsAdmin" stepKey="login"/>
            <amOnPage url="admin/admin/system_config/edit/section/cms/" stepKey="navigateToConfigurationPage" />
            <waitForPageLoad stepKey="wait1"/>
            <conditionalClick stepKey="expandWYSIWYGOptions" selector="" dependentSelector="" visible="true" />
            <waitForElementVisible selector="" stepKey="waitForEnableWYSIWYGDropdown1" />
            <waitForElementVisible  selector="" stepKey="waitForUseSystemValueVisible"/>
            <uncheckOption selector="" stepKey="uncheckUseSystemValue"/>
            <selectOption selector="" userInput="Enabled by Default" stepKey="selectOption1"/>
            <click selector="" stepKey="collapseWYSIWYGOptions" />
            <click selector="" stepKey="saveConfig" />
        </before>
        <after>
            <actionGroup ref="LoginAsAdmin" stepKey="login"/>
            <actionGroup ref="DisabledWYSIWYG" stepKey="disable"/>
        </after>
        <include>
            <group name="WYSIWYG"/>
        </include>
    </suite>
</suites>

This example declares a suite with the name WYSIWYG. The suite requires enabling WYSIWYG before running tests:

Log in to the backend.
Navigate to the Configuration page.
Enable WYSIWYG in the Magento instance.

After the testing, the Magento instance returns to its state prior to the test:

Log back in.
Disable WYSIWYG so that

This suite includes all tests that contain the <group value="WYSIWYG"/> annotation.

Execute Magento CLI commands in suite conditions

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="Cache">
        <before>
            <magentoCLI stepKey="disableCache" command="cache:disable"/>
        </before>
        <after>
            <magentoCLI stepKey="enableCache" command="cache:enable"/>
        </after>
        <include>
            <test name="SomeCacheRelatedTest"/>
            <group name="CacheRelated"/>
        </include>
    </suite>
</suites>

This example declares a suite with the name Cache.

Preconditions:

It disables the Magento instance cache entirely before running the included tests.
After the testing, it re-enables the cache.

The suite includes a specific test SomeCacheRelatedTest and every <test> that includes the <group value="CacheRelated"/> annotation.

Change Magento configurations in suite conditions

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
    <suite name="PaypalConfiguration">
        <before>
            <createData entity="SamplePaypalConfig" stepKey="createSamplePaypalConfig"/>
        </before>
        <after>
            <createData entity="DefaultPayPalConfig" stepKey="restoreDefaultPaypalConfig"/>
        </after>
        <include>
            <module name="Catalog"/>
        </include>
        <exclude>
            <test name="PaypalIncompatibleTest"/>
        </exclude>
    </suite>
</suites>

This example declares a suite with the name PaypalConfiguration:

<before> block persists a Paypal Configuration enabling all tests in this suite to run under the newly reconfigured Magento instance.
<after> block deletes the persisted configuration, returning Magento to its initial state.
The suite includes all tests from the Catalog module, except the PaypalIncompatibleTest test.

Elements reference

suites

The root element for suites.

suite

A set of “before” and “after” preconditions, and test filters to include and exclude tests in the scope of suite.

Attributes	Type	Use	Description
`name`	string	required	Unique suite name identifier.
`remove`	boolean	optional	Removing the suite during merging.

It can contain <before>, <after>, <include>, and <exclude>.

before

A suite hook with preconditions that executes once before the suite tests.

It may contain test steps with any actions and action groups.

Tests in the suite are not run and screenshots are not saved in case of a failure in the before hook. To troubleshoot the failure, run the suite locally.

after

A suite hook with postconditions executed once after the suite tests.

It may contain test steps with any actions and action groups.

include

A set of filters to be used for including tests into the suite.

It may contain filters by:

test which names a specific <test>.
group which refers to a declared group annotation.
module which refers to test files under a specific Magento Module.

The element may contain <test>, <group>, and <module>.

exclude

A set of filters to be used for excluding tests from the the suite.

There two types of behaviour:

Applying filters to the included tests when the suite contains <include> filters. The MFTF will exclude tests from the previously included set and generate the remaining tests in the suite.
Applying filter to all tests when the suite does not contain <include> filters. The MFTF will generate all existing tests except the excluded. In this case, the custom suite will contain all generated tests except excluded, and the default suite will contain the excluded tests only.