E2E Testing with Cucumber

---

While Angular CLI will set up your E2E testing to use Jasmine, the Cucumber testing framework can also be used for end-to-end testing. The Jasmine framework allows flexibility in how the tests are named, simply as strings in the describe() and it() methods. Using the Gherkin language and Cucumber allows a bit more structure to the way the feature definition files are written.

Cucumber consists of two primary file types. The feature file describes the tests in a descriptive language and can generally be written by business analysts. This allows a non-developer to describe what the code must do. Listing 43 shows a simple example feature file.

Code Listing 43: Feature file example

In addition to the feature files are the step definition files. These files consist of a pattern (that matches one or more entries in the feature file) and the snippet of code necessary to test that feature. These are typically written by the developer to translate the business requirements defined in the feature file into test conditions and results.

This combination creates a living spec document. If the analyst changes the feature file, the tests should break, which means the underlying code needs to be reviewed.

Feature files

Feature files are specification files for defining acceptance tests. They are based on the Gherkin language, which is a simple language designed to be created and read by business analysts. Due to its simple syntax, the Cucumber testing framework can read this language to run the acceptance tests.

The general format of the language is a keyword, followed by some text, such as:

Given I enter a bad password

Then I see a bad credentials message in the error window

Some keywords are primarily for documentation purposes, describing what is being tested in the file. The Given, When, and Then keywords typically perform actions and test for results. The Gherkin keywords are described in this section.

Feature

The feature is a high-level overview of what is being tested, such as the login page or a shopping cart. The feature description begins with the keyword Feature:. All feature files must begin with the keyword feature, followed by a colon.

After the keyword is free-form text, which ends when a keyword is found. Typically, the feature can be a story or some other description of what this file will test. The file itself will have multiple examples detailing what should be tested to confirm this feature works.

A user story typically has the format:

As a <role>, I want <some goal> so that <some reason>

A user story is typically small enough that a team can complete it within a single sprint. Another story format is the job story, which has three parts, but a different view:

When <situation>, I want to <some task> so I can <expected outcome>

Job stories focus on something occurring without a specific role, while user stories focus on the person and why they want to perform the task. With either story approach, or even just descriptive text, you should keep the content consistent so anyone opening the feature file can quickly tell exactly what should be tested.

The feature being tested will have multiple examples or scenarios that, taken as a whole, should completely test the feature.

Example

The example (or scenario) keyword is one small test of the feature file. These are the actual tests that will be run, and should handle both positive results (expected behavior) and error conditions (such as bad data being entered). In general, the test describes a starting point (application page, for example), action taken (entering a good password), and expected result. By using the Gherkin syntax, a feature file and examples should be readable, so business analysts can write what they expect the example to test out. The scenario will consist of the Gherkin keywords to describe preconditions (Given), actions taken (When), and expected outcomes (Then).

Given

This command specifies a precondition that needs to occur before the scenario actions are taken. For example, we might start the test with a statement such as:

Given “I am on the splash screen”

An entry in the step’s definition will translate this into the necessary browser navigation to get to the application’s initial screen.

When

This command represents the action a user might take, such as clicking a button or entering text. Our example in Code Listing 43 combines Given (precondition) and When (action taken) to define what the user is doing. Although with end-to-end testing, the system will perform the steps, a quality assurance resource could manually read the feature file and perform the test steps manually to confirm the feature works.

Then

The Then command is used to describe the expected outcome of the test example. In our example, some actions lead to another webpage, while others display an error message. The steps definition will provide the proper comparison code to confirm the test worked.

And/But

The And and But commands are primarily used to improve the readability of the feature file. They basically continue the previous command. For example, if you had two preconditions:

Given “I have successfully logged in”

Given “I am on the main application screen”

To make the feature file more readable, you could express these as:

Given “I have successfully logged in”

And “I am on the main application screen”

Similarly, expected outcomes such as:

Then “I should see the grid results”

Then “I should not see an error message”

Could be written as:

Then “I should see the grid results”

But “I should not see an error message”

The And and But commands rely on the previous keyword to interpret their actual meaning.

Background

The background keyword (Background:) is used to specify some common steps that should be used with each scenario. The background section of commands (typically Given and When commands) is run prior to each scenario. This eliminates the need to repeat code multiple times in the feature file.

Scenario outline

You might create scenarios where the steps are the same and need to be repeated for a set of values. This is what the Scenario Outline (or scenario template) keyword allows you to do.

Let’s imagine you want to ensure that an error message matches the user-selected language. You might have the following scenarios defined:

Given I’ve set the language to French

When An invalid password is provided

Then The message should be Mot de passe incorrect

Given I’ve set the language to Spanish

When An invalid password is provided

Then The message should be Contraseña invalida

As more languages are offered, these scenarios can become inefficient and time-consuming to enter each time.

Using the scenario outline feature, we can define a table and create a single scenario that would be able to call values from the table.

Scenario Outline

Given I’ve set the language to <language>

When An invalid password is provided

Then The message should be <passwordMessage>

Examples:

Language   | passwordMessage          |

French     | Mot de passe incorrect   |

Spanish    | Contraseña invalida      |

English    | Invalid password         |

The scenario will be repeated three times, one for each row in the examples table. The first row is the column headers and provides the names of the variables to substitute in the scenarios. The table values will be tested one row at a time.

If you find yourself writing scenarios that are the same thing with slight variations, consider the Scenario Outline keyword and a table of values to loop through. You can use the keywords examples or scenarios to begin the table definition. The table can have as many rows and columns as needed for your test.

Parameters

The text following the action commands (Given, When, And, But, and Then) is typically a line of text. You can include parameters within that line by putting the parameter in quotes. This simplifies the coding required in the step definition files. For example, consider the following Then lines:

Then  I see a bad credentials message in the error window

Then  I see a logged-in message in the success window

These two lines would require separate steps since the text is different. However, by using parameters, the lines would now look like this:

Then  I see a “bad credentials” message in the “error” window

Then  I see a “logged-in successfully” message in the “success” window

Readability doesn’t suffer much, and it allows a single step to be used to handle either condition. We will cover this in more detail when we review the step files.

Example feature file

Let’s take our example feature file from the beginning of this chapter (Code Listing 43) and expand it to complete the scenario. The first action is to break this into two feature files: one to test the initial screen (copyright notice appears, help button appears, etc.), and a second to test the login screen. Feature files should be small, testing small areas of the application. Although there is no hard and fast rule, it is general practice that a feature file covers a single application page.

Structure

Typically, you will have the following folders with the E2E folder structure:

• Features: Feature files written by business analysts.
•           Src     : Common page object files to define AppPage and other classes.
• Steps: Code to implement features.

You can customize your setup any way you’d prefer, however, by updating the Protractor.conf file to provide folders for features and steps. The src file (page objects) are referenced directly in the step definitions.

Steps

The Steps folder should contain the code to implement the statements found in the feature files. A step definition will contain a keyword function (Given, Then, or When) followed by a text string, and a function to implement the step.

During the testing, all the specified step files are combined into a collection of step functions. As the feature files are processed, the feature lines are looked up across the entire set of step files. This can cause some unexpected behavior if a step definition that resolves the feature is found in an unexpected file. In addition, if multiple steps are found that could match the feature line, it will be flagged as an ambiguous step definition as shown:

Given I am unauthenticated user

       Multiple step definitions match:

         I am unauthenticated user - e2e\steps\common.steps.ts:750

         I am unauthenticated user - e2e\steps\authentication.steps.ts:14

In this case, you need to change the wording in the step definition to remove the ambiguity. As a matter of practice, I suggest creating a common.step.ts file, which will contain any steps that are applicable to any page in the application, such as navigation between pages or clicking buttons. For each feature file that might need specific step definitions, create a corresponding step file to implement just those steps.

Page object files

The page object files (app.po.ts) typically contain some Protractor methods to interact with the webpage. The default page generated by Angular is the app.po.ts file, which defines the AppPage class. This class has some basic methods likely to be used throughout the application test. If you need specific features added to test an application page, you can extend this class to create a new class for that page. The following snippet shows an example:

export class WithdrawalPage extends AppPage

Common methods are added to this file, so all definitions can use these commands.

Navigation

Most feature files will start at a page within your application. Code Listing 44 shows code to navigate to the home page (base URL) and a page within the application.

Code Listing 44: Navigation

Clicking elements

Another common task is to click on an element, such as a button or a link.

Code Listing 45: Clicking elements

Even though clicking a link would be identical code, I would suggest having a clickLink(link: string) method as well to improve readability.

Entering text

You will often need to enter text into an edit control. Code Listing 46 shows a method that takes an element id and a string of text and enters the text into the control (first clearing the text).

Code Listing 46: Enter new text

Code Listing 47 is a slight variation called appendText that performs the same function, but without clearing the previous text.

Code Listing 47: appendText

Selecting an option  

If you have a list box control, you might want to select an option from the list by either name or position. To select the element by name, we use the by.cssContainingText() locator. This locator takes two arguments: the CSS locator, and the text to find. Code Listing 48 shows a function to click on an element from the list.

Code Listing 48: Click an option in list box by name

You can also add a method to select an option by numeric position in the list. Code Listing 49 shows a method called selectOptionByNumber.

Code Listing 49: Select option by number

This method relies on element.all to return an array of all option tags within the specified select element. It then clicks the array element by number.

Matching steps and features

When a feature file is processed, the keyword and text in the feature file is used to search for a matching step definition (from all step files in the configuration). Our example in Code Listing 50 will match the exact feature “I am on the Access Denied page” when used as part of a Given keyword sentence.

However, business analysts might not use that exact wording, so if they enter “I’ve opened the access denied page,” the E2E test will report it and won’t be able to find a matching step definition. Fortunately, you can use regular expressions to help with matching. Code Listing 50 shows a simple step definition for navigation.

Code Listing 50: Access Denied page step definition

If we update the Given function to the following, we can match a variety of ways a user might enter the feature to navigate to the Access Denied page. The second parameter (i) to the RegExp() function tells the expression to be case-insensitive.

Code Listing 51: Access Denied page step definition, with RegExp() function

While the use of regular expressions can help make things easier for the feature file author, it also increases the possibility of more ambiguous step definitions. If another step definition file has the following step, it would be ambiguous with the regular expression version:

Given(‘I’ve opened the Access Denied Site’)

You need to strike a balance between flexibility for the feature file authors and the complexity of the step definition files. I generally prefer some degree of flexibility, but also providing the feature file authors a template to use for common features.

Using parameters

You can also use parameters in the feature file by enclosing the parameter value in quotes. For example, if we want the user to enter text into a username and password prompt, the feature file statement might be written as:

When I enter “Joe” into the username and “Bad” into the password

The matching step definition would look like:

When(‘I enter {string} into the username and {string} into the password’,

The function would then be written with two parameters, such as:

async function(username: string, password: string)

This would allow the feature file author to use the same syntax when testing a good password for successful login. Code Listing 52 shows a sample step definition to process the login feature line.

Code Listing 52: Sample login step definition

Note: Parameters and regular expressions cannot be used together.

Our code in Listing 53 directly works with the browser object; however, we could create a simpler version using our methods defined in the page object.

Code Listing 53: Sample login using page object

Lookup tables

While there are numerous ways to find an element on the screen, the by.id method is the most likely to find a single element (assuming you’ve added the id to the element). However, the feature file authors likely do not know your element id values. You can create a lookup dictionary, with a text word and the corresponding id for the element. This would allow the feature file to have a statement like:

When I enter 2.50 into the “Tax Amount” field

Your matching step definition would then get the parameter string “Tax Amount” and look in the dictionary to discover the field id is TaxAmt. This allows the feature author to have more flexibility in writing the feature files.

Installation

To install the Cucumber framework so you can use the Gerkhin syntax, you need to run the following installation step:

npm install --save-dev @types/{chai,cucumber} chai cucumber

    protractor-cucumber-framework

This will allow you to create feature files and step definitions for testing your Angular application. You can visit this website to look for other implementations.

Configuration

The protractor.config.js file will contain the necessary configuration options to indicate the feature files to run and set the Cucumber options to run the tests.

The config file creates a JSON object called exports.config. This object holds the various settings needed to run Cucumber tests. Some key properties are:

framework

You need to provide the testing framework to the configuration file, with two options:

   framework: 'custom',

   frameworkPath: require.resolve('protractor-cucumber-framework'),

This indicates we are using a custom framework and provides the name for the framework to use.

specs

This property is a list of test specification files. Typically, you will place your feature files in their own directory, and the specs property lists that folder and the files to run. The following example runs all feature files in the features folder in the E2E structure.

specs: ['./features/**/*.feature']

You can have multiple sets in the specs property.

cucumberOpts

This section is an object that provides the parameters used by Cucumber to process the tests.

require

This is a list of the various TypeScript files that provide the step definitions for the tests. Assuming you place your step definitions in their own folder, you could use the following:

require: ['./steps/**/*.ts']

This will load all the step definitions when building the code to match against the features.

tags

The tags collection allows you to provide a list of regular expressions that provide control over which feature files are run. For example, you might not want to run any draft features, so you can add the following regular expression to exclude scenarios beginning with the text @Draft:

tags: [ ‘~@Draft’ ]

Business analysts could now add the text @Draft prior to the scenario to prevent it from being run or tested while they are still writing the feature.

strict

When this flag is set to true, if any steps are pending or undefined, then the test will fail.

format

The format option allows you to specify the location and format of the output report from the test. For example, the following syntax creates a JSON report in the e2e-report folder:

format: 'json:.e2e-report/results.json'

Visit this URL for a complete list of the configuration syntax.

Summary

Cucumber is a testing framework option for Angular that allows analysts to write specifications in a pseudo-natural language, and not worry about the testing details. A developer or QA resource will code-test step definitions to match the specifications found in the feature files. By using a more structured syntax (like Gerkhin), you can add some consistency to the test plan and provide a common code base to use to run the tests.