Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Acceptance Test POC and Setup #41

Merged
merged 42 commits into from
Jun 23, 2021
Merged

Acceptance Test POC and Setup #41

merged 42 commits into from
Jun 23, 2021

Conversation

slaymance
Copy link
Contributor

@slaymance slaymance commented Jun 11, 2021

Issue #, if available: Issue #33

Acceptance Tests

This PR is meant as a proof of concept and initial setup for our Acceptance Testing strategy. Our Acceptance Tests are end-to-end tests which cover supported customer use cases of our Amplify UI components and give us confidence that component rewrites, refactors, and migrations continue to support said use cases.

Tooling

  • Cypress: test runner whose API is used to interact with UI components and make assertions about expected behavior. This is the testing framework for JavaScript framework implementations
  • Cucumber and cypress-cucumber-preprocesssor: testing tool which allows for test definitions in a Given, When, Then syntax that allow test cases to be decoupled from test implementation
  • Testing Library and Cypress Testing Library: testing utilities which allow you to write tests which interact with our components in a way real users would while keeping accessibility at the forefront of testable interactions

Setup

Cypress requires a running development instance of a sample application on a specified URL that it may interact with in order to test assertions. For the purposes of this POC, I've created a separate Next.js page in our react docs which uses the new Authenticator component. I've also configured Amplify with my own credentials for demonstration purposes.

// docs/src/pages/auth.tsx
import { Authenticator } from "@aws-amplify/ui-react";
import { Amplify } from "aws-amplify";

import awsExports from "../aws-exports";

Amplify.configure(awsExports);

const authPage = () => (
  <Authenticator>
    {({ state, send }) => (
      <div>
        <h1>Welcome {state.context.user.username}!</h1>
        <button
          className="px-2 bg-white rounded shadow"
          disabled={state.matches("signOut")}
          onClick={() => send("SIGN_OUT")}
        >
          Sign Out
        </button>
      </div>
    )}
  </Authenticator>
);

export default authPage;

Now we can run yarn dev from the repository root directory and use the above page for our Cypress tests.

Testing Work Flow

Writing Cucumber test definitions

The first step to creating an acceptance test is to use Cucumber's Gherkin language to declare the test steps in a Given, When, Then syntax. Within the acceptance-tests workspace (acceptance-tests -> cypress -> tests -> acceptance), Cucumber tests are defined in files with the .feature file extension (e.g. Authentication.feature). A cucumber test definition can look like this:

Feature: Authentication

  Amplify's Authenticator is a wrapper for a developer's application.
  It provides an application with authentication features using AWS Cognito.
  Its features include sign in, sign up, and sign out.

  Scenario: Sign in with invalid credentials
    Given I'm at the sign in page
    When I type an invalid email address "fake@email.com"
    And I type an invalid password "fakepassword"
    And I attempt to sign in
    Then I see the error "User does not exist."

The keywords Given, When, Then, And, and But indicate step definitions for a given scenario. While this Cucumber definition isn't a functioning test yet, it provides a syntax which can be used across frameworks and across test runner implementations.

For example, when aiming for feature parity with a native implementation of Amplify UI which won't use Cypress as a test runner, this Cucumber test can be used alongside whichever test runner is used for the native implementation to allow for Behavior Driven Development when writing code.

This syntax is also intuitive regardless of coding language proficiency and can be used to communicate test coverage of use cases for external stakeholders.

Using Cypress for application actions and assertions

Now that our Cypress test scenario is defined, we need to write the actual logic which executes each of the steps defined in our Authentication.feature scenario. Next to our feature file, we create a directory of the same name with a typescript file of the same name:

tests
|__acceptance
   |__Authentication.feature
   |__Authentication
      |__Authentication.ts

This file, Authentication.ts will use Cypress and cypress-cucumber-preprocessor to define the logic for each step through interacting with our sample application and making assertions about outcomes of interactions:

// Authentication.ts
import { And, Given, Then, When } from "cypress-cucumber-preprocessor/steps";

Given("I'm at the sign in page", () => {
  cy.visit("/auth");
});

When("I type an invalid email address {string}", (email: string) => {
  cy.findByRole("textbox", { name: /username/i }).type(email);
});

And("I type an invalid password {string}", (password: string) => {
  cy.findByLabelText(/password/i).type(password);
});

And("I attempt to sign in", () => {
  cy.findByRole("button", { name: "Sign In" }).click();
});

Then("I see the error {string}", (errorMessage: string) => {
  cy.findByRole("alert", { name: /error/i }).should("contain", errorMessage);
});

Note: the methods .findByRole and .findByLabelText are part of Testing Library and not native to the Cypress API. These will be discussed below.
Each step for every scenario is clearly defined and its logic laid out with the Then step serving as our assertion.

Using Testing Library for user-centric and accessible actions

As mentioned above, we defined the logic of our test steps using methods from Testing Library. These allow us to interact with our sample application using methods that more closely align with how a user would interact with an application. Instead of using data attributes for elements and querying by those selectors, we instead can query our UI for elements based on their roles or labels. This forces us to test our application with accessibility in mind first since being unable to query an element on a page by role or label indicates it may lack accessibility.

Running Tests

From the acceptance-tests workspace, Cypress tests can be run with the following scripts:
yarn cypress:open: opens a browser which will be used for the test runner
yarn cypress:run: test runner executes in a headless Electron browser

Next Steps

  • Discovery regarding using Cucumber's Gherkin syntax; what are the best practices and advanced features
  • Discovery regarding using Testing Library's API; what are the best practices
  • Determine strategy for shared logic between tests that interact with the application (Page Object Model vs. App Actions)
  • Discovery for what is necessary for a staging/testing AWS environment for true end-to-end testing
  • Discovery for integrating with GitHub Actions and Secrets
  • Determine an appropriate folder structure that highlights multi-framework testing, intuitive feature testing, and ease of integration with documentation

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@slaymance slaymance self-assigned this Jun 16, 2021
@slaymance slaymance marked this pull request as draft June 16, 2021 15:53
@slaymance slaymance changed the title WIP: Acceptance tests Acceptance Test POC and Setup Jun 16, 2021
@slaymance slaymance requested a review from ericclemmons June 16, 2021 20:42
@ericclemmons
Copy link
Contributor

Thanks for the demo! What I'm interested in investigating is:

  1. Documentation for Authenticator lives @ https://main.d3inl0muob87le.amplifyapp.com/components/authenticator
  2. Features live @ https://main.d3inl0muob87le.amplifyapp.com/components/authenticator/features/${feature}
  3. Those features are automatically included in the documentation by convention (not sure to what degree it makes sense). Perhaps https://nextjs.org/docs/api-reference/next/image#layout is a good example of high-level balanced with "demos"
  4. Demos/Examples live at https://main.d3inl0muob87le.amplifyapp.com/components/authenticator/examples/${background}
  5. Cypress tests run against https://main.d3inl0muob87le.amplifyapp.com/components/authenticator/examples/${background} for each scenario.

This would serve as the foundation for our CONTRIBUTING.md that would close the gap between Docs, Development & Testing.

We'll further expand this to include Vue, Angular, etc. by convention with the URL. (For example, /components/examples/${background} could be served by Vue for that same Cypress test!)

@slaymance
Copy link
Contributor Author

And some additional questions:

  • To what extent do our steps (Given, When, Then) show up in our documentation?
  • How do we run the same Cucumber test multiple times across different frameworks?
  • How do we configure the tests to run against a staging/testing AWS environment?
  • Can we integrate Gherkin syntax with markdown?

@ericclemmons
Copy link
Contributor

@slaymance Great question:

To what extent do our steps (Given, When, Then) show up in our documentation?

Initially I considered https://github.com/cucumber/common/blob/main/gherkin/MARKDOWN_WITH_GHERKIN.md, but have since changed my mind.

It's a 2-day door, but keeping the widely-supported Gherkin syntax in .feature still allows our docs to parse it (in getStaticProps) and render whatever we deem necessary (likely the Feature/Description) and a link to the examples/* matching it.

How do we run the same Cucumber test multiple times across different frameworks?

I did some quick Googling and found https://github.com/mathanpec/react-native-detox-cucumber, https://github.com/Ahmed-Ali/Cucumberish, and others.

How do we configure the tests to run against a staging/testing AWS environment?

Today, we store aws-exports.js with our samples: https://github.com/aws-amplify/amplify-js-samples-staging/blob/master/samples/react/auth/authenticator/src/aws-exports.js

We'd have to work with security on this in a public repo, but I'm interested in re-using examples across different environments.

The Amplify.configure backend will need to exist somewhere and then injected somehow into the test runner or example.

My current idea is to leverage .env.local (https://nextjs.org/docs/basic-features/environment-variables#loading-environment-variables) and having _app.js inject these as JSON into Amplify.configure for our examples or our docs.

Another alternative is to serialize them and load http://localhost:3000/.../some-example#{aws_region:"something"} that gets deserialized on the client and put into Amplify.configure.

There's also https://docs.cypress.io/api/cypress-api/env.

.gitignore Outdated Show resolved Hide resolved
acceptance-tests/cypress/tests/acceptance/SignIn.feature Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
@slaymance slaymance marked this pull request as ready for review June 23, 2021 00:41
Copy link
Contributor

@eddiekeller eddiekeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid first PR to get this all set up! Thanks for this. Left a few comments that aren't blocking really but I think would be worth considering.

@slaymance slaymance merged commit ffc491a into main Jun 23, 2021
@slaymance slaymance deleted the acceptance-tests branch June 23, 2021 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants