End-to-end Tests that Don’t Suck with Puppeteer

Test runner results

How we are using headless Chrome to write end-to-end tests that don’t drive you crazy

What are end-to-end tests?

Tests written to check software functionality can be grouped into a few categories. Some of the most popular categories include:

  • unit tests check input => output of self-contained functions.
  • integration tests check that individual pieces of your app play nicely together.
  • end-to-end tests check that entire features work from the user’s perspective.

This last group of tests is what we are talking about in this post. They are sometimes known as acceptance tests or functional tests. I’ll be referring to them as e2e tests.

Why write e2e tests?

The most important thing for any app is that it works for your users. Good e2e tests let you know when at least one piece of a feature (database, API, UI) isn’t working as expected. This can be extremely valuable. It removes the need to manually check existing features in a browser whenever you make changes.

But e2e tests are horrible, disgusting, dreadful pieces of garbage

e2e tests have historically been awful. They tend to be sluggish and brittle. They tend to break easily and eat away at valuable developer time. Most teams either don’t write them or write them with distaste, like forcefully taking a pill you think will be good for you. But there is a better way! <cue infomercial music>.

Using Puppeteer instead of Selenium

One of the most popular tools for e2e testing is Selenium, which is a tool for automating web browsers. Selenium sounds cool in theory: write one set of tests that run on all browsers and devices, woohoo! Jk. In practice, Selenium tests are slow, brittle, and costly. So, on Ropig we are using Puppeteer – the official headless Chrome library. A “headless” browser is just a browser that doesn’t have a graphical user interface.

A Puppeteer test looks like this

test('can logout', async () => {
  await page.click('[data-testid="userMenuButton"]')
  await page.waitForSelector('[data-testid="userMenuOpen"]')
  await page.click('[data-testid="logoutLink"]')
  await page.waitForSelector('[data-testid="userLoginForm"]')
})

In this case, we tap a drop-down menu, wait for it to open, tap a logout link, and wait for the login form to show. If any of these steps don’t work, the test will fail.

A few more real examples pulled from the Ropig test suite

import faker from 'faker'
import puppeteer from 'puppeteer'

const appUrlBase = 'http://localhost:4000'
const routes = {
  public: {
    register: `${appUrlBase}/register`,
    login: `${appUrlBase}/login`,
    noMatch: `${appUrlBase}/asdf`,
  },
  private: {
    events: appUrlBase,
    alerts: `${appUrlBase}/alerts`,
    services: `${appUrlBase}/services`,
    team: `${appUrlBase}/team`,
  },
  admin: {
    templates: `${appUrlBase}/templates`,
  },
}

const user = {
  email: faker.internet.email(),
  password: 'test',
  firstName: faker.name.firstName(),
  lastName: faker.name.lastName(),
  mobile: faker.phone.phoneNumber(),
  companyName: faker.company.companyName(),
}

let browser
let page
beforeAll(async () => {
  browser = await puppeteer.launch(
    process.env.DEBUG
      ? {
          headless: false,
          slowMo: 100,
        }
      : {}
  )
  page = await browser.newPage()
})

describe('private routes', () => {
  test('redirects to login route when logged out', async () => {
    await page.goto(routes.private.events)
    await page.waitForSelector('[data-testid="userLoginForm"]')
  })
})

describe('registration', () => {
  test('can get to register route from login form', async () => {
    await page.click('[data-testid="registerLink"]')
    await page.waitForSelector('[data-testid="userAccountForm"]')
  })

  test('can create new user account', async () => {
    await page.goto(routes.public.register)
    await page.waitForSelector('[data-testid="userAccountForm"]')
    await page.click('[data-testid="userRegisterInputWithEmail"]')
    await page.type(user.email)
    await page.click('[data-testid="userRegisterInputWithPassword"]')
    await page.type(user.password)
    await page.click('[data-testId="userAccountSubmitButton"]')
    await page.waitForSelector('[data-testid="userSettingsForm"]')
  })

  test('logs in and redirects to events route when registration is complete', async () => {
    await page.waitForSelector('[data-testid="events"]')
  })
})

describe('logout', () => {
  test('can logout', async () => {
    await page.waitForSelector('[data-testid="userMenuButton"]')
    await page.click('[data-testid="userMenuButton"]')
    await page.waitForSelector('[data-testid="userMenuOpen"]')
    await page.click('[data-testid="logoutLink"]')
    await page.waitForSelector('[data-testid="userLoginForm"]')
  })
})

describe('login', () => {
  test('can login', async () => {
    await page.waitForSelector('[data-testid="userLoginInputWithEmail"]')
    await page.click('[data-testid="userLoginInputWithEmail"]')
    await page.type(user.email)
    await page.click('[data-testid="userLoginInputWithPassword"]')
    await page.type(user.password)
    await page.click('[data-testid="userLoginSubmitButton"]')
    await page.waitForSelector('[data-testid="events"]')
  })
})

describe('on call', () => {
  test('starts off call', async () => {
    await page.waitForSelector('[data-testid="offCallStatus"]')
  })

  test('can toggle on call status', async () => {
    await page.click('[data-testid="onCallButton"]')
    await page.waitForSelector('[data-testid="onCallStatus"]')
  })

  test('shows on call list with alerts', async () => {
    await page.goto(routes.private.alerts)
    await page.waitForSelector('[data-testid="someOnCallButton"]')
    await page.click('[data-testid="someOnCallButton"]')
    await page.waitForSelector('[data-testid="onCallBadge"]')
  })

  test('shows on call badge in team list', async () => {
    await page.goto(routes.private.team)
    await page.waitForSelector('[data-testid="onCallBadge"]')
  })
})

describe('errors', () => {
  test(`shows 404 message when route doesn't exist`, async () => {
    await page.goto(routes.public.noMatch)
    await page.waitForSelector('[data-testid="noMatch"]')
  })
})

describe('admin', () => {
  test('redirects to root route when not an admin', async () => {
    await page.goto(routes.admin.templates)
    await page.waitForSelector('[data-testid="events"]')
  })
})

afterAll(() => {
  if (!process.env.DEBUG) {
    browser.close()
  }
})

We are using Jest as our test runner, but you can use any testing tools you want with Puppeteer.

Headless mode

Here are what these tests look like when you run them in headless mode:

A screenshot of running end-to-end tests in headless mode

Running end-to-end tests in headless mode

Debug mode

Here is a video of what these tests look like when you run them in debug mode. Debug mode opens a real browser and slows down each step so you can see what is happening:

 

Some of the things I really like about Puppeteer

  • It’s official from the Chrome team. This means it has a solid future. This also means it supports all modern JavaScript syntax available in Chrome (like async/await).
  • Puppeteer is headless so it can run without a visual browser; this makes running tests faster. Additionally, tests can run in Continuous Integration without extra setup or costs.
  • It has a simple API to do common things like typing in inputs, clicking etc.
  • Puppeteer can be used for any browser automation, not just testing.
  • It doesn’t need to know anything about your stack. We are using Elixir and React, but we could just as well be using any other tools.

Note that Puppeteer only runs tests in Chrome. For many apps like Ropig, this is enough because we only support modern browsers which have minimal inconsistencies. If your app has a lot of device or browser specific code, you may still want Selenium. For everyone else, Puppeteer makes a lot of sense. 🙂

Tips for writing e2e tests

Tip 1: Test features, not implementation

The purpose of e2e tests is to fail when you break some expected user-facing functionality. When you have a failing test it means you either broke something that should be fixed, or the feature has changed (so the test needs to be updated). If you find yourself dealing with failing tests outside these two situations it means you have brittle tests. Brittle tests check the implementation of a feature, which ties you to the implementation. Instead, I highly recommend only testing the end result of the feature (what the user expects – the behavior).

A bad example

test('can logout', async () => {
  await page.click('#menu div > a')
  sleep 500
})

This is a brittle test because it relies on implementation details (arbitrary nested elements and wait times).

A good example

test('can logout', async () => {
  await page.click('[data-testid="userMenuButton"]')
  await page.waitForSelector('[data-testid="userMenuOpen"]')
  await page.click('[data-testid="logoutLink"]')
  await page.waitForSelector('[data-testid="userLoginForm"]')
})

This test is less brittle because it uses test IDs and waits for events before proceeding.

Test IDs

We use test IDs like this to provide interaction as a user would with key elements. We use these as a contract between implementation and user interaction. The benefit of test IDs is that we could change the underlying implementation without breaking the test. For example, we could move the logoutLink test ID to a button tag instead of an a tag. Or we could switch our view rendering from Angular to React. The test would still pass because the log out feature still works.

Tip 2: Stick to the happy path features

Even with Puppeteer, e2e tests are still slower and more brittle than unit tests. We try to use unit tests where we can, especially edge cases. Then we add e2e tests only for the “happy path” of a user. This lets us know when something breaks for the majority use case.

Tip 3: Use async/await for asynchronous things

Using async/await is a great way to deal with chains of async events, which is most of what e2e testing is. async/await is cleaner than callback chains. And please, whatever you do, DON’T use arbitrary wait times. These tests will fail from race conditions with different network and computer speeds.

Tip 4: Use a fake data generator like faker

Using a fake data generator like faker ensures that your app is flexible. It guarantees your app has the same output each time it is run with the same input. This is in contrast to using a single test account for each test run that has a bunch of state sitting around, making your tests inconsistent. For example, in Ropig we use faker like this to create a random user for each test run:

import faker from 'faker'

const user = {
  email: faker.internet.email(),
  password: 'test',
  firstName: faker.name.firstName(),
  lastName: faker.name.lastName(),
  mobile: faker.phone.phoneNumber(),
  companyName: faker.company.companyName(),
}

Summary

e2e testing has traditionally been difficult. Using headless Chrome has made e2e testing more reliable and simple here on the Ropig team. I recommend you try it out on your projects!

Trevor Miller
Trevor Miller
I'm a software developer. I love learning and solving problems. I strive to learn new things every day and share what I learn.

41 Comments

  1. Wout Mertens says:

    So do you ship your app in prod with all those test ids attached? Seems a bit wasteful, but only a little. Or do you have some setup that only adds them when `NODE_ENV=test`?

  2. Vytautas Butkus says:

    Hey Trever, nice article! Did you had a chance to compare it’s speed with currently hyping cypress e2e framework? What about nightmare.js which also runs headlessly?

    • Hi @vytautasbutkus:disqus, thank you for your comment and checking out the post. I haven’t tested our test suite with other headless options, but I have seen some benchmarks and most headless options are pretty comparable speed-wise. Some of the headless tools are actually switching to use Puppeteer under the hood (like Chromeless), so they should it theory be the same speed. I like using Puppeteer directly since it is officially supported by the Chrome team, but there are a lot of other great options like Cypress, Nightmare, Webdriver.io, Chromeless etc. as you mentioned which all have different trade-offs. I think most of the benefits mentioned in this apply to any headless option so I say use whatever you prefer!

  3. miron says:

    Great article, thanks Trevor. I love the idea about test ids!

  4. Rahul Yadav says:

    Nice article Trevor. I haven’t checked out Puppeteer but will do it soon. I have one question though- is it possible to have something on the lines of Page Object Model with Puppeteer too? If yes, can you add a working example in your blog. That’d be pretty handy.

  5. zeev rosental says:

    Hi Trevor,

    Great article!

    Can you please elaborate about
    the test-ids you generate?

    I’m in a react project, and we
    wanted an efficient way to generate ids (that will generate the same
    ids each run and that will give some context of the components that the ids
    belong to). Do you have any good input?

    Thank you!

  6. Cédric Brancourt says:

    Hi, nice summary.
    This is how addressing the front end part of the end 2 end tests.
    Also, e2e testing involves sometimes some side effects testing. Let say your app has to send SMS on some event. This should be part of the end 2 end testing. But this is not part of the frontend.
    This subject seems always forgotten when I read articles about e2e testing.

    Kind Regards

  7. Joe says:

    Can this be used as an alternative to RSpec/Capybara on a Rails application?

  8. […] do you test your applications? A lot of other folks are doing E2E testing with Puppeteer. And […]

  9. Mike says:

    this is very helpful, great write-up thank you

  10. […] Возможно, вам пока не вполне удобно работать с самими Puppeteer или его API. Я вас понимаю. И если новизна этого проекта наполняет вас сомнением в его практической применимости, вы можете взглянуть, например, на Cypress. Однако, Puppeteer даёт разработчикам поистине безграничные возможности. Сейчас создаются тестировочные фреймворки на основе этой библиотеки. Конечно, со временем API Puppeteer может и поменяться, но полагаю, что базовые вещи, о которых мы тут говорили, никуда не денутся. Кроме того, нельзя не отметить, что Puppeteer отлично сочетается с Jest. Многие, кроме того, выполняют с помощью Puppeteer E2E-тестирование. […]

  11. […] End-to-end Tests that Don’t Suck with Puppeteer → […]

  12. nagarajan raman says:

    This is great. But even with selenium, we could run the tests with chrome in headless mode. So is there any other advantage we have in puppeteer compared to selenium

    • Hey Nagarajan, you are right that Selenium can be used with headless browsers. I think that can be a great approach for apps that need to run tests on every browser but also want a headless option. But for me, I prefer the simplicity of Puppeteer over the many options and features that Selenium has. Because Puppeteer keeps it’s API focused, it makes using best practices easy and has less bugs. But it just depends on if you like a simple/limited API or if you want lots of options; for lots of options, Selenium definitely has more 🙂

      • jellman says:

        Hi, I find that a stange reason. You’re not forced to use the entire Selenium API, you just use the API you need. Anyway a best practice in architecting automated test suites is to create your own DSL(Domain Specific Language) in a page-object framework. This way, you create your own API that focuses on the functionality your company needs for its tests. While I think Puppeteer is damn cool, I just can’t see it being useful in a production test environment, since those would often require cross-browser testing. Also, the Webdriver protocol is now a w3 standard that’s being used by products industry wide, doesn’t this break from that emerging consensus?

        • I think it depends on what type of app you are building. For many modern lightweight apps being built by startups or small teams that only support modern browsers, Puppeteer is a much simpler + lightweight option. But yes, if you want browser-specific tests, Selenium and the Webdriver protocol are the way to go 🙂

    • Imobach Ramírez says:

      Selenium does not currently allow you to obtain the headers or status code of your requests (https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/141). For me that’s a big limitation. With Puppeteer if you can do it.

  13. shwetasabne says:

    Hey Trevor ! Awesome article. I am curious have you been able to get puppeteer to work with nock ? I am trying to mock out my backend using nock but it doesn’t seem to working with puppeteer.

  14. […] end-to-end tests check functionality works from the user’s perspective (here’s a post on how we end-to-end test the Ropig app). […]

  15. john harkin says:

    Hi,
    Great article.
    Any reason for using test ids rather than the html id attribute?
    Thanks
    John

  16. Buy cialis says:

    Purchasing cialis on the internet http://cialisessale.com/

    Great delivery. Great arguments. Keep up the good spirit.

  17. writeessay says:

    write an essay for me http://dekrtyuijg.com/

    You’ve made your position very effectively!!

  18. Pavlo Kochubei says:

    Trevor, thanks for the great article! It’s really helpful. Looking at the code, I can’t get how you save the login information between the tests?

  19. dman101 says:

    I see “await page” on every line!! How can you avoid this?

  20. Eagle says:

    Making puppeteer play nicely with docker can be such a pain though. I never got debug mode working.

  21. Google says:

    Google

    One of our guests lately suggested the following website.

  22. Google says:

    Google

    Please stop by the web pages we adhere to, which includes this one, because it represents our picks through the web.

  23. […] 現代瀏覽器效能優化-JS篇談談 HTTPS《演算法》第一章學習筆記js實現JS 裡為什麼會有 thisHi,我們做了一款 “極客時間” AppRebuilding slack.comEnd-to-end Tests that Don’t Suck with Puppeteer […]

Leave a Reply

Your email address will not be published.

Send this to a friend