E2E テストはどれくらい遅いのか？

実行環境

実行環境のマシンスペックや並列処理によって結果が大きく変わると思われるので、先に実行環境を記載します。GitHub Actions のフリープランで検証をしました。

Usage limits, billing, and administration - GitHub DocsThere are usage limits for GitHub Actions workflows. Usage charges apply to repositories that go beyond the amount of free minutes and storage for a repository.docs.github.com

なお、テストの件数を揃えるため、Playwright は対象ブラウザを Chromium のみに絞っています。

実行するテスト

テストの内容は重要ではないので、適宜読み飛ばしてください。フォーム入力に関するテストを React Testing Library と Playwright でそれぞれ用意しました。内容は似通っており、テストケースを 5 つ含んでいます。

React Testing Library

import SignUpForm from '.';
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';

import type { UserEvent } from '@testing-library/user-event/dist/types/setup/setup';

const mockFn = jest.fn();
let user: UserEvent;
let fields: {
  emailInput: HTMLInputElement;
  passwordInput: HTMLInputElement;
  submitButton: HTMLInputElement;
};

beforeEach(() => {
  mockFn.mockClear();
  user = userEvent.setup();

  render(<SignUpForm saveData={mockFn} />);
  fields = {
    emailInput: screen.getByRole('textbox', { name: 'Email' }),
    passwordInput: screen.getByLabelText('Password'),
    submitButton: screen.getByRole('button', { name: 'Submit' }),
  };
});

describe('SignUpForm', () => {
  test('should render the basic fields', () => {
    const { emailInput, passwordInput, submitButton } = fields;

    expect(emailInput).toBeInTheDocument();
    expect(passwordInput).toBeInTheDocument();
    expect(submitButton).toBeInTheDocument();
  });

  describe('happy path', () => {
    describe('if a user enters required fields, then click the submit button', () => {
      test('should call a callback with an object having an email and password', async () => {
        const { emailInput, passwordInput, submitButton } = fields;

        await user.clear(emailInput);
        await user.type(emailInput, 'alice@example.com');
        await user.clear(passwordInput);
        await user.type(passwordInput, 'my-secret');

        await user.click(submitButton);

        expect(mockFn).toHaveBeenCalledWith({
          email: 'alice@example.com',
          password: 'my-secret',
        });
      });
    });
  });

  describe('unhappy path', () => {
    describe('if a user forgot to enter an email, then click the submit button', () => {
      test('should tell a user that an email is required', async () => {
        const { passwordInput, submitButton } = fields;

        await user.clear(passwordInput);
        await user.type(passwordInput, 'alice@example.com');

        // There's no alert before submitting.
        expect(screen.queryByRole('alert')).not.toBeInTheDocument();

        // After submitting, the alert appears.
        await user.click(submitButton);
        expect(screen.getByRole('alert')).toHaveTextContent(
          'Email is required.',
        );
      });
    });

    describe('if a user forgot to enter a password', () => {
      test('should tell a user that a password is required', async () => {
        const { emailInput, submitButton } = fields;

        await user.clear(emailInput);
        await user.type(emailInput, 'alice@example.com');

        // There's no alert before submitting.
        expect(screen.queryByRole('alert')).not.toBeInTheDocument();

        // After submitting, the alert appears.
        await user.click(submitButton);
        expect(screen.getByRole('alert')).toHaveTextContent(
          'Password is required.',
        );
      });
    });

    describe('if a user entered a password that is shorter than 8 characters', () => {
      test('should tell a user the minimum length of password', async () => {
        const { emailInput, passwordInput, submitButton } = fields;

        await user.clear(emailInput);
        await user.type(emailInput, 'alice@example.com');
        await user.clear(passwordInput);
        // 'secret' is too short to use it as a password.
        await user.type(passwordInput, 'secret');

        // There's no alert before submitting.
        expect(screen.queryByRole('alert')).not.toBeInTheDocument();

        // After submitting, the alert appears.
        await user.click(submitButton);
        expect(screen.getByRole('alert')).toHaveTextContent(
          'Password must be at least 8 characters.',
        );
      });
    });
  });
});

Playwright

import { expect, test } from '@playwright/test';

import type { Locator } from '@playwright/test';

let fields: {
  emailInput: Locator;
  passwordInput: Locator;
  submitButton: Locator;
};

test.beforeEach(async ({ page }) => {
  await page.goto('/');

  fields = {
    emailInput: page.getByRole('textbox', { name: 'Email' }),
    passwordInput: page.getByLabel('Password'),
    submitButton: page.getByRole('button', { name: 'Submit' }),
  };
});

test.describe('SignUpForm', () => {
  test('should render the basic fields', async () => {
    const { emailInput, passwordInput, submitButton } = fields;

    await expect(emailInput).toBeVisible();
    await expect(passwordInput).toBeVisible();
    await expect(submitButton).toBeVisible();
  });

  test.describe('happy path', () => {
    test.describe('if a user enters required fields, then click the submit button', () => {
      test('should call a callback with an object having an email and password', async () => {
        const { emailInput, passwordInput, submitButton } = fields;

        await emailInput.clear();
        await emailInput.fill('alice@example.com');
        await passwordInput.clear();
        await passwordInput.fill('my-secret');

        await submitButton.click();
      });
    });
  });

  test.describe('unhappy path', () => {
    test.describe('if a user forgot to enter an email, then click the submit button', () => {
      test('should tell a user that an email is required', async ({ page }) => {
        const { passwordInput, submitButton } = fields;

        await passwordInput.clear();
        await passwordInput.fill('my-secret');

        // There's no alert before submitting.
        await expect(page.getByText('Email is required.')).not.toBeVisible();

        // After submitting, the alert appears.
        await submitButton.click();
        await expect(page.getByText('Email is required.')).toBeVisible();
      });
    });

    test.describe('if a user forgot to enter a password, then click the submit button', () => {
      test('should tell a user that a password is required', async ({
        page,
      }) => {
        const { emailInput, submitButton } = fields;

        await emailInput.clear();
        await emailInput.fill('alice@example.com');

        // There's no alert before submitting.
        await expect(page.getByText('Password is required')).not.toBeVisible();

        // After submitting, the alert appears.
        await submitButton.click();
        await expect(page.getByText('Password is required')).toBeVisible();
      });
    });

    test.describe('if a user enter password that is shorter than 8 characters', () => {
      test('should warn a user to enter password at least 8 characters', async ({
        page,
      }) => {
        const { emailInput, passwordInput, submitButton } = fields;

        await emailInput.clear();
        await emailInput.fill('alice@example.com');
        await passwordInput.clear();
        // 'secret' is too short to use it as a password.
        await passwordInput.fill('secret');

        // There's no alert before submitting.
        await expect(
          page.getByText('Password must be at least 8 characters.'),
        ).not.toBeVisible();

        // After submitting, the alert appears.
        await submitButton.click();
        await expect(
          page.getByText('Password must be at least 8 characters.'),
        ).toBeVisible();
      });
    });
  });
});

これらのテストをコピーして、件数を 1、10、20、30、50 と増やしていきます。

実行時間の比較

GitHub 上でテストの実行結果を見ていきます。テストが 1 件だけのとき、Jest (React Testing Library) はこんな感じで・・・

Playwright はこんな具合でした。

最終的な結果はこうなりました。

"圧倒的じゃないか Testing Library は！"

はい・・・。案の定な結果となりましたので使い分けは必須です 🤦‍♂️
通常時は Playwright でクロスブラウザのテストをしているため、その差はさらに広がると思われます。

おわりに

今回の検証で E2E テストとの付き合い方が見えてきました。例に出したような Validation 目的のテストは E2E の領分ではないということです。

E2E テストの粒度をコンポーネントと同等まで細かくすると、実行時間が問題になります。反対に粒度を荒くした E2E テストからは、失敗した際の有益なフィードバック (テストが失敗した要因を端的に示すもの) が得られないのです。どのテスト種別を採用するかは適材適所ということになります。