Skip to content

Configure Playwright

Adapter status: ✅ Complete. It is used by lippia-neo-generate-test-suite (to write UI and API tests) and lippia-neo-run-and-heal (to run them and fix them).

Value in qa-stack.yaml test_framework: playwright
Language javascript (.spec.js tests) or typescript (.spec.ts tests, requires tsconfig.json)
App types web, api, and hybrid
Authentication google_sso, basic_auth, api_token, or none
Default tests folder tests/playwright/

Want to write the tests in Gherkin (.feature)? Use playwright-bdd or playwright-cucumberjs:

playwright playwright-bdd playwright-cucumberjs
Test unit One .spec per case, in a folder per story A Scenario in the business flow's .feature Same as playwright-bdd
Who runs it npx playwright test npx bddgen + npx playwright test npx cucumber-js (Playwright as a library)
Filter by story or case By folder By .feature, folder, or tag By .feature, folder, or tag
Data-driven data — ✅ ✅

1. What you need installed

Tool Version What it's for How to check it
Node.js 18 or higher Running Playwright and the scripts node -v
@playwright/test The one in the project's package.json The framework npx playwright --version
Playwright browsers The ones for your Playwright version Running the UI tests npx playwright install
dotenv Any So playwright.config reads .env.tests Listed in package.json
Bash (Git Bash or WSL on Windows) — Running run.sh and parse-results.sh bash --version

From the root of the project (where package.json is):

npm install
npx playwright install

npx playwright install downloads the browsers. You only need to run it once per machine, or each time the Playwright version changes.

For the skills that browse the application live (lippia-neo-run-manual-testing, lippia-neo-reproduce-bug), you also need the Playwright MCP with its browser:

npx @playwright/mcp install-browser chrome-for-testing

If that step fails or the MCP doesn't connect, follow Playwright MCP setup.


2. Sample qa-stack.yaml

stack:
  test_framework: playwright
  language: typescript        # or javascript
  app_type: web               # web | api | hybrid
  auth: basic_auth            # google_sso | basic_auth | api_token | none
  pipeline_provider: github

environment:
  base_url: https://techstore-staging.com
  app_version: 2.4.1
  os: Windows 11

paths:
  user_stories_storage:
    type: local
    cloud_provider_mcp: null
    local_path: user-stories/
  test_plans_dir: test-plans/
  tests_dir: tests/playwright/
  reports_dir: test-results/reports/
  scripted_dir: execution/
  exploratory_dir: exploration/
  env_file: .env.tests

commands:
  run_tests: null        # uses the adapter's run.sh → npx playwright test
  install_deps: null     # uses npm ci

3. Configure playwright.config

This is the most important step: without it, NEO generates the tests but can't run them or read the results. lippia-neo-generate-test-suite shows you the block that matches your app type when it finishes, but it's a good idea to have it ready beforehand.

Your playwright.config.js (or .ts) must have:

What Why Value
testDir So Playwright looks for the tests where NEO writes them The same value as paths.tests_dir
JSON reporter parse-results.sh reads that file to know what failed ['json', { outputFile: 'test-results/results.json' }]
Loading .env.tests So the tests read the credentials without having them written in require('dotenv').config({ path: '.env.tests' })
use.baseURL So the tests use relative paths (page.goto('/')) The URL of the environment under test
testMatch with the correct extension So Playwright finds the generated files **/tc-*.spec.ts if the project is TypeScript, **/tc-*.spec.js if it's JavaScript

Minimal example for a web app:

const { defineConfig, devices } = require('@playwright/test');
require('dotenv').config({ path: '.env.tests' }); // = paths.env_file

module.exports = defineConfig({
  testDir: './tests/playwright',                  // = paths.tests_dir
  retries: process.env.CI ? 2 : 1,
  timeout: 60000,
  reporter: [
    ['json', { outputFile: 'test-results/results.json' }], // required by lippia-neo-run-and-heal
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['list'],
  ],
  use: {
    baseURL: process.env.APP_BASE_URL || 'https://techstore-staging.com',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] }, testMatch: '**/tc-*.spec.js' },
    { name: 'firefox',  use: { ...devices['Desktop Firefox'] }, testMatch: '**/tc-*.spec.js' },
  ],
});

Typical TypeScript mistake: if the project is typescript, NEO generates .spec.ts files. If you leave testMatch at .spec.js, Playwright can't find any tests. In TypeScript this isn't a rare case: it happens every time you don't change it.

By app type

app_type What to add in projects
web The browser projects (chromium, firefox) with testMatch: '**/tc-*.spec.<ext>'
api An api project with no browser, with testMatch: '**/tc-*.api.spec.<ext>'
hybrid Both: browsers for the tc-*.spec files and api for the tc-*.api.spec files

UI tests run on Chromium and Firefox through the config's projects. The generated tests do not have their own logic to detect the browser.


4. Authentication

basic_auth and api_token

Fill in the values in .env.tests (the file is created by lippia-neo-bootstrap-stack with empty keys and is kept out of git):

# .env.tests
APP_USER=test-user
APP_PASSWORD=test-password
# if app_type is api or hybrid:
API_BASE_URL=https://api.techstore-staging.com
API_TOKEN_MANAGER=...
API_TOKEN_VIEWER=...

The generated tests read those values with process.env.NOMBRE and never write them as literals.

google_sso

Google login is not automated (Google detects it and blocks it). Instead:

  1. Just once, you open a visible browser with the project's script (scripts/save-auth.sh or npm run auth:save) and log in by hand.
  2. Playwright saves the session (cookies and localStorage) to a JSON file.
  3. playwright.config points to that file with storageState, and each test starts already logged in.
  4. When the session expires, you run the script again.

⚠️ Check that the session file path matches in both places. The configuration templates use playwright/.auth/user.json. However, run.sh checks .auth/auth-state.json by default, and if it can't find it, it stops with Auth state no encontrado. If your project uses a different path, set the QA_AUTH_STATE_PATH variable to the real path before running.

Never push the session file to git. It contains valid tokens. Add its folder (.auth/ or playwright/.auth/) to .gitignore.


5. What NEO generates

One file per test case, inside one folder per user story:

tests/playwright/              ← paths.tests_dir
└── US-002/
    ├── tc-001-agregar-producto-al-carrito.spec.ts
    ├── tc-002-carrito-vacio.spec.ts
    └── tc-003-crear-orden.api.spec.ts     ← API test (app_type api / hybrid)

Rules the generated tests follow:

  • Selectors, in this order of preference: role + name (getByRole), label, data-testid, visible text and, as a last resort, CSS class or id. If a selector isn't known, an explicit TODO is left.
  • Waits based on conditions (waitForURL, expect(...).toBeVisible()), never fixed pauses like waitForTimeout(2000).
  • Imports with require (CommonJS), unless package.json has "type": "module".

6. How lippia-neo-run-and-heal runs it

lippia-neo-run-and-heal uses the adapter's run.sh, which:

  1. Checks that node_modules exists and that Playwright is installed. If anything is missing, it exits with code 2.
  2. If auth is google_sso, checks that the session file exists.
  3. Deletes the result of the previous run, so old results aren't mixed up with new ones.
  4. Runs npx playwright test --reporter=json <tests_dir>.
  5. Checks that test-results/results.json was generated.

Variables you can use to adjust the run:

Variable Default What it's for
QA_TESTS_DIR paths.tests_dir Folder or folders to run, separated by spaces (e.g. "tests/playwright/US-001 tests/playwright/US-002")
QA_EXCLUDE_DIRS empty Subfolders to exclude, separated by spaces (e.g. "api-contract-v1")
PLAYWRIGHT_FLAGS empty Extra flags, for example --project=chromium
PLAYWRIGHT_CONFIG empty Use a different config file (e.g. playwright.config.ci.ts)
PLAYWRIGHT_JSON_OUTPUT test-results/results.json Where the JSON report is saved
QA_AUTH_STATE_PATH .auth/auth-state.json Session file that is checked if auth is google_sso

To run only some folders, use QA_TESTS_DIR or QA_EXCLUDE_DIRS, which filter by the real path. --grep-invert filters by the test title, not by folder.


7. Quick checklist

node -v                       # 18 or higher
npx playwright --version      # Playwright installed
ls node_modules > /dev/null && echo "dependencias OK"
grep -n "results.json" playwright.config.*   # JSON reporter configured
ls .env.tests                 # credentials (if auth is basic_auth / api_token)

If everything checks out, ask the assistant: "Run the suite with lippia-neo-run-and-heal".


8. Common problems (and what to do)

What you see Why it may happen What to do
node_modules no encontrado The dependencies weren't installed npm ci (or npm install)
@playwright/test no encontrado Playwright isn't in package.json npm install -D @playwright/test
Playwright no escribió el JSON en 'test-results/results.json' The runner failed before finishing (config with errors, folder that doesn't exist) Check the output that appears above the error
Auth state no encontrado The Google SSO session doesn't exist or is at a different path Run scripts/save-auth.sh, or set QA_AUTH_STATE_PATH to the real path
"No tests found" testDir or testMatch don't match the generated folder or extension Align testDir with paths.tests_dir and testMatch with .js / .ts
Browser "chrome-for-testing" is not installed while exploring The Playwright MCP was updated and expects a different browser version npx @playwright/mcp@latest install-browser chrome-for-testing (see Playwright MCP setup)
Tests that fail with 5xx or network errors The environment is down; it's not a problem with the test NEO marks it as environment and doesn't fix it: check the environment