Skip to content

Data-driven testing

This guide explains how NEO handles test data when it writes scenarios in Gherkin (.feature): which data is written in the scenario, which goes to a data file, which to an environment variable, and when several cases are grouped in a Scenario Outline. It also explains how the test plan declares that data, so you can review a plan or a suite and know what to expect.

The goal is for the .feature to remain a specification that someone from the business can read: it says what is tested and with which cases, never how each piece of data is loaded.

Where it applies

Who What it does with these guidelines
lippia-neo-create-test-plan Designs the data for each scenario: partitions, boundary values, reference data, blocked cases (section 6)
lippia-neo-generate-test-suite Writes new .feature files following these rules
lippia-neo-refactor-gherkin-features Brings an existing Gherkin suite in line with these rules, without changing what it verifies

Today it applies to the playwright-bdd and playwright-cucumberjs runners. Lippia and Cypress don't use it yet.


1. Declarative steps, one behavior per scenario

The step describes the intent in business language. Selectors, URLs, technical IDs, payloads, and click sequences live in the page objects and the steps, not in the .feature.

Cuando el usuario registra un cliente "con saldo insuficiente"
Cuando completa el campo "saldo" con "-150.00" y hace clic en "Guardar"
  • Data setup that isn't what's being tested (creating a customer so you can then invoice them) is a declarative Given (Dado que existe un cliente "con cuenta corriente") that is resolved through the API or a data file, not through the UI.
  • A scenario tests one behavior: one block of Given, one of When, and one of Then. An intermediate Then that only checks the precondition becomes a Given.
  • If a case mixes two behaviors and only the data changes, it's a Scenario Outline with one more column; if the steps change, they are separate scenarios with the same case tag.
  • The exception is a sequence described by the acceptance criterion (raising the quantity from 1 to the stock and lowering it back): there, When / Then are chained.
  • As a guideline: a scenario with more than ~7 steps usually has detail that should move down into the code.

2. Where each piece of data goes

For each piece of data in a case, input or expected, use the first row that applies:

# The data… Goes as Example
1 Is secret (password, token, API key) Environment variable, by name. Never in the .feature or in data/ process.env.APP_PASSWORD in the step
2 Is the rule being tested (a limit, an amount, a status) Literal in the step or in Examples con 101 caracteres, con stock 15 muestra "Pocas unidades"
3 Must be unique on every run (email, code, ID document) Generated by the code at run time, never written The step says un cliente nuevo; the email is generated
4 Is tied to the user's identity without being secret (display name, role) Environment variable alongside the credentials (APP_USER_DISPLAY_NAME) el encabezado saluda al usuario autenticado
5 Describes the state of the environment (catalog, stock, prices, and the counts derived from them) Reference file (section 5). A count is computed from the file, never written el catálogo muestra todos los productos, not muestra 6 productos
6 Changes depending on the data profile, or is a set of fields the business doesn't need to read Business alias, in quotes Dado un cliente "con saldo insuficiente"
7 None of the above, short and readable Literal el producto "Mouse Inalámbrico"

Rule of thumb

If the data repeats across two or more scenarios, or changes when the environment changes, it goes to a data file (rows 5 or 6). If it identifies the case, it goes in Examples.

  • Row 2 takes precedence over row 5: if the value is precisely the rule (with stock ≤ 15, "Pocas unidades" is shown), it stays in the Gherkin even if it's also in the reference file.
  • New variables (rows 1 and 4): if they aren't in the project's .env.example, their name is added with an empty value. NEO never reads the file with the real values.
  • Credentials in Examples: never a value or the name of a variable. Use a column with an alias (credenciales = "usuario válido con contraseña incorrecta") that the code resolves.
  • Data Tables: they're still useful for verifying the case's own values as a block (the products a search returns). If the table would list the full state of a domain (the whole catalog), that's row 5: the step names it and the values come from the reference file.

An alias describes the data by what it represents for the case (con saldo insuficiente, mayorista), not by its value.


3. Scenario Outline: cases that differ only in data

When several cases of the same flow have the same structure (same precondition, same action, same kind of result) and only the data changes, one Scenario Outline is written instead of repeated scenarios. Also when a case comes with a matrix of combinations (only the ones the source provides; others are never added).

  Scenario Outline: <id> — <caso>
    Given el usuario está en el alta de proyectos
    When el usuario guarda un proyecto con "<campo>" vacío
    Then ve el mensaje "<mensaje>"

    @HU-31 @TC-04
    Examples:
      | id    | caso            | campo  | mensaje                   |
      | TC-04 | falta el nombre | nombre | El nombre es obligatorio. |

    @HU-31 @TC-05
    Examples:
      | id    | caso            | campo  | mensaje                   |
      | TC-05 | falta el código | código | El código es obligatorio. |
Rule Why
One Examples block per case, with the @<US-ID> @<case ID> tags above the block. The Outline carries no case tags Each case can still be filtered by its tag, just like a Scenario
Title Scenario Outline: <id> — <caso>, with id and caso columns The runner interpolates the title per row: each test is named TC-04 — falta el nombre and ID traceability doesn't change
The id column is equal to the block's tag If they don't match, verification reports an error (L4, section 8)
Up to ~8 rows and ~5 columns Readability. If it goes over, several columns are grouped into an alias
Rows are always written in the .feature Runners count tests from the .feature: rows aren't generated from a CSV or with a preprocessor

When generating new cases, NEO never adds rows to an Outline that already existed: the case goes separately, with a warning. To merge them later, use lippia-neo-refactor-gherkin-features.


4. Data profiles

The data profile is the axis along which the concrete values change without the case changing: environment, customer, tenant, country. There is a single .feature for all profiles; what changes is which data the aliases and references resolve against.

  • The active profile is chosen by the run, never by the .feature. By default it's default.
  • If an alias or a reference file doesn't exist in the active profile or in default, the test fails with an error saying what was missing. An invented value is never used.
Runner How the profile is chosen
playwright-bdd One project per profile in playwright.config with use: { dataProfile: '<perfil>' }, or the TEST_DATA_PROFILE variable
playwright-cucumberjs One cucumber-js profile per data profile (worldParameters: { dataProfile: '<perfil>' }), run with --profile <perfil>, or the TEST_DATA_PROFILE variable

5. Data files

The files live in <tests_dir>/data/<perfil>/. What's common to all profiles goes in data/default/. They're only created if some case uses them.

Alias files — <flujo>.json

They're named the same as the .feature (without .feature or .api). One object per alias. A profile only declares the aliases that change relative to default.

{
  "con saldo insuficiente": {
    "_meta": { "case_type": "invalid", "description": "saldo menor al total de la factura", "expected": "" },
    "data": { "nombre": "Cliente Prueba SA", "saldo": "-150.00", "categoria": "minorista" }
  }
}

Reference files — <dominio>.json

The expected state of the app (row 5), shared by the .feature files that need it. They're named after the entity (productos, usuarios), never after a flow. A profile's file fully replaces the one in default.

{
  "_meta": { "description": "catálogo de productos del ambiente", "source": "plan US-002, datos tomados de la app" },
  "records": [
    { "nombre": "Auriculares Wireless Pro", "categoria": "Audio", "precio": "$89.99", "stock": 15 }
  ]
}
  • The records come from the case's source (the plan, the data received, or the .feature that already had them): they are never invented. _meta.source says where they came from.
  • If the file already exists, new aliases or records are only added; existing ones are never changed, and a warning is given.

How the steps use them

The code NEO generates includes a data helper (test-data.<ext>) with four functions:

Function What it's for
get(flujo, alias) Returns the data of an alias from the active profile (with fallback to default)
reference(dominio) Returns the records of a reference file
unique(prefijo) A unique value per run (row 3)
uniqueEmail() A unique email per run (qa-<id>-<n>@example.com)
Then('el catálogo muestra todos los productos', async ({ testData, catalogoPage }) => {
  const productos = testData.reference('productos');   // data/<perfil>/productos.json
  await expect(catalogoPage.nombres).toHaveText(productos.map((p) => p.nombre));
});

6. From the test plan to Gherkin

Data design starts in the plan. lippia-neo-create-test-plan doesn't repeat cases: for each input it builds equivalence partitions and boundary values, picks one value per partition, and marks it as válido, inválido, or borde.

Input AC Partition Type Data Scenario
cantidad CA-04 minimum (1) borde 1 TC-06
cantidad CA-04 maximum (the stock) borde the product's stock TC-06
cantidad CA-04 below the minimum inválido 0 TC-07
cantidad CA-04 above the maximum inválido the product's stock + 1 TC-08

Also:

  • Negative and boundary cases always proposed for forms and text fields (empty, only one filled in, invalid, spaces, accents, special characters, maximum length). If they go beyond the acceptance criterion, they're left as **Alcance:** sugerido.
  • Reference data declared only once per domain in "Datos de prueba"; scenarios name it without copying values.
  • Derived data as a reference to its origin ("up to the product's stock"), never the computed value ("up to 8").
  • Credentials only by the name of the environment variable.

Each scenario in the plan has a scope and a status, and each piece of data a recognizable form. This is how they translate when the suite is generated:

In the plan What NEO generates
**Alcance:** sugerido Scenario with the @sugerido tag
**Estado:** bloqueado — <motivo> (ver pregunta abierta N) Blocked case (section 7)
**Estado:** sin resultado esperado (or a step with ⚠️ sin resultado esperado) Not generated: it stays in skipped_scenarios until someone confirms the result
`teclado` — partición: coincide por nombre — tipo: válido (literal) An Examples row; the partition goes in the caso column
el stock del producto + 1 — partición: … (derived) A row with a column that names the derivation (cantidad = stock + 1); the computed value is never written
pendiente (ver pregunta abierta N) The case is blocked by data
referencia: <qué> del dominio <d> The step reads the reference file <d>.json
usuario: APP_USER (variable) Environment variable by name
Table whose first column is Dominio Reference file data/default/<dominio>.json. An empty cell stays null, it's never filled in

A bloqueado or sin resultado esperado scenario is also not run in manual exploration, nor exported to the test management tool. See lippia-neo-read-test-plan.


7. Blocked and suggested cases

A case is blocked by data when the plan marks it that way or when the data it needs isn't in any source. Data is never invented to unblock it. A suggested case is one that goes beyond the acceptance criterion.

playwright-bdd playwright-cucumberjs
Blocked case Still written in the .feature, with the @fixme tag (it doesn't run and the report shows it as skipped) Not written: this runner has no marker for it. It only shows up in skipped_scenarios
Suggested case @sugerido tag; runs by default @sugerido tag; runs by default
Exclude suggested cases npx playwright test --grep-invert @sugerido npx cucumber-js --tags "not @sugerido"

A blocked case in playwright-bdd looks like this, so the omission is visible in the suite and in the report, not just in the plan:

  # bloqueado: no hay productos con stock 0 en el ambiente — ver US-002-test-plan.md, pregunta abierta 2
  @US-002 @TC-04 @fixme
  Scenario: TC-04 — Un producto sin stock tiene deshabilitado "Añadir al carrito"
    Given existe un producto sin stock
    Then no se puede agregar al carrito

Nobody removes the marker until the data is resolved

A @fixme scenario with # bloqueado: is an intentional exclusion: lippia-neo-run-and-heal doesn't repair it and nobody should complete it until the plan resolves the data. That's different from a regression test, which never carries @fixme: it stays red until the bug is fixed.

In playwright-bdd, if all the scenarios in a .feature are @fixme, no test is generated for that file and the case doesn't show up in the report until there's one that runs.


8. Verification

NEO verifies the written suite with a data lint that doesn't run anything and always exits with code 0 (it reports findings, it doesn't stop the flow). lippia-neo-generate-test-suite runs it when it finishes, and lippia-neo-refactor-gherkin-features runs it before and after restructuring.

Check Severity What it detects
L1 warning The same value repeated in two or more scenarios
L2 warning A count written in a step (muestra 6 productos)
L3 warning A domain value written in the step code
L4 error A row's id column doesn't match the tag of its Examples block
L5 warning Two rows with the same input in different cases (they add no coverage)
L6 warning A plan case with no scenario in the suite (only if the plan is passed in)
L7 warning A When after a Then (the scenario tests more than one behavior)

error findings are fixed before delivering the suite. When the cause of a warning is in the plan (L5, or an L6 the plan doesn't explain), the fix goes in the plan, not in the suite.


9. What is not done

  • Values from a specific profile or the state of the environment are not written in the .feature.
  • Domain values (names, amounts, lists of possible values) are not written in the step code.
  • Examples rows are not generated from external files, and no build step that rewrites the .feature files is added.
  • Secrets are not put in Examples or in data/.
  • Data is not invented: whatever the source doesn't provide stays empty, blocked, or as an open question.