Data-driven testing
This guide explains how NEO handles test data when it writes scenarios in Gherkin (.feature): which data is written in the scenario, which goes to a data file, which to an environment variable, and when several cases are grouped in a Scenario Outline. It also explains how the test plan declares that data, so you can review a plan or a suite and know what to expect.
The goal is for the .feature to remain a specification that someone from the business can read: it says what is tested and with which cases, never how each piece of data is loaded.
Where it applies
| Who | What it does with these guidelines |
|---|---|
lippia-neo-create-test-plan |
Designs the data for each scenario: partitions, boundary values, reference data, blocked cases (section 6) |
lippia-neo-generate-test-suite |
Writes new .feature files following these rules |
lippia-neo-refactor-gherkin-features |
Brings an existing Gherkin suite in line with these rules, without changing what it verifies |
Today it applies to the playwright-bdd and playwright-cucumberjs runners. Lippia and Cypress don't use it yet.
1. Declarative steps, one behavior per scenario
The step describes the intent in business language. Selectors, URLs, technical IDs, payloads, and click sequences live in the page objects and the steps, not in the .feature.
Cuando el usuario registra un cliente "con saldo insuficiente"
Cuando completa el campo "saldo" con "-150.00" y hace clic en "Guardar"
- Data setup that isn't what's being tested (creating a customer so you can then invoice them) is a declarative
Given(Dado que existe un cliente "con cuenta corriente") that is resolved through the API or a data file, not through the UI. - A scenario tests one behavior: one block of
Given, one ofWhen, and one ofThen. An intermediateThenthat only checks the precondition becomes aGiven. - If a case mixes two behaviors and only the data changes, it's a
Scenario Outlinewith one more column; if the steps change, they are separate scenarios with the same case tag. - The exception is a sequence described by the acceptance criterion (raising the quantity from 1 to the stock and lowering it back): there,
When/Thenare chained. - As a guideline: a scenario with more than ~7 steps usually has detail that should move down into the code.
2. Where each piece of data goes
For each piece of data in a case, input or expected, use the first row that applies:
| # | The data… | Goes as | Example |
|---|---|---|---|
| 1 | Is secret (password, token, API key) | Environment variable, by name. Never in the .feature or in data/ |
process.env.APP_PASSWORD in the step |
| 2 | Is the rule being tested (a limit, an amount, a status) | Literal in the step or in Examples |
con 101 caracteres, con stock 15 muestra "Pocas unidades" |
| 3 | Must be unique on every run (email, code, ID document) | Generated by the code at run time, never written | The step says un cliente nuevo; the email is generated |
| 4 | Is tied to the user's identity without being secret (display name, role) | Environment variable alongside the credentials (APP_USER_DISPLAY_NAME) |
el encabezado saluda al usuario autenticado |
| 5 | Describes the state of the environment (catalog, stock, prices, and the counts derived from them) | Reference file (section 5). A count is computed from the file, never written | el catálogo muestra todos los productos, not muestra 6 productos |
| 6 | Changes depending on the data profile, or is a set of fields the business doesn't need to read | Business alias, in quotes | Dado un cliente "con saldo insuficiente" |
| 7 | None of the above, short and readable | Literal | el producto "Mouse Inalámbrico" |
Rule of thumb
If the data repeats across two or more scenarios, or changes when the environment changes, it goes to a data file (rows 5 or 6). If it identifies the case, it goes in Examples.
- Row 2 takes precedence over row 5: if the value is precisely the rule (with stock ≤ 15, "Pocas unidades" is shown), it stays in the Gherkin even if it's also in the reference file.
- New variables (rows 1 and 4): if they aren't in the project's
.env.example, their name is added with an empty value.NEOnever reads the file with the real values. - Credentials in
Examples: never a value or the name of a variable. Use a column with an alias (credenciales="usuario válido con contraseña incorrecta") that the code resolves. - Data Tables: they're still useful for verifying the case's own values as a block (the products a search returns). If the table would list the full state of a domain (the whole catalog), that's row 5: the step names it and the values come from the reference file.
An alias describes the data by what it represents for the case (con saldo insuficiente, mayorista), not by its value.
3. Scenario Outline: cases that differ only in data
When several cases of the same flow have the same structure (same precondition, same action, same kind of result) and only the data changes, one Scenario Outline is written instead of repeated scenarios. Also when a case comes with a matrix of combinations (only the ones the source provides; others are never added).
Scenario Outline: <id> — <caso>
Given el usuario está en el alta de proyectos
When el usuario guarda un proyecto con "<campo>" vacío
Then ve el mensaje "<mensaje>"
@HU-31 @TC-04
Examples:
| id | caso | campo | mensaje |
| TC-04 | falta el nombre | nombre | El nombre es obligatorio. |
@HU-31 @TC-05
Examples:
| id | caso | campo | mensaje |
| TC-05 | falta el código | código | El código es obligatorio. |
| Rule | Why |
|---|---|
One Examples block per case, with the @<US-ID> @<case ID> tags above the block. The Outline carries no case tags |
Each case can still be filtered by its tag, just like a Scenario |
Title Scenario Outline: <id> — <caso>, with id and caso columns |
The runner interpolates the title per row: each test is named TC-04 — falta el nombre and ID traceability doesn't change |
The id column is equal to the block's tag |
If they don't match, verification reports an error (L4, section 8) |
| Up to ~8 rows and ~5 columns | Readability. If it goes over, several columns are grouped into an alias |
Rows are always written in the .feature |
Runners count tests from the .feature: rows aren't generated from a CSV or with a preprocessor |
When generating new cases,
NEOnever adds rows to an Outline that already existed: the case goes separately, with a warning. To merge them later, uselippia-neo-refactor-gherkin-features.
4. Data profiles
The data profile is the axis along which the concrete values change without the case changing: environment, customer, tenant, country. There is a single .feature for all profiles; what changes is which data the aliases and references resolve against.
- The active profile is chosen by the run, never by the
.feature. By default it'sdefault. - If an alias or a reference file doesn't exist in the active profile or in
default, the test fails with an error saying what was missing. An invented value is never used.
| Runner | How the profile is chosen |
|---|---|
| playwright-bdd | One project per profile in playwright.config with use: { dataProfile: '<perfil>' }, or the TEST_DATA_PROFILE variable |
| playwright-cucumberjs | One cucumber-js profile per data profile (worldParameters: { dataProfile: '<perfil>' }), run with --profile <perfil>, or the TEST_DATA_PROFILE variable |
5. Data files
The files live in <tests_dir>/data/<perfil>/. What's common to all profiles goes in data/default/. They're only created if some case uses them.
Alias files — <flujo>.json
They're named the same as the .feature (without .feature or .api). One object per alias. A profile only declares the aliases that change relative to default.
{
"con saldo insuficiente": {
"_meta": { "case_type": "invalid", "description": "saldo menor al total de la factura", "expected": "" },
"data": { "nombre": "Cliente Prueba SA", "saldo": "-150.00", "categoria": "minorista" }
}
}
Reference files — <dominio>.json
The expected state of the app (row 5), shared by the .feature files that need it. They're named after the entity (productos, usuarios), never after a flow. A profile's file fully replaces the one in default.
{
"_meta": { "description": "catálogo de productos del ambiente", "source": "plan US-002, datos tomados de la app" },
"records": [
{ "nombre": "Auriculares Wireless Pro", "categoria": "Audio", "precio": "$89.99", "stock": 15 }
]
}
- The records come from the case's source (the plan, the data received, or the
.featurethat already had them): they are never invented._meta.sourcesays where they came from. - If the file already exists, new aliases or records are only added; existing ones are never changed, and a warning is given.
How the steps use them
The code NEO generates includes a data helper (test-data.<ext>) with four functions:
| Function | What it's for |
|---|---|
get(flujo, alias) |
Returns the data of an alias from the active profile (with fallback to default) |
reference(dominio) |
Returns the records of a reference file |
unique(prefijo) |
A unique value per run (row 3) |
uniqueEmail() |
A unique email per run (qa-<id>-<n>@example.com) |
Then('el catálogo muestra todos los productos', async ({ testData, catalogoPage }) => {
const productos = testData.reference('productos'); // data/<perfil>/productos.json
await expect(catalogoPage.nombres).toHaveText(productos.map((p) => p.nombre));
});
6. From the test plan to Gherkin
Data design starts in the plan. lippia-neo-create-test-plan doesn't repeat cases: for each input it builds equivalence partitions and boundary values, picks one value per partition, and marks it as válido, inválido, or borde.
| Input | AC | Partition | Type | Data | Scenario |
|---|---|---|---|---|---|
| cantidad | CA-04 | minimum (1) | borde | 1 |
TC-06 |
| cantidad | CA-04 | maximum (the stock) | borde | the product's stock | TC-06 |
| cantidad | CA-04 | below the minimum | inválido | 0 |
TC-07 |
| cantidad | CA-04 | above the maximum | inválido | the product's stock + 1 | TC-08 |
Also:
- Negative and boundary cases always proposed for forms and text fields (empty, only one filled in, invalid, spaces, accents, special characters, maximum length). If they go beyond the acceptance criterion, they're left as
**Alcance:** sugerido. - Reference data declared only once per domain in "Datos de prueba"; scenarios name it without copying values.
- Derived data as a reference to its origin ("up to the product's stock"), never the computed value ("up to 8").
- Credentials only by the name of the environment variable.
Each scenario in the plan has a scope and a status, and each piece of data a recognizable form. This is how they translate when the suite is generated:
| In the plan | What NEO generates |
|---|---|
**Alcance:** sugerido |
Scenario with the @sugerido tag |
**Estado:** bloqueado — <motivo> (ver pregunta abierta N) |
Blocked case (section 7) |
**Estado:** sin resultado esperado (or a step with ⚠️ sin resultado esperado) |
Not generated: it stays in skipped_scenarios until someone confirms the result |
`teclado` — partición: coincide por nombre — tipo: válido (literal) |
An Examples row; the partition goes in the caso column |
el stock del producto + 1 — partición: … (derived) |
A row with a column that names the derivation (cantidad = stock + 1); the computed value is never written |
pendiente (ver pregunta abierta N) |
The case is blocked by data |
referencia: <qué> del dominio <d> |
The step reads the reference file <d>.json |
usuario: APP_USER (variable) |
Environment variable by name |
Table whose first column is Dominio |
Reference file data/default/<dominio>.json. An empty cell stays null, it's never filled in |
A
bloqueadoorsin resultado esperadoscenario is also not run in manual exploration, nor exported to the test management tool. Seelippia-neo-read-test-plan.
7. Blocked and suggested cases
A case is blocked by data when the plan marks it that way or when the data it needs isn't in any source. Data is never invented to unblock it. A suggested case is one that goes beyond the acceptance criterion.
| playwright-bdd | playwright-cucumberjs | |
|---|---|---|
| Blocked case | Still written in the .feature, with the @fixme tag (it doesn't run and the report shows it as skipped) |
Not written: this runner has no marker for it. It only shows up in skipped_scenarios |
| Suggested case | @sugerido tag; runs by default |
@sugerido tag; runs by default |
| Exclude suggested cases | npx playwright test --grep-invert @sugerido |
npx cucumber-js --tags "not @sugerido" |
A blocked case in playwright-bdd looks like this, so the omission is visible in the suite and in the report, not just in the plan:
# bloqueado: no hay productos con stock 0 en el ambiente — ver US-002-test-plan.md, pregunta abierta 2
@US-002 @TC-04 @fixme
Scenario: TC-04 — Un producto sin stock tiene deshabilitado "Añadir al carrito"
Given existe un producto sin stock
Then no se puede agregar al carrito
Nobody removes the marker until the data is resolved
A @fixme scenario with # bloqueado: is an intentional exclusion: lippia-neo-run-and-heal doesn't repair it and nobody should complete it until the plan resolves the data. That's different from a regression test, which never carries @fixme: it stays red until the bug is fixed.
In playwright-bdd, if all the scenarios in a
.featureare@fixme, no test is generated for that file and the case doesn't show up in the report until there's one that runs.
8. Verification
NEO verifies the written suite with a data lint that doesn't run anything and always exits with code 0 (it reports findings, it doesn't stop the flow). lippia-neo-generate-test-suite runs it when it finishes, and lippia-neo-refactor-gherkin-features runs it before and after restructuring.
| Check | Severity | What it detects |
|---|---|---|
| L1 | warning | The same value repeated in two or more scenarios |
| L2 | warning | A count written in a step (muestra 6 productos) |
| L3 | warning | A domain value written in the step code |
| L4 | error | A row's id column doesn't match the tag of its Examples block |
| L5 | warning | Two rows with the same input in different cases (they add no coverage) |
| L6 | warning | A plan case with no scenario in the suite (only if the plan is passed in) |
| L7 | warning | A When after a Then (the scenario tests more than one behavior) |
error findings are fixed before delivering the suite. When the cause of a warning is in the plan (L5, or an L6 the plan doesn't explain), the fix goes in the plan, not in the suite.
9. What is not done
- Values from a specific profile or the state of the environment are not written in the
.feature. - Domain values (names, amounts, lists of possible values) are not written in the step code.
Examplesrows are not generated from external files, and no build step that rewrites the.featurefiles is added.- Secrets are not put in
Examplesor indata/. - Data is not invented: whatever the source doesn't provide stays empty, blocked, or as an open question.