Automated test execution
These four skills take care of the automated part of the flow: generating the executable suite in the project's framework, running it, and automatically repairing what fails for minor reasons —whether based on a real failed run or on a PR diff that broke selectors—, and restructuring an existing Gherkin suite so that it is data-driven.
lippia-neo-generate-test-suite
When to use it
When a test plan exists and you need to generate the executable suite in the configured
framework (it reads qa-stack.yaml to choose the adapter: Playwright, playwright-bdd,
playwright-cucumberjs, Lippia, or Cypress; Pytest is in stub state and aborts). It supports FE,
BE, or hybrid suites.
It also covers alternative modes without a full plan: TDD mode (a single acceptance criterion)
and "scenario expansion" mode (only the missing cases). Key rule: if any executed scenario
failed (status: "fail"), generating its regression test is mandatory and unconditional
regardless of the active mode — the only exception is if a file with the bug-id tag of that
same bug already exists. Before generating each case, it checks whether an existing test already covers the same flow
(precondition, action, and result) so as not to duplicate it. It doesn't write reports: that's done by
lippia-neo-write-generation-report.
Gherkin suites (playwright-bdd and playwright-cucumberjs)
With these two frameworks, each case is a Scenario inside the .feature of its business
flow (there is no file per case and no folders per story), and the data follows the
guidelines in Data-driven testing: cases that only
differ in data go into a Scenario Outline with one Examples block per case, sugerido cases
get @sugerido, bloqueado cases are left with @fixme (playwright-bdd) or are not written
(playwright-cucumberjs), and sin resultado esperado cases are not generated. It never adds rows
to a Scenario Outline that already existed. When it finishes, it runs the data lint (L1–L7). See
playwright-bdd and
playwright-cucumberjs.
{
"config": {
"test_framework": "playwright",
"app_type": "web",
"tests_dir": "tests/",
"test_plans_dir": "test-plans/",
"auth": "basic_auth",
"language": "javascript"
}
}
| Input | Type | Description |
|---|---|---|
config.test_framework, config.app_type, config.tests_dir, config.test_plans_dir, config.auth, config.language |
Required | All resolved from qa-stack.yaml. |
config.env_file |
Optional | Path to the environment's credentials/tokens; if it's null, the templates leave a TODO instead of the real environment variable. |
config.lippia_config_path |
Optional | Only relevant if test_framework is Lippia. |
input.manual_execution_report_path |
Optional | Handoff artifact (ui_selectors, executed_scenarios, findings, summary) produced by any skill that reproduces or explores the app. |
input.tdd_single_ac |
Optional | TDD mode: string with the AC-ID; it generates only that test. |
input.existing_tests |
Optional | Output of lippia-neo-find-existing-tests; if there are matches from the same module, the new files are added to that same folder. |
input.missing_cases |
Optional | Output of lippia-neo-review-test-coverage; if present (even if empty), it enables "scenario expansion" mode. |
input.test_plan |
Optional | The already structured plan (typed output of lippia-neo-read-test-plan). If present, the scope, state, and data of each case come from here instead of reading the Markdown. |
input.test_data |
Optional | Data records per case (scenario_id, _meta with case_type/description/expected, data). Each record is a row of the case; description is used as the caso column of the Examples. If absent, the data comes from the plan. |
Performance frameworks
With test_framework: k6 or jmeter this skill doesn't apply: it aborts with status: "aborted" without generating anything, with the reason framework_not_supported_by_this_skill. Those frameworks are handled with the performance skills.
Output
| Field | Description |
|---|---|
status |
success / partial (some case skipped) / aborted (adapter in stub state or plan not found). |
generated_files |
path, test_case_id, suite_type (fe/be) of each generated file. In Gherkin there is one entry per case and path is the flow's .feature: several entries can share the same path. |
skipped_scenarios |
Skipped cases, with test_case_id and reason (for example, already covered by an existing test, filtered out by TDD mode, bloqueado por datos: …, or no expected result). |
warnings |
Pending TODOs or other degradations. In Gherkin, also the support files created or modified (steps, page objects, glue, and data/ files) and the data lint findings. |
Chains with
Consumes existing_tests, missing_cases, and manual_execution_report_path from the previous skills; its
output feeds lippia-neo-run-and-heal and then lippia-neo-write-generation-report/
lippia-neo-write-test-report.
lippia-neo-run-and-heal
When to use it
To run the test suite (in any configured framework) and auto-repair the ones that
fail until they all pass or the maximum number of iterations is reached (it reads qa-stack.yaml to
choose the adapter: Playwright, playwright-bdd, playwright-cucumberjs, Lippia, or Cypress; Pytest
is in stub state and aborts). It never
exceeds max_iterations (cap of 3); before escalating, it completes all
pending TODOs and replaces hardcoded credentials with environment variables — neither
of these is escalated as an excuse; they're always resolved.
Gherkin suites (playwright-bdd and playwright-cucumberjs)
Selector and timeout fixes go to the page object, and assertion fixes go to the step definition:
it never changes the text of a step in the .feature to make it match. A missing,
duplicate, or ambiguous step is treated as incomplete code (it's completed in steps/), not as a
red test. A case blocked by data (@fixme with # bloqueado:) is not repaired.
Publish the result to LTM
In Lippia projects, the run can be published to Lippia Test Manager while it runs, with
QA_RESULTS_EXPORT=ltm. The skill doesn't turn it on in the fixing iterations: one final
run is published. If the export fails, it escalates it as environment.
See Configure Lippia.
{
"config": {
"test_framework": "playwright",
"tests_dir": "tests/",
"reports_dir": "test-results/reports/",
"commands": {
"run_tests": "npx playwright test"
}
}
}
| Input | Type | Description |
|---|---|---|
config.test_framework, config.tests_dir, config.reports_dir, config.commands.run_tests |
Required | |
config.max_iterations |
Optional | Defaults to 3; use 1 for a post-deploy smoke test without healing. |
config.plateau_detection |
Optional | Defaults to true; pass false when max_iterations=1. |
config.run_label |
Optional | Label for the run in the report (post-fix, regresion-modulo, smoke). |
config.include_dirs / config.exclude_dirs |
Optional | Subfolders to include/exclude (Playwright, playwright-bdd, playwright-cucumberjs, and Cypress, mutually exclusive). In the Gherkin frameworks they are subfolders of features/. |
config.target_tests |
Optional | Specific files to run instead of the full suite (strings or objects with path; e.g. existing_tests from lippia-neo-find-existing-tests or repaired_tests from lippia-neo-repair-selectors-by-diff); takes priority over include/exclude_dirs. Playwright, playwright-bdd, playwright-cucumberjs, and Cypress (in the Gherkin ones, a .feature runs all the scenarios of its flow) — in Lippia it's ignored (it doesn't support scoping by folder or file) and this is noted in escalation_notes. |
Performance frameworks
With test_framework: k6 or jmeter this skill doesn't apply: it aborts without running anything, with the reason framework_not_supported_by_this_skill. Those frameworks are handled with the performance skills.
Output
| Field | Description |
|---|---|
status |
green / plateau_abort / max_iter_abort. |
iterations_used, final_pass_count, final_fail_count |
|
healed_tests |
file, failure_category (selector_broken/timeout/assertion_mismatch/api_error/environment/unknown), fix_applied, fix_successful. |
escalation_required, escalation_notes |
If human intervention is needed: tests that keep failing, attempted fixes, and a concrete suggestion. |
report_path |
JSON of the run, written to reports_dir. |
Chains with
Runs after lippia-neo-generate-test-suite (or directly on existing tests); when
escalation_required is true, that result is reported to a human via
lippia-neo-write-test-report. It's also step 3 of the ui-selector-repair plan and the only
step of ui-selector-regression, scoped with target_tests to the tests that
lippia-neo-repair-selectors-by-diff repaired.
lippia-neo-repair-selectors-by-diff
When to use it
Right after lippia-neo-find-ui-tests-by-code-diff, in a selector repair flow:
it statically repairs the selectors of the tests returned
in matched_tests, rewriting them based on what the diff itself changed (testing attributes,
aria-label, role, id, route, class, or renamed/removed text). If the diff adds a testing
attribute (data-testid, data-test, data-qa, data-pw) to an element that the test
was looking for by class, text, or named role, it migrates the selector to that attribute ("promotion").
It doesn't run the suite or open a browser — the PR isn't deployed to any environment yet, so
the diff is the only source of truth. It never "guesses" a replacement: if the diff doesn't establish an unambiguous
old→new relationship, the case stays in unresolved for human review.
{
"input": {
"pr_diff_path": "tmp/pr-1234.diff",
"matched_tests": [
{
"path": "tests/checkout.spec.ts",
"suite_type": "fe",
"matched_on": [
{ "term": "apply-coupon", "matched_location": "structured_body", "match_strength": "partial", "match_mode": "exact" }
]
}
]
},
"config": {
"test_framework": "playwright"
}
}
| Input | Type | Description |
|---|---|---|
input.pr_diff_path |
Required | The same diff that lippia-neo-find-ui-tests-by-code-diff consumed. |
input.matched_tests |
Required | The array returned by that skill, never rebuilt by hand; all the terms of each file are evaluated. If it's empty, it returns none without opening anything. |
config.test_framework |
Required | Determines what a selector looks like in the test code (playwright/playwright-bdd/playwright-cucumberjs/lippia/pytest; with any other framework, including cypress, it aborts with a warning). In the Gherkin frameworks it repairs page objects and steps, never the text of a .feature. |
Output
| Field | Description |
|---|---|
status |
repaired (everything resolved) / partial (there are replacements and also unresolved) / blocked (only unresolved) / none (nothing mechanically repairable). |
repaired_tests |
Per file, changes with selector_type, old_value, new_value, occurrences, and old_selector_type only when there was a promotion (the selector type changed). |
unresolved |
path, matched_on (the term, as a string), reason. |
warnings |
Scope-only matches with nothing to repair, applied selector promotions, traceability of received/evaluated/repaired/unresolved terms. |
Chains with
Step 2 of the ui-selector-repair plan, gated by has_matches from
lippia-neo-find-ui-tests-by-code-diff; unresolved is the signal to escalate to a human. Its
repaired_tests feeds config.target_tests of lippia-neo-run-and-heal (step 3 of the same
plan and, after the deploy, the ui-selector-regression plan).
lippia-neo-refactor-gherkin-features
When to use it
When you already have a written Gherkin suite (by hand or with an earlier version of NEO) with
scenarios that repeat the same steps with different values, counts written into the steps
(muestra 4 productos), or the same catalog copied into several scenarios, and you want to bring it
in line with the guidelines in Data-driven testing without
changing what it verifies. It unifies the scenarios that only differ in data into a Scenario
Outline (with one Examples block per case), and moves the reference data that the suite already
has written into data files. It's also useful to see what would change (preview) before
touching anything.
It's triggered by phrases like "refactor the feature files", "move the Gherkin suite to
data-driven", or "unify the repeated scenarios into a Scenario Outline".
{
"input": {
"features": ["tests/playwright-bdd/features/catalogo.feature"],
"apply": false
}
}
| Input | Type | Description |
|---|---|---|
input.features |
Required | Paths to .feature files or to folders that contain them, relative to the repository root. Only those files are restructured. |
input.apply |
Optional | Defaults to false: preview only, it writes nothing. With true, it writes the changes, validates them and, if anything fails, restores everything. |
It also reads from qa-stack.yaml the framework (stack.test_framework), the language (stack.language), and
the suite folder (paths.tests_dir). It never reads the credentials file (paths.env_file).
Output
| Field | Description |
|---|---|
status |
preview (there are changes and apply wasn't requested) / applied (they were applied and passed the three validations) / no_changes (there's nothing to restructure; there may be suggestions) / not_supported (the framework isn't a supported Gherkin runner) / blocked (a validation failed and the files were restored) / failed (qa-stack.yaml or Node.js is missing, or there are no .feature files in the paths). |
scenario_mapping |
One item per executable test: where it was (file, line, title, tags) and where it ends up (file, title, kind: scenario or examples_row, changed). |
changes |
Files that are modified or created (feature, steps, glue, or data), with a summary and the diff. Sensitive data appears masked (********). |
suggestions |
What the skill detects but does not apply, with file and line: imperative_steps, multiple_behaviors, count_without_records, repeated_value, secret_in_feature, excluded_scenario, lint. |
metrics |
Before and after: number of .feature files, scenarios, executable tests, steps, lines, and lint findings. after is null if nothing was applied. |
validation |
Result of the three validations (runner, tests, lint). null without apply. |
warnings |
Ignored paths, data files that already existed, masked values and, in playwright-bdd, the notice about the path change in the CI history. |
What changes (and what doesn't)
| Situation in the suite | What it does |
|---|---|
| Several scenarios with the same steps that only change in quoted values or numbers | It unifies them into a Scenario Outline with one Examples block per original scenario, with its tags and comments above the block. Only if the values are case data (not secrets), the steps already receive them as a parameter (or a new one can be added without touching the existing ones), and the titles can be preserved |
A written count (muestra 4 productos) whose records are in the suite (a Data Table) |
It moves the records to data/default/<dominio>.json and changes the step so that it computes the count from the file (el catálogo muestra todos los productos) |
| A catalog copied into two or more scenarios | It moves it to a reference file, with _meta.source pointing to the .feature it came from |
| A count without records in the suite, or a short value that repeats | It doesn't touch it: it stays as a suggestion (count_without_records, repeated_value) |
Imperative steps, or a When after a Then |
It doesn't rewrite them: they stay as a suggestion (imperative_steps, multiple_behaviors) |
What it guarantees
- No test is lost or added. Each test from before ends up as exactly one test afterwards, in the same file, with the same title (in an Outline, the interpolated title of its row) and the same tags.
- It doesn't change what is verified. Each
Thenkeeps verifying the same thing with the same value. The data that is moved is exactly what was written: it's never invented or filled in. - Comments are preserved (
bug-id, regression, TDD,# bloqueado:) above their scenario or theirExamplesblock. - Other
.featurefiles don't break: it never changes or deletes a step definition used by a file outsideinput.features. The ones left unused are reported as a suggestion.
What is never unified
- Scenarios with
@fixme,@skip,@only,@fail,@ignore,@wip, or@manual. - Regression tests (
bug-id:comment) and TDD tests: they are always a test of their own. - Scenarios with a written secret (password, token, email): they're left as they are and
reported as
secret_in_featureso they can be moved to an environment variable by hand. A credentials alias (credenciales "inválidas") or an expected text ("Username and password do not match") doesn't count as a secret. - Scenarios with doc strings, and
Scenario Outlines that already existed. - Scenarios from different
.featurefiles: they're only unified within the same file.
How it validates (with apply: true)
Before writing, it saves a copy of each file. After writing (in order: data, glue,
steps, and .feature), it runs three validations, without running the suite against the app:
| Validation | What it checks | playwright-bdd | playwright-cucumberjs |
|---|---|---|---|
runner |
That there are no missing, duplicate, or ambiguous steps, and that the number of tests doesn't change | npx bddgen + npx playwright test --list |
npx cucumber-js --dry-run |
tests |
That the same tests are there, with the same titles, tags, and comments | Before / after comparison of the .feature files |
Same |
lint |
That the data lint doesn't have more errors than before | lint-ddt.mjs |
Same |
If any of them fails, it restores all the files, deletes the created ones, and returns blocked, with the
reason as the first warning.
playwright-bdd: CI history
playwright-bdd groups the rows of a Scenario Outline inside a block with the Outline's
title. The title of each test doesn't change, but its full path in the report does: if your CI
keeps history by full path, it will see those tests as new once. The skill warns about it in
warnings.
Example
@US-003 @TC-03
Scenario: TC-03 — La búsqueda encuentra un producto por nombre
When el usuario busca "teclado"
Then ve el producto "Teclado Mecánico RGB" en los resultados
@US-003 @TC-04
Scenario: TC-04 — La búsqueda no distingue mayúsculas
When el usuario busca "MOUSE"
Then ve el producto "Mouse Inalámbrico" en los resultados
# bug-id: BUG-12
@US-003 @TC-05 @regression
Scenario: TC-05 — La búsqueda encuentra un producto por categoría
When el usuario busca "audio"
Then ve el producto "Auriculares Wireless Pro" en los resultados
Scenario Outline: <id> — <caso>
When el usuario busca "<busqueda>"
Then ve el producto "<producto>" en los resultados
@US-003 @TC-03
Examples:
| id | caso | busqueda | producto |
| TC-03 | La búsqueda encuentra un producto por nombre | teclado | Teclado Mecánico RGB |
@US-003 @TC-04
Examples:
| id | caso | busqueda | producto |
| TC-04 | La búsqueda no distingue mayúsculas | MOUSE | Mouse Inalámbrico |
# bug-id: BUG-12
@US-003 @TC-05 @regression
Scenario: TC-05 — La búsqueda encuentra un producto por categoría
When el usuario busca "audio"
Then ve el producto "Auriculares Wireless Pro" en los resultados
TC-03 and TC-04 become rows of the same Outline and are still called TC-03 — … and TC-04 — …,
with their tags. TC-05 is a regression test: it stays as it was.
Supported runners
Currently, playwright-bdd and playwright-cucumberjs. With any other framework it returns
not_supported without reading or writing anything. Lippia and Cypress are pending.
Chains with
It isn't part of any playbook: it's invoked directly on an existing suite. A typical
flow is to run it first in preview, review the diffs and the suggestions, run it again with
apply: true, and then run the suite with lippia-neo-run-and-heal to confirm it's still
green against the app.