Skip to content

Automated test execution

These four skills take care of the automated part of the flow: generating the executable suite in the project's framework, running it, and automatically repairing what fails for minor reasons —whether based on a real failed run or on a PR diff that broke selectors—, and restructuring an existing Gherkin suite so that it is data-driven.


lippia-neo-generate-test-suite

When to use it

When a test plan exists and you need to generate the executable suite in the configured framework (it reads qa-stack.yaml to choose the adapter: Playwright, playwright-bdd, playwright-cucumberjs, Lippia, or Cypress; Pytest is in stub state and aborts). It supports FE, BE, or hybrid suites. It also covers alternative modes without a full plan: TDD mode (a single acceptance criterion) and "scenario expansion" mode (only the missing cases). Key rule: if any executed scenario failed (status: "fail"), generating its regression test is mandatory and unconditional regardless of the active mode — the only exception is if a file with the bug-id tag of that same bug already exists. Before generating each case, it checks whether an existing test already covers the same flow (precondition, action, and result) so as not to duplicate it. It doesn't write reports: that's done by lippia-neo-write-generation-report.

Gherkin suites (playwright-bdd and playwright-cucumberjs)

With these two frameworks, each case is a Scenario inside the .feature of its business flow (there is no file per case and no folders per story), and the data follows the guidelines in Data-driven testing: cases that only differ in data go into a Scenario Outline with one Examples block per case, sugerido cases get @sugerido, bloqueado cases are left with @fixme (playwright-bdd) or are not written (playwright-cucumberjs), and sin resultado esperado cases are not generated. It never adds rows to a Scenario Outline that already existed. When it finishes, it runs the data lint (L1–L7). See playwright-bdd and playwright-cucumberjs.

Invocation example
{
  "config": {
    "test_framework": "playwright",
    "app_type": "web",
    "tests_dir": "tests/",
    "test_plans_dir": "test-plans/",
    "auth": "basic_auth",
    "language": "javascript"
  }
}
Input Type Description
config.test_framework, config.app_type, config.tests_dir, config.test_plans_dir, config.auth, config.language Required All resolved from qa-stack.yaml.
config.env_file Optional Path to the environment's credentials/tokens; if it's null, the templates leave a TODO instead of the real environment variable.
config.lippia_config_path Optional Only relevant if test_framework is Lippia.
input.manual_execution_report_path Optional Handoff artifact (ui_selectors, executed_scenarios, findings, summary) produced by any skill that reproduces or explores the app.
input.tdd_single_ac Optional TDD mode: string with the AC-ID; it generates only that test.
input.existing_tests Optional Output of lippia-neo-find-existing-tests; if there are matches from the same module, the new files are added to that same folder.
input.missing_cases Optional Output of lippia-neo-review-test-coverage; if present (even if empty), it enables "scenario expansion" mode.
input.test_plan Optional The already structured plan (typed output of lippia-neo-read-test-plan). If present, the scope, state, and data of each case come from here instead of reading the Markdown.
input.test_data Optional Data records per case (scenario_id, _meta with case_type/description/expected, data). Each record is a row of the case; description is used as the caso column of the Examples. If absent, the data comes from the plan.

Performance frameworks

With test_framework: k6 or jmeter this skill doesn't apply: it aborts with status: "aborted" without generating anything, with the reason framework_not_supported_by_this_skill. Those frameworks are handled with the performance skills.

Output

Field Description
status success / partial (some case skipped) / aborted (adapter in stub state or plan not found).
generated_files path, test_case_id, suite_type (fe/be) of each generated file. In Gherkin there is one entry per case and path is the flow's .feature: several entries can share the same path.
skipped_scenarios Skipped cases, with test_case_id and reason (for example, already covered by an existing test, filtered out by TDD mode, bloqueado por datos: …, or no expected result).
warnings Pending TODOs or other degradations. In Gherkin, also the support files created or modified (steps, page objects, glue, and data/ files) and the data lint findings.
Chains with

Consumes existing_tests, missing_cases, and manual_execution_report_path from the previous skills; its output feeds lippia-neo-run-and-heal and then lippia-neo-write-generation-report/ lippia-neo-write-test-report.


lippia-neo-run-and-heal

When to use it

To run the test suite (in any configured framework) and auto-repair the ones that fail until they all pass or the maximum number of iterations is reached (it reads qa-stack.yaml to choose the adapter: Playwright, playwright-bdd, playwright-cucumberjs, Lippia, or Cypress; Pytest is in stub state and aborts). It never exceeds max_iterations (cap of 3); before escalating, it completes all pending TODOs and replaces hardcoded credentials with environment variables — neither of these is escalated as an excuse; they're always resolved.

Gherkin suites (playwright-bdd and playwright-cucumberjs)

Selector and timeout fixes go to the page object, and assertion fixes go to the step definition: it never changes the text of a step in the .feature to make it match. A missing, duplicate, or ambiguous step is treated as incomplete code (it's completed in steps/), not as a red test. A case blocked by data (@fixme with # bloqueado:) is not repaired.

Publish the result to LTM

In Lippia projects, the run can be published to Lippia Test Manager while it runs, with QA_RESULTS_EXPORT=ltm. The skill doesn't turn it on in the fixing iterations: one final run is published. If the export fails, it escalates it as environment. See Configure Lippia.

Invocation example
{
  "config": {
    "test_framework": "playwright",
    "tests_dir": "tests/",
    "reports_dir": "test-results/reports/",
    "commands": {
      "run_tests": "npx playwright test"
    }
  }
}
Input Type Description
config.test_framework, config.tests_dir, config.reports_dir, config.commands.run_tests Required
config.max_iterations Optional Defaults to 3; use 1 for a post-deploy smoke test without healing.
config.plateau_detection Optional Defaults to true; pass false when max_iterations=1.
config.run_label Optional Label for the run in the report (post-fix, regresion-modulo, smoke).
config.include_dirs / config.exclude_dirs Optional Subfolders to include/exclude (Playwright, playwright-bdd, playwright-cucumberjs, and Cypress, mutually exclusive). In the Gherkin frameworks they are subfolders of features/.
config.target_tests Optional Specific files to run instead of the full suite (strings or objects with path; e.g. existing_tests from lippia-neo-find-existing-tests or repaired_tests from lippia-neo-repair-selectors-by-diff); takes priority over include/exclude_dirs. Playwright, playwright-bdd, playwright-cucumberjs, and Cypress (in the Gherkin ones, a .feature runs all the scenarios of its flow) — in Lippia it's ignored (it doesn't support scoping by folder or file) and this is noted in escalation_notes.

Performance frameworks

With test_framework: k6 or jmeter this skill doesn't apply: it aborts without running anything, with the reason framework_not_supported_by_this_skill. Those frameworks are handled with the performance skills.

Output

Field Description
status green / plateau_abort / max_iter_abort.
iterations_used, final_pass_count, final_fail_count
healed_tests file, failure_category (selector_broken/timeout/assertion_mismatch/api_error/environment/unknown), fix_applied, fix_successful.
escalation_required, escalation_notes If human intervention is needed: tests that keep failing, attempted fixes, and a concrete suggestion.
report_path JSON of the run, written to reports_dir.
Chains with

Runs after lippia-neo-generate-test-suite (or directly on existing tests); when escalation_required is true, that result is reported to a human via lippia-neo-write-test-report. It's also step 3 of the ui-selector-repair plan and the only step of ui-selector-regression, scoped with target_tests to the tests that lippia-neo-repair-selectors-by-diff repaired.


lippia-neo-repair-selectors-by-diff

When to use it

Right after lippia-neo-find-ui-tests-by-code-diff, in a selector repair flow: it statically repairs the selectors of the tests returned in matched_tests, rewriting them based on what the diff itself changed (testing attributes, aria-label, role, id, route, class, or renamed/removed text). If the diff adds a testing attribute (data-testid, data-test, data-qa, data-pw) to an element that the test was looking for by class, text, or named role, it migrates the selector to that attribute ("promotion"). It doesn't run the suite or open a browser — the PR isn't deployed to any environment yet, so the diff is the only source of truth. It never "guesses" a replacement: if the diff doesn't establish an unambiguous old→new relationship, the case stays in unresolved for human review.

Invocation example
{
  "input": {
    "pr_diff_path": "tmp/pr-1234.diff",
    "matched_tests": [
      {
        "path": "tests/checkout.spec.ts",
        "suite_type": "fe",
        "matched_on": [
          { "term": "apply-coupon", "matched_location": "structured_body", "match_strength": "partial", "match_mode": "exact" }
        ]
      }
    ]
  },
  "config": {
    "test_framework": "playwright"
  }
}
Input Type Description
input.pr_diff_path Required The same diff that lippia-neo-find-ui-tests-by-code-diff consumed.
input.matched_tests Required The array returned by that skill, never rebuilt by hand; all the terms of each file are evaluated. If it's empty, it returns none without opening anything.
config.test_framework Required Determines what a selector looks like in the test code (playwright/playwright-bdd/playwright-cucumberjs/lippia/pytest; with any other framework, including cypress, it aborts with a warning). In the Gherkin frameworks it repairs page objects and steps, never the text of a .feature.

Output

Field Description
status repaired (everything resolved) / partial (there are replacements and also unresolved) / blocked (only unresolved) / none (nothing mechanically repairable).
repaired_tests Per file, changes with selector_type, old_value, new_value, occurrences, and old_selector_type only when there was a promotion (the selector type changed).
unresolved path, matched_on (the term, as a string), reason.
warnings Scope-only matches with nothing to repair, applied selector promotions, traceability of received/evaluated/repaired/unresolved terms.
Chains with

Step 2 of the ui-selector-repair plan, gated by has_matches from lippia-neo-find-ui-tests-by-code-diff; unresolved is the signal to escalate to a human. Its repaired_tests feeds config.target_tests of lippia-neo-run-and-heal (step 3 of the same plan and, after the deploy, the ui-selector-regression plan).


lippia-neo-refactor-gherkin-features

When to use it

When you already have a written Gherkin suite (by hand or with an earlier version of NEO) with scenarios that repeat the same steps with different values, counts written into the steps (muestra 4 productos), or the same catalog copied into several scenarios, and you want to bring it in line with the guidelines in Data-driven testing without changing what it verifies. It unifies the scenarios that only differ in data into a Scenario Outline (with one Examples block per case), and moves the reference data that the suite already has written into data files. It's also useful to see what would change (preview) before touching anything. It's triggered by phrases like "refactor the feature files", "move the Gherkin suite to data-driven", or "unify the repeated scenarios into a Scenario Outline".

Invocation example
{
  "input": {
    "features": ["tests/playwright-bdd/features/catalogo.feature"],
    "apply": false
  }
}
Input Type Description
input.features Required Paths to .feature files or to folders that contain them, relative to the repository root. Only those files are restructured.
input.apply Optional Defaults to false: preview only, it writes nothing. With true, it writes the changes, validates them and, if anything fails, restores everything.

It also reads from qa-stack.yaml the framework (stack.test_framework), the language (stack.language), and the suite folder (paths.tests_dir). It never reads the credentials file (paths.env_file).

Output

Field Description
status preview (there are changes and apply wasn't requested) / applied (they were applied and passed the three validations) / no_changes (there's nothing to restructure; there may be suggestions) / not_supported (the framework isn't a supported Gherkin runner) / blocked (a validation failed and the files were restored) / failed (qa-stack.yaml or Node.js is missing, or there are no .feature files in the paths).
scenario_mapping One item per executable test: where it was (file, line, title, tags) and where it ends up (file, title, kind: scenario or examples_row, changed).
changes Files that are modified or created (feature, steps, glue, or data), with a summary and the diff. Sensitive data appears masked (********).
suggestions What the skill detects but does not apply, with file and line: imperative_steps, multiple_behaviors, count_without_records, repeated_value, secret_in_feature, excluded_scenario, lint.
metrics Before and after: number of .feature files, scenarios, executable tests, steps, lines, and lint findings. after is null if nothing was applied.
validation Result of the three validations (runner, tests, lint). null without apply.
warnings Ignored paths, data files that already existed, masked values and, in playwright-bdd, the notice about the path change in the CI history.

What changes (and what doesn't)

Situation in the suite What it does
Several scenarios with the same steps that only change in quoted values or numbers It unifies them into a Scenario Outline with one Examples block per original scenario, with its tags and comments above the block. Only if the values are case data (not secrets), the steps already receive them as a parameter (or a new one can be added without touching the existing ones), and the titles can be preserved
A written count (muestra 4 productos) whose records are in the suite (a Data Table) It moves the records to data/default/<dominio>.json and changes the step so that it computes the count from the file (el catálogo muestra todos los productos)
A catalog copied into two or more scenarios It moves it to a reference file, with _meta.source pointing to the .feature it came from
A count without records in the suite, or a short value that repeats It doesn't touch it: it stays as a suggestion (count_without_records, repeated_value)
Imperative steps, or a When after a Then It doesn't rewrite them: they stay as a suggestion (imperative_steps, multiple_behaviors)

What it guarantees

  • No test is lost or added. Each test from before ends up as exactly one test afterwards, in the same file, with the same title (in an Outline, the interpolated title of its row) and the same tags.
  • It doesn't change what is verified. Each Then keeps verifying the same thing with the same value. The data that is moved is exactly what was written: it's never invented or filled in.
  • Comments are preserved (bug-id, regression, TDD, # bloqueado:) above their scenario or their Examples block.
  • Other .feature files don't break: it never changes or deletes a step definition used by a file outside input.features. The ones left unused are reported as a suggestion.

What is never unified

  • Scenarios with @fixme, @skip, @only, @fail, @ignore, @wip, or @manual.
  • Regression tests (bug-id: comment) and TDD tests: they are always a test of their own.
  • Scenarios with a written secret (password, token, email): they're left as they are and reported as secret_in_feature so they can be moved to an environment variable by hand. A credentials alias (credenciales "inválidas") or an expected text ("Username and password do not match") doesn't count as a secret.
  • Scenarios with doc strings, and Scenario Outlines that already existed.
  • Scenarios from different .feature files: they're only unified within the same file.

How it validates (with apply: true)

Before writing, it saves a copy of each file. After writing (in order: data, glue, steps, and .feature), it runs three validations, without running the suite against the app:

Validation What it checks playwright-bdd playwright-cucumberjs
runner That there are no missing, duplicate, or ambiguous steps, and that the number of tests doesn't change npx bddgen + npx playwright test --list npx cucumber-js --dry-run
tests That the same tests are there, with the same titles, tags, and comments Before / after comparison of the .feature files Same
lint That the data lint doesn't have more errors than before lint-ddt.mjs Same

If any of them fails, it restores all the files, deletes the created ones, and returns blocked, with the reason as the first warning.

playwright-bdd: CI history

playwright-bdd groups the rows of a Scenario Outline inside a block with the Outline's title. The title of each test doesn't change, but its full path in the report does: if your CI keeps history by full path, it will see those tests as new once. The skill warns about it in warnings.

Example

  @US-003 @TC-03
  Scenario: TC-03 — La búsqueda encuentra un producto por nombre
    When el usuario busca "teclado"
    Then ve el producto "Teclado Mecánico RGB" en los resultados

  @US-003 @TC-04
  Scenario: TC-04 — La búsqueda no distingue mayúsculas
    When el usuario busca "MOUSE"
    Then ve el producto "Mouse Inalámbrico" en los resultados

  # bug-id: BUG-12
  @US-003 @TC-05 @regression
  Scenario: TC-05 — La búsqueda encuentra un producto por categoría
    When el usuario busca "audio"
    Then ve el producto "Auriculares Wireless Pro" en los resultados
  Scenario Outline: <id> — <caso>
    When el usuario busca "<busqueda>"
    Then ve el producto "<producto>" en los resultados

    @US-003 @TC-03
    Examples:
      | id    | caso                                         | busqueda | producto             |
      | TC-03 | La búsqueda encuentra un producto por nombre | teclado  | Teclado Mecánico RGB |

    @US-003 @TC-04
    Examples:
      | id    | caso                                | busqueda | producto          |
      | TC-04 | La búsqueda no distingue mayúsculas | MOUSE    | Mouse Inalámbrico |

  # bug-id: BUG-12
  @US-003 @TC-05 @regression
  Scenario: TC-05 — La búsqueda encuentra un producto por categoría
    When el usuario busca "audio"
    Then ve el producto "Auriculares Wireless Pro" en los resultados

TC-03 and TC-04 become rows of the same Outline and are still called TC-03 — … and TC-04 — …, with their tags. TC-05 is a regression test: it stays as it was.

Supported runners

Currently, playwright-bdd and playwright-cucumberjs. With any other framework it returns not_supported without reading or writing anything. Lippia and Cypress are pending.

Chains with

It isn't part of any playbook: it's invoked directly on an existing suite. A typical flow is to run it first in preview, review the diffs and the suggestions, run it again with apply: true, and then run the suite with lippia-neo-run-and-heal to confirm it's still green against the app.