Skip to content

QA Bug Fix Verification

This guide explains what the NEO engine does when it runs this plan. The idea is that you can read it, understand what happens at each step without anyone stepping in, and know what's needed for the work to turn out well.

In short

  • What it's for: confirming, with tests, that a bug's fix really works, and leaving a report with the "before" (red) and the "after" (green).
  • When it runs: after the developer applied the fix for a bug that previously went through the QA Bug Coverage plan.
  • What it needs: the same bug reference used in QA Bug Coverage, and the test that captures that bug must already exist.
  • What it leaves: the bug's tests run (and repaired if they failed for minor reasons) and a final verification report with the before and after.
  • Steps: 4 — re-read the bug, find its tests, run them until they pass, and report. No pauses.
What is a plan? (if this is the first time you see one)

A plan is a recipe that a machine runs instead of a person. Like a playbook, it chains several skills in a fixed order. The difference is that a playbook is followed by the AI assistant together with you, with pauses for you to decide; a plan is run by the NEO engine on its own, end to end.

See Plans for the full comparison between plan and playbook.


1. What does this plan do?

It's the second half of a bug's cycle: after the fix, it confirms that the test that used to fail now passes. The first half (confirming the bug and creating that test) is done by the QA Bug Coverage plan.

Its intent is that no bug is closed "on someone's word": closing it has to be backed by the same test that proved the bug, now green. To do that, the plan:

  1. Reads the bug again, just as QA Bug Coverage did. It doesn't depend on anything left over from that earlier run: it builds its own context from scratch.
  2. Finds the tests that cover the bug: the test QA Bug Coverage generated, plus the ones that already covered that feature.
  3. Runs only those tests and repairs the ones that fail for minor reasons, until they pass.
  4. Writes the final report, comparing the before (the bug's evidence) with the after (this result).

The plan doesn't apply or suggest any fix: it assumes the developer has already applied it.


2. When does it run?

Situation Business example
The developer applied the fix for a bug that already has its red test TechStore's developer fixed the cart's stock check and deployed it to staging
The bug has to be closed with evidence TechStore's QA lead won't close TS-BUG-31 without a report showing the test green

It's triggered once the fix is ready: in free text (Model 2: the engine chooses this plan), or from the CI pipeline with whatever rule each project defines (for example, a push to the fix branch, or a manual run when the developer considers the fix ready).

Important

  • QA Bug Coverage has to have run first. This plan looks for the test that captures the bug; if it doesn't find it, it stops without running anything.
  • The fix must be deployed at the URL where the tests run. Otherwise, the tests validate the previous version and will stay red.
  • It's best to use the same bug reference as in QA Bug Coverage, so both plans find the same tests and the same identifier.

3. How does it differ from the playbook?

This plan does, with nobody on the other side, the second half of the Bug Fix Cycle playbook:

Bug Fix Cycle playbook This plan
Who runs it The AI assistant, talking with you The engine, on its own
How it knows the fix is ready You tell it, after the pause It runs when CI or a person triggers it, after the fix
Which tests it runs The bug's test and then the module's regression Only the tests that cover the bug (the bug's test plus the ones that already covered that feature), not the whole module
Context Carries it over from the earlier steps of the same conversation Builds it from scratch: it reads the bug again
Report Before and after Before and after, the same

4. Guiding example: "TechStore" fixes the stock check

We continue with the QA Bug Coverage plan's case. At TechStore, the cart let you add more units than were in stock (TS-BUG-31). QA Bug Coverage confirmed the bug and left a red test. Now the developer applied the fix and deployed it to staging: the cart shows an error when the stock is exceeded.

4.1 What the plan needs to receive

Data Required What it is TechStore example
bug_report Yes The reference to the bug: a path to a file, the URL of a Jira or Azure DevOps ticket, or the report text. The same one used in QA Bug Coverage https://techstore.atlassian.net/browse/TS-BUG-31
identifier Yes The bug's identifier TS-BUG-31
credentials Yes Test credentials to log into the application Staging test username and password

4.2 How it's triggered

neo run plans/qa-bug-fix-verification.yaml --input-file input.json --workspace .

Where input.json has the data in the table above. With Model 2, the engine can also choose this plan from a request like "bug TS-BUG-31 is fixed, verify it".


4.3 Step by step

Step 1 of 4 — Read the bug again

Skill used: lippia-neo-read-bug-report

What happens: it reads the bug report just as QA Bug Coverage did, to recover its identifier, its acceptance criteria, and the application's URL. It doesn't reuse anything from the earlier run: that way the plan works even if it runs days later.


Step 2 of 4 — Find the bug's tests

Skill used: lippia-neo-find-existing-tests

What happens: it searches the project's tests folder for the ones that cover the bug. The test generated by QA Bug Coverage should show up there, plus the ones that already covered that feature.

With TechStore: it finds the new "don't allow exceeding stock" test and the cart tests that already existed.

Stopping point: no tests, nothing to verify

If it doesn't find any test covering the bug, the plan stops here (stopped). Running other tests wouldn't prove the reported bug was fixed. It usually means QA Bug Coverage hasn't run for that bug yet.


Step 3 of 4 — Run the tests until they pass

Skill used: lippia-neo-run-and-heal

What happens: it runs only the tests Step 2 found. If one fails for a minor reason (a selector that changed with the fix, a short wait), it repairs it and runs again, up to 3 times.

Result What it means
✅ All tests pass The fix works: the bug's test went from red to green
❌ Some test keeps failing The fix doesn't resolve the bug, or broke something else in that feature

It repairs tests, not the application

If the bug's test keeps failing because the bug still happens, this step doesn't "fix" it so that it passes: it stays as a failure in the report.

With Lippia the whole module runs

Running only some tests works with Playwright. If the project uses Lippia, it can't be narrowed down by file and the whole module runs.


Step 4 of 4 — Final report: before and after

Skill used: lippia-neo-write-test-report

What happens: it writes the verification report comparing two moments:

  • Before the fix: the evidence that the bug reproduced, left by QA Bug Coverage.
  • After the fix: the result of Step 3.

The report is always written, whether the tests pass or not: it has to reflect the real result, not just the happy path.

What you get: the report, with date and time, in the project's reports folder (by default test-results/reports/).


4.4 How does it end?

How it ended Plan status What it means What to do
✅ The tests pass success The fix works Close the bug, attaching the report
🔴 Some test keeps failing success The plan worked, but the fix isn't enough Review the report with the developer and run again after the new fix
⏹ There were no tests covering the bug stopped There's nothing to verify Run QA Bug Coverage first
❌ A skill returned something malformed, or something broke failed / error The step failed, not the application Check the log and result.json

Red tests aren't an engine error

If the fix doesn't work, the plan still ends in success: the engine did its job and the test results are in the report and in result.json. If the pipeline has to be blocked in that case, the rule is defined by CI, reading that result.


5. Common problems (and what to do)

What you see Why it can happen What to do
The plan ends stopped at Step 2 QA Bug Coverage hasn't run for this bug yet, or a different bug reference was used Run QA Bug Coverage first, or use the same reference used there
The bug's test is still red The fix isn't deployed at the test URL, or doesn't resolve the bug Confirm the deployment; if it's deployed, review the report with the developer
Other tests fail besides the bug's The fix broke another part of the same feature Don't close the bug; review with the developer
The report doesn't show the "before" The reproduction evidence left by QA Bug Coverage wasn't found Confirm QA Bug Coverage ran in the same project and with the same bug identifier

6. Visual summary of the flow

flowchart TD
    Z([QA Bug Coverage left<br/>the bug's test in red]) -.->|the developer applies the fix| A
    A([🔧 Fix deployed<br/>e.g.: TechStore, stock check]) --> B["<b>Step 1</b> — Read the bug again"]
    B --> C["<b>Step 2</b> — Find the bug's tests"]
    C --> D{Did it find<br/>tests?}
    D -->|No| X([⏹ stopped: run QA Bug Coverage first])
    D -->|Yes| E["<b>Step 3</b> — Run and repair<br/>until they pass"]
    E --> F["<b>Step 4</b> — Report: before and after"]
    F --> G{Did they pass?}
    G -->|Yes| H([✅ Bug closed with evidence])
    G -->|No| I([🔴 The fix isn't enough])

Original plan

The plan file is plans/qa-bug-fix-verification.yaml, in the engine repository (crowdar-engine-mvp). The format reference is in Anatomy of a plan.