QA Bug Fix Verification
This guide explains what the NEO engine does when it runs this plan. The idea is that you can read it, understand what happens at each step without anyone stepping in, and know what's needed for the work to turn out well.
In short
- What it's for: confirming, with tests, that a bug's fix really works, and leaving a report with the "before" (red) and the "after" (green).
- When it runs: after the developer applied the fix for a bug that previously went through the QA Bug Coverage plan.
- What it needs: the same bug reference used in QA Bug Coverage, and the test that captures that bug must already exist.
- What it leaves: the bug's tests run (and repaired if they failed for minor reasons) and a final verification report with the before and after.
- Steps: 4 — re-read the bug, find its tests, run them until they pass, and report. No pauses.
What is a plan? (if this is the first time you see one)
A plan is a recipe that a machine runs instead of a person. Like a playbook, it chains several skills in a fixed order. The difference is that a playbook is followed by the AI assistant together with you, with pauses for you to decide; a plan is run by the NEO engine on its own, end to end.
See Plans for the full comparison between plan and playbook.
1. What does this plan do?
It's the second half of a bug's cycle: after the fix, it confirms that the test that used to fail now passes. The first half (confirming the bug and creating that test) is done by the QA Bug Coverage plan.
Its intent is that no bug is closed "on someone's word": closing it has to be backed by the same test that proved the bug, now green. To do that, the plan:
- Reads the bug again, just as QA Bug Coverage did. It doesn't depend on anything left over from that earlier run: it builds its own context from scratch.
- Finds the tests that cover the bug: the test QA Bug Coverage generated, plus the ones that already covered that feature.
- Runs only those tests and repairs the ones that fail for minor reasons, until they pass.
- Writes the final report, comparing the before (the bug's evidence) with the after (this result).
The plan doesn't apply or suggest any fix: it assumes the developer has already applied it.
2. When does it run?
| Situation | Business example |
|---|---|
| The developer applied the fix for a bug that already has its red test | TechStore's developer fixed the cart's stock check and deployed it to staging |
| The bug has to be closed with evidence | TechStore's QA lead won't close TS-BUG-31 without a report showing the test green |
It's triggered once the fix is ready: in free text (Model 2: the engine chooses this plan), or from the CI pipeline with whatever rule each project defines (for example, a push to the fix branch, or a manual run when the developer considers the fix ready).
Important
- QA Bug Coverage has to have run first. This plan looks for the test that captures the bug; if it doesn't find it, it stops without running anything.
- The fix must be deployed at the URL where the tests run. Otherwise, the tests validate the previous version and will stay red.
- It's best to use the same bug reference as in QA Bug Coverage, so both plans find the same tests and the same identifier.
3. How does it differ from the playbook?
This plan does, with nobody on the other side, the second half of the Bug Fix Cycle playbook:
| Bug Fix Cycle playbook | This plan | |
|---|---|---|
| Who runs it | The AI assistant, talking with you | The engine, on its own |
| How it knows the fix is ready | You tell it, after the pause | It runs when CI or a person triggers it, after the fix |
| Which tests it runs | The bug's test and then the module's regression | Only the tests that cover the bug (the bug's test plus the ones that already covered that feature), not the whole module |
| Context | Carries it over from the earlier steps of the same conversation | Builds it from scratch: it reads the bug again |
| Report | Before and after | Before and after, the same |
4. Guiding example: "TechStore" fixes the stock check
We continue with the QA Bug Coverage plan's case. At TechStore, the cart let you add more units than were in stock (TS-BUG-31). QA Bug Coverage confirmed the bug and left a red test. Now the developer applied the fix and deployed it to staging: the cart shows an error when the stock is exceeded.
4.1 What the plan needs to receive
| Data | Required | What it is | TechStore example |
|---|---|---|---|
bug_report |
Yes | The reference to the bug: a path to a file, the URL of a Jira or Azure DevOps ticket, or the report text. The same one used in QA Bug Coverage | https://techstore.atlassian.net/browse/TS-BUG-31 |
identifier |
Yes | The bug's identifier | TS-BUG-31 |
credentials |
Yes | Test credentials to log into the application | Staging test username and password |
4.2 How it's triggered
neo run plans/qa-bug-fix-verification.yaml --input-file input.json --workspace .
Where input.json has the data in the table above. With Model 2, the engine can also choose this plan from a request like "bug TS-BUG-31 is fixed, verify it".
4.3 Step by step
Step 1 of 4 — Read the bug again
Skill used: lippia-neo-read-bug-report
What happens: it reads the bug report just as QA Bug Coverage did, to recover its identifier, its acceptance criteria, and the application's URL. It doesn't reuse anything from the earlier run: that way the plan works even if it runs days later.
Step 2 of 4 — Find the bug's tests
Skill used: lippia-neo-find-existing-tests
What happens: it searches the project's tests folder for the ones that cover the bug. The test generated by QA Bug Coverage should show up there, plus the ones that already covered that feature.
With TechStore: it finds the new "don't allow exceeding stock" test and the cart tests that already existed.
Stopping point: no tests, nothing to verify
If it doesn't find any test covering the bug, the plan stops here (stopped). Running other tests wouldn't prove the reported bug was fixed. It usually means QA Bug Coverage hasn't run for that bug yet.
Step 3 of 4 — Run the tests until they pass
Skill used: lippia-neo-run-and-heal
What happens: it runs only the tests Step 2 found. If one fails for a minor reason (a selector that changed with the fix, a short wait), it repairs it and runs again, up to 3 times.
| Result | What it means |
|---|---|
| ✅ All tests pass | The fix works: the bug's test went from red to green |
| ❌ Some test keeps failing | The fix doesn't resolve the bug, or broke something else in that feature |
It repairs tests, not the application
If the bug's test keeps failing because the bug still happens, this step doesn't "fix" it so that it passes: it stays as a failure in the report.
With Lippia the whole module runs
Running only some tests works with Playwright. If the project uses Lippia, it can't be narrowed down by file and the whole module runs.
Step 4 of 4 — Final report: before and after
Skill used: lippia-neo-write-test-report
What happens: it writes the verification report comparing two moments:
- Before the fix: the evidence that the bug reproduced, left by QA Bug Coverage.
- After the fix: the result of Step 3.
The report is always written, whether the tests pass or not: it has to reflect the real result, not just the happy path.
What you get: the report, with date and time, in the project's reports folder (by default test-results/reports/).
4.4 How does it end?
| How it ended | Plan status | What it means | What to do |
|---|---|---|---|
| ✅ The tests pass | success |
The fix works | Close the bug, attaching the report |
| 🔴 Some test keeps failing | success |
The plan worked, but the fix isn't enough | Review the report with the developer and run again after the new fix |
| ⏹ There were no tests covering the bug | stopped |
There's nothing to verify | Run QA Bug Coverage first |
| ❌ A skill returned something malformed, or something broke | failed / error |
The step failed, not the application | Check the log and result.json |
Red tests aren't an engine error
If the fix doesn't work, the plan still ends in success: the engine did its job and the test results are in the report and in result.json. If the pipeline has to be blocked in that case, the rule is defined by CI, reading that result.
5. Common problems (and what to do)
| What you see | Why it can happen | What to do |
|---|---|---|
The plan ends stopped at Step 2 |
QA Bug Coverage hasn't run for this bug yet, or a different bug reference was used | Run QA Bug Coverage first, or use the same reference used there |
| The bug's test is still red | The fix isn't deployed at the test URL, or doesn't resolve the bug | Confirm the deployment; if it's deployed, review the report with the developer |
| Other tests fail besides the bug's | The fix broke another part of the same feature | Don't close the bug; review with the developer |
| The report doesn't show the "before" | The reproduction evidence left by QA Bug Coverage wasn't found | Confirm QA Bug Coverage ran in the same project and with the same bug identifier |
6. Visual summary of the flow
flowchart TD
Z([QA Bug Coverage left<br/>the bug's test in red]) -.->|the developer applies the fix| A
A([🔧 Fix deployed<br/>e.g.: TechStore, stock check]) --> B["<b>Step 1</b> — Read the bug again"]
B --> C["<b>Step 2</b> — Find the bug's tests"]
C --> D{Did it find<br/>tests?}
D -->|No| X([⏹ stopped: run QA Bug Coverage first])
D -->|Yes| E["<b>Step 3</b> — Run and repair<br/>until they pass"]
E --> F["<b>Step 4</b> — Report: before and after"]
F --> G{Did they pass?}
G -->|Yes| H([✅ Bug closed with evidence])
G -->|No| I([🔴 The fix isn't enough])
Original plan
The plan file is plans/qa-bug-fix-verification.yaml, in the engine repository (crowdar-engine-mvp). The format reference is in Anatomy of a plan.