QA Bug Coverage
This guide explains what the NEO engine does when it runs this plan. The idea is that you can read it, understand what happens at each step without anyone stepping in, and know what information the bug needs to include for the work to turn out well.
In short
- What it's for: when a bug moves to QA, confirming that it really happens and leaving an automated test that captures it, ready for the developer to use while fixing it.
- When it runs: when coverage is requested for a bug, in free text (Model 2: the engine chooses this plan) or from the CI pipeline.
- What it needs: the reference to the bug (a file, a Jira / Azure DevOps ticket, or the report text), the project's
qa-stack.yaml, and the application reachable. - What it leaves: evidence that the bug reproduces, an automated test in red that captures it (plus the cases that were missing coverage), and a report of what was generated.
- Steps: 6 — read the bug, find the tests that already exist, review what's missing, reproduce the bug, generate the tests, and report. No pauses.
What is a plan? (if this is the first time you see one)
A plan is a recipe that a machine runs instead of a person. Like a playbook, it chains several skills in a fixed order. The difference is that a playbook is followed by the AI assistant together with you, with pauses for you to decide; a plan is run by the NEO engine on its own, end to end.
See Plans for the full comparison between plan and playbook.
1. What does this plan do?
It's the first half of a bug's cycle: it leaves the bug confirmed and "caught" in an automated test, before the developer fixes it. The second half, after the fix, is done by the QA Bug Fix Verification plan.
Its intent is that no bug gets fixed "blindly": before touching the code, there has to be a test that fails because of the bug. That way, when the developer fixes it, that same test will turn green and prove the fix works. To do that, the plan:
- Reads the bug report and, if it mentions a user story, the story too.
- Looks for automated tests that already cover that part of the application.
- Reviews which scenarios of the story or the bug have no coverage.
- Walks through the real application to confirm the bug really happens.
- If it does, generates the test that captures it, plus the tests for the scenarios that were missing.
- Writes a report with everything it generated.
The plan doesn't fix the bug or verify any fix: it ends with the test in red, ready for the developer.
2. When does it run?
| Situation | Business example |
|---|---|
| A bug moves to QA and needs to be confirmed and covered by a test | The TechStore team moves the stock bug to QA: the cart lets you add more units than are available |
| Someone asks for coverage for a bug in natural language | "Bug TS-BUG-31 is already in QA, build its coverage" |
It can be triggered in two ways:
- Model 2 — Adaptive hybrid: someone writes the request in free text (
neo run --input "..."), the engine classifies it and chooses this plan. See Orchestration models in the engine. - From CI: the
lippia-neo-ci-bootstraptemplates include aqa-bug-coveragejob that runs this plan when the pipeline is launched with the variableNEO_PLAN=increase_bug_coverage.
Important
- The bug report must describe how to reproduce it (steps, expected result, and actual result). If it can't be reproduced, the plan stops without generating anything.
- If the bug is in Jira or Azure DevOps, the project must have that MCP configured in
qa-stack.yaml(see Set up the MCPs). - The application must be deployed and reachable: Step 4 walks through it live.
- If the project doesn't have a
qa-stack.yamlyet, you first need to run the project bootstrap playbook.
3. How does it differ from the playbook?
This plan does, with nobody on the other side, the first half of the Bug Fix Cycle playbook:
| Bug Fix Cycle playbook | This plan | |
|---|---|---|
| Who runs it | The AI assistant, talking with you | The engine, on its own |
| Scope | The full cycle: reproduce, create the test, wait for the fix, verify it, and report | Only up to the red test. Verifying the fix is a different plan: QA Bug Fix Verification |
| Existing tests | Doesn't look for them | First looks for the ones that already cover the feature and reviews which scenarios are missing |
| What it generates | The bug's test | The bug's test plus the tests for the scenarios that were missing |
| Pauses | Confirm the test fails and wait for the fix | None: if the bug doesn't reproduce, it stops on its own |
4. Guiding example: "TechStore" and the cart that doesn't check stock
We continue with the same case as in the Bug Fix Cycle playbook. A TechStore customer reports:
"I tried to buy 5 units of a mouse that the website said had only 2 left, and the system let me add them to the cart anyway without any warning."
The business rule in story TS-US-08 says that "the system must show an error message if the customer tries to add more units than are in stock". The team files the bug as TS-BUG-31 and moves it to QA.
4.1 What the plan needs to receive
| Data | Required | What it is | TechStore example |
|---|---|---|---|
description |
Yes | The reference to the bug: a path to a file, the URL of a Jira or Azure DevOps ticket, or the report text itself | https://techstore.atlassian.net/browse/TS-BUG-31 |
identifier |
No | The bug's identifier, if it can't be deduced from the report | TS-BUG-31 |
credentials |
No | Test credentials to log into the application, if they're not in the report | Staging test username and password |
In Model 2 the engine extracts this data from the free-text request. If the required one is missing, it runs nothing and escalates to a person.
4.2 How it's triggered
With Model 2, from the terminal or a script:
neo run --input "bug https://techstore.atlassian.net/browse/TS-BUG-31 is already in QA, build its coverage" \
--plans-dir plans --workspace .
Or directly, choosing the plan:
neo run plans/qa-bug-coverage.yaml --workspace . \
--data '{"description":"https://techstore.atlassian.net/browse/TS-BUG-31"}'
4.3 Step by step
Step 1 of 6 — Read the bug (and its story, if it mentions one)
Skills used: lippia-neo-read-bug-report and, when applicable, lippia-neo-read-user-story
What happens: it reads the bug report (from the file, the ticket, or the text) and extracts the essentials: how to reproduce it, what should happen, what actually happens, the severity, and the environment. If the bug mentions a user story, it reads that too, to get its real acceptance criteria.
With TechStore: it reads TS-BUG-31, detects that it's related to TS-US-08, and fetches the stock rule.
Step 2 of 6 — Find the tests that already exist
Skill used: lippia-neo-find-existing-tests
What happens: it searches the project's tests folder for the ones that already cover that feature (by the story's identifier, its acceptance criteria, or its key features).
With TechStore: it finds the cart tests that already existed, for example "add a product to the cart".
Step 3 of 6 — Review what's missing
Skill used: lippia-neo-review-test-coverage
What happens: it compares the tests it found with the scenarios described by the story and the bug, and builds the list of scenarios that have no test at all.
With TechStore: it detects that no test checks what happens when the available stock is exceeded.
Step 4 of 6 — Reproduce the bug in the application
Skill used: lippia-neo-reproduce-bug
What happens: it opens a browser and follows the report's steps in the real application, to confirm whether the bug really happens. It saves the evidence: screenshots, errors, and the real selectors on the screen. If it doesn't succeed on the first try, it tries a few variants.
With TechStore: it tries to add 5 mice with a stock of 2 and confirms the cart accepts them without showing any error.
Stopping point: if the bug doesn't reproduce, the plan stops
If the bug doesn't reproduce, the plan stops here (stopped) and generates nothing: there's no point writing a test for a bug that doesn't exist in the environment. The result is documented, with the reason, for review.
Today it always tries to reproduce
This step runs even if Step 2 found tests that already cover the bug. In a future version, the plan is expected to run those tests instead of reproducing the bug by hand when coverage already exists.
Step 5 of 6 — Generate the missing tests
Skill used: lippia-neo-generate-test-suite
What happens: it generates, in the project's framework, two kinds of test:
- The bug's test: reproduces the bug confirmed in Step 4. Today it fails (it's red), and it will turn green when the developer fixes it. It's always generated when the bug reproduced.
- The tests for the missing scenarios: the ones Step 3 flagged as having no coverage.
The new tests are saved next to the ones that already existed for that module.
Step 6 of 6 — Report of what was generated
Skill used: lippia-neo-write-generation-report
What happens: it writes a report with the tests that were generated, the ones that already existed, and the evidence of the bug's reproduction.
What you get: the report in the project's reports folder, ready for the developer to start the fix.
4.4 How does it end?
| How it ended | Plan status | What it means | What to do |
|---|---|---|---|
| ✅ The bug reproduced and the tests were generated | success |
There's a red test that captures the bug | The developer fixes the bug; then QA Bug Fix Verification runs |
| ⏹ The bug didn't reproduce | stopped |
Nothing was generated | Check the reason in result.json: it may be a different environment, test data, or a bug that was already fixed |
| ❌ A skill returned something malformed, or something broke | failed / error |
The step failed, not the application | Check the log and result.json |
The bug's test failing is expected
When this plan ends, the bug's test must be red: that proves it captures the bug. If it passed green, it wouldn't be testing the reported problem.
5. Common problems (and what to do)
| What you see | Why it can happen | What to do |
|---|---|---|
The plan ends stopped at Step 4 |
The bug doesn't reproduce in that environment, the report doesn't have clear steps, or the bug was already fixed | Complete the reproduction steps in the report, confirm the environment, and run again |
| In Model 2, the engine escalates to a person | The request was ambiguous or didn't include the reference to the bug | Rephrase the request with the bug's link or ID, or run the plan directly with neo run plans/qa-bug-coverage.yaml |
| Step 1 can't read the ticket | The Jira or Azure DevOps MCP isn't configured, or the credentials are missing | See Set up the MCPs |
| Step 2 doesn't find tests even though they exist | The bug doesn't mention the story, so the search is narrowed down to the bug's identifier | Mention the story (for example TS-US-08) in the bug report |
| No tests were generated for missing scenarios | Step 3 found no uncovered scenarios | That's correct: only the bug's test is generated |
6. Visual summary of the flow
flowchart TD
A([🐞 Bug in QA<br/>e.g.: TechStore, cart stock]) --> B["<b>Step 1</b> — Read the bug<br/>(and its story)"]
B --> C["<b>Step 2</b> — Find the tests<br/>that already exist"]
C --> D["<b>Step 3</b> — Review what's missing"]
D --> E["<b>Step 4</b> — Reproduce the bug<br/>in the app"]
E --> F{Did it reproduce?}
F -->|No| X([⏹ stopped: nothing is generated])
F -->|Yes| G["<b>Step 5</b> — Generate the bug's test<br/>and the missing ones"]
G --> H["<b>Step 6</b> — Report of what was generated"]
H --> I([🔴 Red test, ready for the developer])
I -.->|after the fix| J([QA Bug Fix Verification])
Original plan
The plan file is plans/qa-bug-coverage.yaml, in the engine repository (crowdar-engine-mvp). The format reference is in Anatomy of a plan.