Skip to content

Bug Fix Cycle

This guide is meant to help you understand what the AI assistant does when it runs this playbook. The idea is that you can read it, understand what happens at each step, and know what information you need to give the assistant so the work turns out well.

In short

  • What it's for: getting a bug fixed and "recorded" in an automated test that will warn you if it ever comes back.
  • When to use it: when someone reported a specific bug and you need to prove, with evidence, that it was fixed without breaking anything else.
  • What you need: the bug description, the environment where it happens, the story's test plan, the project's qa-stack.yaml and a developer available to apply the fix.
  • What you get: evidence of the bug, a new automated test, confirmation of the fix and a report with the "before" and the "after".
  • Steps: 6 — understand the rule, reproduce the bug, create the failing test, (pause for the fix), verify the fix, run the module regression and put together the before/after report.
What is a playbook? (if this is your first time using one)

Think of it as a cooking recipe for the AI assistant. You tell it "I want to do this" (for example, "I want to fix this bug and leave it tested") and the playbook is the recipe the assistant follows step by step, in order, without skipping anything, so the result is always the same no matter who asks for it.

Each step of the recipe uses a specific tool (in technical jargon they're called "skills"), something like a kitchen utensil: one is for peeling, another for cutting, another for baking. You don't need to know how each utensil works inside — you only need to know which ingredient goes into each step (for example, what the application gets wrong, or at which URL it can be reproduced) so the final dish turns out well.

A playbook = an ordered guide of steps. A skill = the specific tool that carries out each step. You interact with the playbook — the assistant takes care of using the right tool at each moment.


Also available as automated plans

This cycle can run without the assistant, from the NEO engine, with two plans: QA Bug Coverage confirms the bug and leaves the test in red (before the fix), and QA Bug Fix Verification verifies that it turns green (after the fix).


1. What does this playbook do?

It's the playbook you use when a bug shows up (something the application does wrong) and you want it fixed and, above all, "recorded" in a set of tests.

Today, when someone reports a bug, the usual routine is:

  1. Someone tries to reproduce it by hand to confirm it exists.
  2. The developer fixes the bug.
  3. Nobody leaves behind an automated test that confirms, once and for all, that this specific bug won't come back in the future (what's known as a "regression").

This playbook changes that order: first a test is written that proves the bug exists (and that fails, on purpose), then the bug is fixed, and only then is it confirmed that the test now passes. This technique is called TDD (you don't need to pay attention to the acronym — what matters is the idea: "first the failing test, then the fix, then the passing test").

When you're done, the bug ends up with:

  • Real evidence that the bug existed (exact steps + a screenshot of the problem).
  • An automated test saved permanently, which in the future will warn you on its own if that bug comes back.
  • Confirmation that the developer's fix actually worked.
  • Confirmation that fixing that bug didn't break any other part of the application.
  • A report that shows, on the same page, the "before" (bug present) and the "after" (bug fixed) — ideal for closing the ticket with formal evidence.

2. When should I use this playbook?

Use it when you're facing any of these situations:

Situation Business example
Someone reported a specific bug and it needs to be confirmed and fixed with evidence A TechStore (online store) customer reports that they were able to buy 5 units of a product that only had 2 in stock, and nobody showed them an error message.
The ticket requires proving that the bug was fixed, not just "trusting" the developer The TechStore team has a policy: no bug is closed without a test that checks it before and after the fix.
The bug is in a part of the app that already has automated tests and you want to confirm that fixing it didn't break anything else TechStore's "shopping cart" module already has several automated tests running; you want to make sure the fix for the stock bug didn't break the total calculation.
A formal report is needed with details of what was failing and how it was resolved TechStore's QA lead asks for documented evidence to attach to the ticket before closing it.

Important

  • This playbook assumes the project is already set up for automated QA (that is, the qa-stack.yaml file already exists, which is the result of having run the project setup playbook). If TechStore were a brand-new project with none of this set up, you'd need to run that playbook first — this one is for day-to-day work, not for starting from scratch.
  • If there's a user story, this playbook needs its test plan to already exist (the document with the cases to test, which by default lives in the test-plans/ folder). The bug test is built by taking from that plan the case that matches the broken rule; without a plan, the assistant can't generate it and stops.
  • If what you need to test isn't a bug but a visual or design change (for example, a button was moved), this isn't the right playbook — there's a specific one for interface changes.
  • This playbook needs a developer to be available at some point in the process, because it stops halfway through waiting for the fix to be applied. It's not a 100% unattended process.

3. Guide example: "TechStore", an online store with a shopping cart

We'll follow the same business case as in the project setup guide: TechStore, an online electronics store with a shopping cart.

This time, TechStore is already in production and has automated tests running. A customer reports the following to the support team:

"I tried to buy 5 units of a mouse that the website said had only 2 left, and the system let me add them to the cart anyway without warning me at all."

The team confirms it's a real bug: the business rule says "the system must show an error message if the customer tries to add more units than are in stock", and that rule isn't being met. That's exactly the scenario this playbook is for.

Throughout the following steps we'll see what you'd reply to the assistant at each moment, using this TechStore bug as the example.

3.1 Before you start: what you need to have on hand

You don't need to know how to code, but you do need to have this information on hand:

Information TechStore example
What exactly fails, in simple words "The cart lets you add more units of a product than are in stock, without showing any error"
Which business rule is broken (if there's a reference user story or acceptance criterion) "The system must show an error message if someone tries to add more units than the available stock"
The related user story, if there is one: a .md file or the ticket link in Jira or Azure DevOps The cart story file in user-stories/, or the ticket link
The test plan for that story The cart plan, in the test-plans/ folder (for example, test-plans/<story-ID>-test-plan.md)
The environment where the bug can be reproduced https://techstore-staging.com
A bug/ticket identifier, if there is one BUG-42
A developer available to apply the fix when the playbook reaches that point The developer of TechStore's cart module, notified in advance

If you don't have a formal ticket, that's not a problem: describing the bug in simple words is enough, as in the example above.

Original playbook

The general playbook can be found in the NEO-releases repository.

3.2 How to start

Before Step 1, tell the assistant about the bug to fix, with as much detail as possible:

💬 Prompt for the assistant
I want to cover the following bug with TDD:
- Bug ID: BUG-42
- What fails: the TechStore cart lets you add more units of a
  product than are in stock, without showing any error
- Rule that is broken: "the system must show an error message if someone
  tries to add more units than the available stock"
- Related story (if any): user-stories/<story-ID>.md
- Affected module: shopping cart
- The app is at https://techstore-staging.com
Don't take any action, just keep this information in your context

With that context, the assistant can go through all the following steps without you having to repeat it every time.


3.3 Step by step

Step 1 of 6 — Understand which business rule is broken

Skill used: lippia-neo-read-user-story

What happens: if there's a user story or a reference document (a file or the ticket link in Jira or Azure DevOps), the assistant reads it and gives you back the list of rules the story must meet (the "acceptance criteria"), along with the app URL and the login credentials. From that list, you are the one who chooses which rule the bug is violating — the assistant doesn't mark it on its own. If there's no such document, it simply uses the description you gave it in the previous step.

What you have at the end of this step: a clear, specific sentence stating which rule the application should meet and currently doesn't. In the TechStore case: "the system must show an error message if someone tries to add more units than the available stock".

Checkpoint 0 — Confirm the affected rule

Confirm which acceptance criterion the bug breaks. The rest of the playbook revolves around that rule.

If you don't have a user story on hand

No problem — you can skip this step and give it the rule directly in the next step.


Step 2 of 6 — Reproduce the bug with real evidence

Skill used: lippia-neo-run-manual-testing

What happens: the assistant goes into the application (the way a person testing it by hand would, always navigating the screen, never "behind the scenes") and tries to reproduce the problem step by step, just as the customer described it. It records each step it takes and takes a screenshot at the exact moment the error is visible.

With TechStore, this means the assistant goes to the site, finds the mouse with only 2 units in stock, tries to add 5 to the cart, and confirms that indeed no error message appears — recording that moment with a screenshot.

If you did Step 1 (with a story): ask it to focus on the rule you confirmed.

💬 Prompt — Step 2 with a story
/lippia-neo-run-manual-testing
Focus on the AC scenario: "the system must show an error message
if someone tries to add more units than the available stock"

With a story, you need the test plan

In this mode the assistant looks for the case inside the story's test plan (by default in test-plans/). If that plan doesn't exist, it stops with a "no scenarios were found" message. In that case, use the no-story mode below.

If you skipped Step 1 (no story): give it the bug description, including the bug ID — the assistant uses it to name the folder where it stores the evidence, and without it, it can't start.

💬 Prompt — Step 2 without a story
/lippia-neo-run-manual-testing
Explore freely and reproduce this bug:
- Execution ID: BUG-42
- What fails: the cart lets you add more units of a product than
  are in stock, without showing any error
- Expected: show an error message when the available stock is exceeded
- The app is at https://techstore-staging.com

What you have at the end of this step:

  • The exact steps to reproduce the bug (so anyone can repeat it).
  • A screenshot of the moment the problem is visible.
  • The technical data of the affected element (which the assistant will use on its own, without you having to step in, in the next step).

Everything is saved in its own folder inside execution/ (named with the story ID, or with BUG-42 if there was no story).

If the bug is in an API

The assistant always reproduces the bug by navigating the application's screen. If the problem is only in an internal service (an API) and isn't visible on screen, comparing its responses against what's expected is outside the scope of this step.

Checkpoint — Was the bug reproduced?

The bug was reproduced if there are clear steps + a screenshot showing the problem. If the assistant can't reproduce it, the environment may not be the right one or some information may be missing (for example, login credentials).


Step 3 of 6 — Create a test that "catches" the bug

Skill used: lippia-neo-generate-test-suite

What happens: with the evidence from the previous step, the assistant writes one targeted automated test, specifically designed to detect this bug. To do so, it takes from the story's test plan only the case that matches the broken rule (it leaves the other cases aside) and uses the real technical data it collected in Step 2. That test describes how the application should behave (showing the stock error), not how it behaves today (with the bug present) — that's why, at this point, the test will fail on purpose. The file has a comment at the top that warns exactly that: it must fail until the bug is fixed.

💬 Prompt for the assistant
/lippia-neo-generate-test-suite
TDD mode: generate ONLY the test for the affected AC "the system must show an
error message if someone tries to add more units than the available stock"

It's like setting up an alarm that goes off today because the problem it detects hasn't been solved yet — as soon as it's solved, the alarm goes quiet, and if the problem came back in the future, the alarm would go off again on its own.

What you have at the end of this step: a new automated test file, dedicated exclusively to this bug, inside the story's test folder (with a name like tc-<number>-<short-description>).

A second file may appear

If the bug was reproduced in Step 2 (the scenario "failed"), the assistant always also generates a separate regression test for that scenario, tagged with the bug ID — unless one with that tag already exists. It's not an error: check the list of generated files it gives you back to know how many were created.


Checkpoint 1 — Confirm the alarm goes off (pause)

What happens: the assistant runs that new test just once, directly (without any automatic repair), to confirm that it does fail (that "the alarm goes off"). If it fails, that's the right sign: it means the test is really detecting the bug. That result is saved in a separate file (qa-reports/pre-fix-snapshot.json), which is later used as the "before" in the final report.

Result What it means What to do
❌ The test fails Correct — the test does detect the bug The test is saved in the project history (a "commit" in Git) as formal evidence, and you move forward
✅ The test passes Something doesn't add up — the bug may no longer be in this environment, or the business rule may have been misunderstood Review with the team before continuing

This is an important control point

The result of this step is saved as proof that the bug existed. It's the "before" that will later be compared with the "after". You don't move forward without saving the red (failing) test in the project history (the commit): that's the formal evidence of the bug and part of what's required to close the ticket.


[PAUSE] — the developer applies the fix

What happens: here the playbook stops and it's the developer's turn to fix the bug. The assistant hands over the automated test from Step 3 as a clear, runnable instruction: "the application has to pass this test".

The developer:

  1. Runs the test on their computer to confirm they see the same problem.
  2. Fixes the application code.
  3. Confirms that the test now passes (without modifying the test file).
  4. Saves the fix in the project history — without touching the test file.
  5. Lets the QA team know the fix is available to verify (for example, in the staging environment).

Mandatory pause

At this point nobody but the developer is involved. The assistant waits until it's told the fix is ready before continuing.


Step 4 of 6 — Verify that the fix really works

Skill used: lippia-neo-run-and-heal

What happens: once the developer says they've applied the fix, the assistant runs the same bug test again (not a new one), just once and without trying to repair it, to confirm that it now passes.

💬 Prompt for the assistant
/lippia-neo-run-and-heal
Run only the test for bug BUG-42 with:
- max_iterations: 1
- plateau_detection: false
- run_label: "post-fix"

With TechStore (the fictional online store we use as an example), this means the assistant again tries to add 5 units of a product with only 2 in stock, and this time confirms that the expected error message does appear.

One detail before running

Even though nothing is repaired in this step if the test fails, the assistant still makes two automatic checks before running it: it fills in pending data (values marked as TODO) and replaces hand-written credentials with the ones from the configuration file. If it touches any of that, it's noted as an "adjustment" to the test. Also, if the project uses Lippia instead of Playwright, you can't run a single test: the whole suite is run.

Checkpoint 2 — Did the fix pass without help? (pause)

Result What it means What to do
The test passes, and the assistant didn't have to "help" it pass ✅ The fix works correctly Move on to Step 5
The test passes, but the assistant had to adjust it to make it pass ⚠️ Warning sign — the fix may be incomplete Review with the developer before continuing: to close the bug, the test has to pass without any adjustment
The test still fails ❌ The fix didn't work It goes back to the developer with full details of why it still fails

Step 5 of 6 — Confirm nothing else broke

Skill used: lippia-neo-run-and-heal

What happens: the assistant runs all the automated tests of the affected module (not just the one for this specific bug), to confirm that the fix didn't create a new problem in another part of the application. If any of those other tests broke because of the change (for example, because something on the screen moved), the assistant tries to repair it on its own (up to 3 attempts).

💬 Prompt for the assistant
/lippia-neo-run-and-heal
Run all the tests in the cart module with:
- max_iterations: 3
- plateau_detection: true
- run_label: "regresion-modulo"

With TechStore, this means running all the tests in the "shopping cart" module (adding products, viewing the cart, calculating the total, applying coupons, etc.), not just the one for the stock bug.

Checkpoint 3 — Is the module healthy? (pause)

Result What to do
✅ All the module's tests are green You can now move on to merging the fix
⚠️ Some test couldn't be repaired on its own Review the details the assistant leaves before merging — it may need the developer to step in
❌ The original bug test fails again The fix collided with another change — review it with the developer before continuing

Step 6 of 6 — Final report: before and after

Skill used: lippia-neo-write-test-report

What happens: the assistant puts together a final report that shows, side by side, the "before the fix" result (the test failing, with the screenshot of the bug) and the "after the fix" result (the same test passing), plus the result of the full module regression. That comparison appears in its own section of the report, called "Diff antes/después" (before/after diff).

For the report to be able to say which rules of the story were covered, the assistant needs the story and its acceptance criteria from Step 1. If it doesn't have them (for example, because you skipped that step), the coverage part of the report is marked as "not assessable" — it's not an error, it just means there was nothing to compare it against.

What you have at the end of this step: a single document that serves as formal evidence to close the ticket, clearly showing that the bug existed, that it was fixed, and that nothing else broke in the process. It's saved in the project's reports folder (by default test-results/reports/), with a name like <story-ID>-bug-BUG-42-test-report-<date-and-time>.md.


3.4 What do I end up with after all the steps?

What you get What it's for
Real evidence of the bug (steps + screenshot) Proves the reported problem was real and correctly identified
A new automated test, dedicated to this bug If the bug came back in the future, someone would find out immediately, without anyone having to test it by hand again
Confirmation that the developer's fix works It's no longer a "guess" — it was proven with a real test
Confirmation that nothing else in the module broke The fix is safe to merge
A report with the before/after comparison Formal evidence to close the ticket or show to the stakeholder

From here on, the bug is closed with complete evidence, and the test that detected it becomes a permanent part of TechStore's test suite — protecting against the same problem coming back without anyone noticing.


4. Common problems (and what to do)

What you see Why it can happen What to do
At Checkpoint 1, the new test passes instead of failing The bug isn't present in that environment, or the business rule was misunderstood Confirm the correct environment with the team and check whether the rule identified in Step 1 is the right one
In Step 2 or 3 the assistant stops saying it can't find scenarios or the plan The story doesn't have its test plan yet First build the story's test plan, or do Step 2 in no-story mode
In Step 4, the test passes but the assistant had to "help" it The fix may be incomplete Check with the developer whether the fix fully covers the reported problem
In Step 5, other tests in the module fail and can't be repaired on their own The fix changed something on the screen or in the behavior that other tests took for granted It may need the developer to step in — it's not necessarily a mistake in the fix
The bug fails again after the module regression The fix collided with another change made in parallel Coordinate again with the developer before merging
The bug test doesn't use the real technical data from the screen In Step 2 the complete evidence file wasn't generated (only the readable document) Run Step 2 again and confirm that both evidence files were created
The developer accidentally modified the test file instead of just the application code Confusion between "the test file" and "the code to fix" The fix must be only in the application code; the test isn't touched
The test fails at login The project's test credentials (.env.tests file) are empty or incomplete Ask someone on the technical team to fill in the real values — they're never written by hand inside the tests — and repeat Step 4 or 5

5. Visual summary of the flow

flowchart TD
    A([🐞 Bug reported<br/>e.g.: TechStore lets you buy<br/>more than the available stock]) --> B
    B["<b>Step 1</b> — Understand which<br/>business rule is broken"] --> C
    C["<b>Step 2</b> — Reproduce the bug<br/>with real evidence"] --> D
    D["<b>Step 3</b> — Create a test<br/>that catches the bug"] --> E{Does the test fail?}
    E -->|Passes| F([⚠️ Review environment<br/>or rule with the team])
    E -->|Fails| G["<b>Checkpoint 1</b> — Save the red<br/>test as evidence"]
    G --> H["<b>Pause</b> — The developer<br/>applies the fix"]
    H --> I["<b>Step 4</b> — Verify that<br/>the fix works"]
    I --> J{Passes without help?}
    J -->|No| H
    J -->|Yes| K["<b>Step 5</b> — Confirm nothing else<br/>broke in the module"]
    K --> L{Module green?}
    L -->|No| H
    L -->|Yes| M["<b>Step 6</b> — Final report<br/>with before and after"]
    M --> N([✅ Bug closed, with evidence<br/>and a permanent test])