Skip to content

Bug Reproduction

This guide explains what a quality assurance (QA) playbook is and exactly what the "Bug Reproduction" playbook does.

In short

  • What it's for: confirming that a reported bug is real and documenting it with evidence (exact steps + screenshots).
  • When to use it: when someone reported strange behavior and it needs to be confirmed before sending it to the development team.
  • What you need: what fails, what should happen, the URL where it happens, the project's qa-stack.yaml and, if there is one, the related story.
  • What you get: the reproduction steps, screenshots of the problem, the reproducibility status and a date- and time-stamped report ready for the ticket.
  • Steps: 4 — read the story (optional), reproduce the bug, put together the report and update the ticket by hand.
What is a playbook? (if this is your first time using one)

It's like, for example, the checklist a pilot follows before takeoff: they don't improvise every time, they follow a proven sequence of steps. A playbook is that same idea applied to testing an application: a written recipe that tells the AI model, step by step, what to do using different elements/tools called "Skills".

What is a "Skill"?

A skill is the specific tool that carries out each step of the playbook. You interact with the playbook — the assistant takes care of using the right tool at each moment.


1. Basic concepts

Before you start: here are some basic concepts that are good to understand for this playbook. Open the one you need.

Bug

It's like following a cake recipe to the letter and the cake still doesn't come out right. A bug is an imperfection or deficiency in a system or application that can arise as a result of human error, defects in the requirements, or in the implementation/development of the software. Bugs can exist in the code without showing up as errors, and managing them is crucial to improving software quality.

Ticket / user story

A user story represents a software requirement from the point of view of the end user or customer, focusing on the value it brings and not on the technical implementation details. Its goal is to make communication easier between the development team, testers and stakeholders, making sure everyone understands the purpose of the feature before it's developed. It's generally essential for it to include the acceptance criteria.

Acceptance criterion (AC)

These are the conditions that must be met for the story to be considered done. A single ticket can contain several different rules/criteria. An acceptance criterion (AC) is one of those specific rules, e.g., talking about a car ("the rear lights must turn on", "the handbrake holds on a slope", etc.). If any of those criteria isn't met, a bug is reported.

Environments: dev, staging/QA, production and local

These are the places where a copy of the application can live:

  • Production: it's the real theater performance, with an audience that paid for tickets: the version customers use every day.
  • Staging/QA: it's the dress rehearsal before opening night: an identical copy, but without a real audience.
  • Dev: it's the room before the dress rehearsal where the team prepares the delivery to see that everything will go well and that nothing is being forgotten.
  • Local: it's simply rehearsing at your own home, on your own computer, without anyone else seeing it yet.

Depending on where the bug shows up, its severity indicates how urgent it is to fix it: a bug in production means real customers are seeing it; a bug in staging is where it gets prevented and reported, so that the same bug doesn't reach production when the feature goes live.

Reproducing a bug

It's like when you go to an orthopedist and they ask you "what were you doing when the pain started?", and then they try to recreate those same conditions to confirm the symptom in the office, instead of just taking your word for it. Reproducing a bug means following the same exact steps the customer or QA did, until the incorrect behavior happens again right in front of you — only then is it confirmed that the problem is real.

Evidence (steps + screenshots)

It's like a police report or an insurance claim: it's not enough to say "something broke", you need to describe exactly what happened, in what order, and attach photos. Bug evidence is that same idea: the exact steps to get to the problem, plus screenshots showing the incorrect state/problem described. Example: someone stole a wheel, and you attach a photo of the car without the wheel.

Not reproducible / intermittent

A bug is "not reproducible" when, after trying by following the steps, it doesn't happen again under the same conditions. It's "intermittent" when it happens, but not always.

Minimal report

This playbook, unlike others, doesn't write or run automated tests — it only documents. It's like the difference between a police report and a trial: the report doesn't solve the case, but it's what's needed for the case to get started. The minimal report is that document: the data used, specific steps detailed in order, and evidence of the failure.

Bug ticket (BUG-ID)

When the bug is identified, writing down its steps and details requires a ticketing tool where a ticket is opened with all the details; when that ticket is saved, the tool gives us a number/link so we can track it and assign it to someone on the development team to fix. They're usually linked to the user story we're testing, as long as the bug is related to it and isn't one we found by chance while browsing the application as we tested the user story.


2. What does this playbook do?

It's the playbook you use when someone reported strange behavior in the application and you need to confirm it and document it with evidence, before sending it to the development team.

Think of it as a police report or an insurance claim: it's not enough to say "something broke" — you need to describe exactly what happened, in what order, and attach proof. This playbook does just that: it repeats the same steps that lead to the problem, until the incorrect behavior happens again in front of the assistant, and documents everything with screenshots.

Key difference from other playbooks

This one doesn't generate or run automated tests — its only job is to confirm the bug and document it. If the bug is confirmed and it needs to be covered with automated tests, a different playbook does that (for example, the bug fix cycle or the complete flow for a new feature, using the same story): each playbook has its own role (e.g., the waiter doesn't do the cook's job).


3. When should I use this playbook?

Use it when:

  • Someone reported a bug and it needs to be confirmed before sending it to development.
  • Evidence is needed, not just someone's word.
  • There are no automated tests for that part of the app.

Important

  • This playbook doesn't fix the bug, it only confirms and documents it.
  • It needs the project to already have the qa-stack.yaml file in place. If it doesn't exist, you first need to run the project setup playbook to create it.
  • It needs the assistant to have the automated browser (Playwright MCP) enabled: with it, the assistant goes through the application the way a person would. Without it, Step 2 can't start — if you're not sure whether it's enabled, ask the technical team.

4. Guide example: "TechStore", an online store with a shopping cart

Support reports this:

"Several customers say that when they click 'Apply coupon' in the cart, nothing happens — the discount is never applied."

Nobody has seen it happen first-hand yet. Before assigning it to a developer, it needs to be confirmed: the exact steps, and a screenshot of the moment it fails. That's the scenario for this playbook — we'll use it as the example in all the following steps.

4.1 Before you start: what you need to have on hand

You don't need to know how to code, but you do need to have this information on hand — and if you don't have it, here we tell you where to get it:

Information TechStore example Where you get it
What fails "The 'Apply coupon' button doesn't respond when clicked" The person who reported the bug tells you. Ask them: "show me or tell me exactly what you did and what happened"
What should happen instead "It should apply the discount and update the total" Ask the same person: "what did you expect to happen?"
URL where it happens https://techstore.com/carrito The person who reported it gives it to you. If you're not sure whether it's production or a test environment: production is the everyday address; test environments usually have "staging.", "test." or "preview" in the name
Credentials (only if login is needed) Test username and password Ask the technical team for them
Bug ticket ID BUG-42 The ticketing tool gives it to you when you create the bug. If there's no ticket yet, use a simple, temporary identifier: it's used to name the evidence folder and the report
Related story (optional) Ticket TS-US-14 The developer or tester gives it to you. If there's no ticket, it doesn't matter — you continue without that information
Test plan for the story (only if you use a story) test-plans/TS-US-14-test-plan.md It's the document with the cases to test for that story, generated earlier by another playbook. If it doesn't exist, you continue in "no story" mode

Original playbook

The general playbook can be found in the NEO-releases repository.

4.2 How to start

Before Step 1, tell the assistant about the bug you want to reproduce, all together in a single message. Following the TechStore example:

💬 Prompt for the assistant
Here's the context of the bug (don't run anything), just save it:
- Problem description: the "Apply coupon" button doesn't respond when clicked in the cart
- Expected result: it should apply the coupon discount and update the total to pay
- URL: https://techstore.com/carrito
- Related story: user-stories/TS-US-14.md (optional). There may not be one (not recommended), or the project may be very old and the initial development documentation for the application was never created because it was built in a more homemade way, so sometimes the definition is in the code or in the common sense of whoever is using the application; later the IT team will decide whether it's really a bug, an improvement, or even a new requirement for something that already exists without that definition.

The assistant will use that context throughout all the following steps.

Relationship with Step 1

The context you give here is always used. If you also have a story, Step 1 reads it and complements it (exact acceptance criterion, URL, credentials). If you don't have a story, this context replaces Step 1: you skip it and go straight to Step 2.


4.3 Step by step

Step 1 of 4 — Read the affected story (optional)

Skill used: lippia-neo-read-user-story

Following the TechStore example:

💬 Prompt for the assistant
/lippia-neo-read-user-story user-stories/TS-US-14.md

What happens: the assistant reads the related story (if you provided one) and gives you back the list of acceptance criteria (AC) it contains, along with the app URL and the credentials. From that list, you are the one who identifies which one is failing — the specific rule within the ticket, not the whole ticket. The assistant doesn't mark it on its own.

If you don't have a story available: skip this step entirely and provide the context in Step 2 just as you gave it in "How to start" — the assistant records that there's no associated story.

Checkpoint 1 — Identify the affected AC

At the end of this step you should have a specific text, something like: "Affected AC: 'when a valid coupon is applied, the total must be recalculated with the discount applied'" — the rest of the playbook focuses only on that point.


Step 2 of 4 — Exploration focused on the bug

Skill used: lippia-neo-run-manual-testing

If you ran Step 1 (with a story): paste the trigger to the assistant along with the affected AC you identified in Step 1, all in a single message:

💬 Prompt — Step 2 with a story
/lippia-neo-run-manual-testing
Focus on the scenario of the failing AC: "when a valid coupon is applied, the total must be recalculated with the discount applied"

With a story, you need the test plan

In this mode the assistant looks for the scenario inside the story's test plan (by default test-plans/TS-US-14-test-plan.md). If that plan doesn't exist, it stops with a "no scenarios were found" message. In that case, use the no-story mode below.

If you skipped Step 1 (no story): paste the trigger to the assistant along with the bug description you already gave in "How to start", plus the bug ID:

💬 Prompt — Step 2 without a story
/lippia-neo-run-manual-testing
Explore and reproduce this bug:
- Execution ID: BUG-42
- What fails: the "Apply coupon" button doesn't respond when clicked in the cart
- Expected: it should apply the coupon discount and update the total to pay
- The app is at https://techstore.com/carrito

Why the ID is needed in no-story mode

Without a story, the assistant builds a minimal scenario on its own from your description and runs it directly — you don't need to create a plan first. But the ID is mandatory: it uses it to name the folder where it stores the evidence, and without it, it can't start.

What happens: the assistant goes into TechStore and repeats exactly the steps that would reproduce the problem — it adds a product to the cart, types a valid coupon, clicks "Apply coupon" — documenting what happens at each step. It always does this by navigating the screen, like a person: if the bug is only in an internal service (an API) and isn't visible on screen, comparing its responses is outside the scope of this playbook.

What evidence it captures:

Evidence Description
Reproduction steps The exact sequence to get to the bug
Screenshot of the incorrect state What happens today, wrongly
Screenshot of the expected state If there's an alternative path that does work
Affected technical identifiers So whoever fixes it knows where to look
Underlying technical errors Internal messages, if the bug produces any

Everything is saved in its own folder inside execution/: execution/TS-US-14/ if you used the story, or execution/BUG-42/ if there was no story. That's where the execution report and the screenshots are (in the screenshots/ subfolder).

Checkpoint 2 — Was the bug reproduced?

The bug is reproduced if the assistant can list the exact steps, attach at least one screenshot of the failure, and confirm that the expected behavior doesn't happen.

If the assistant can't reproduce it, that's valuable information too

It documents it as "not reproducible in \<environment> with \<conditions tested>" — it isn't discarded; a record is kept of what was tried.


Step 3 of 4 — Minimal report

Skill used: lippia-neo-write-test-report

The only thing that may be missing is the bug ticket ID

If you didn't mention it before, add it when you ask for the report (it's used for the file name):

💬 Prompt — Step 3
/lippia-neo-write-test-report
The ticket ID is BUG-42.

Everything else (findings, affected AC, URL) the assistant takes from the conversation — you don't need to write anything else.

What happens: the assistant gathers everything from Step 2 into a date- and time-stamped report, which does not overwrite previous reports. This report doesn't include automated test results — this playbook doesn't generate any. The report still has all its usual sections: the automated tests section appears marked as "no data", and that's expected.

The report goes into the project's reports folder (by default test-results/reports/), with a name like TS-US-14-bug-BUG-42-test-report-<date-and-time>.md.

Checkpoint 3 — Is the report complete?

Condition Expected value
Reproduction steps At least 3 specific steps
Attached evidence At least 1 screenshot
Reproducibility status yes, no, or intermittent
Affected behavior With a story: the exact text of the AC. Without a story: the bug description (what fails and what was expected) from Step 2

Step 4 of 4 — Update the ticket (manual)

Skill used: none — you do this step yourself, by hand.

Today there's no automatic integration with Jira or any other ticket manager — you do this step yourself, outside the chat:

  1. Open the bug ticket in your system (or create a new one, with a short title) — the same ID you used in Step 3.
  2. Paste the content of the Step 3 report (the file in test-results/reports/) into the body of the ticket (or attach the file if your system allows it).
  3. Attach the screenshots generated in Step 2 (they're in execution/<ID>/screenshots/).
  4. Set the reproducibility status and assign the ticket to the development team.

If you don't have a ticket manager

If the bug was reported by chat or email, reply in that same thread with the report and the screenshots. The report file is still saved with its date and time as a record.


4.4 What do I end up with after all the steps?

What you get What it's for
The exact reproduction steps Anyone can repeat the bug without guessing
Screenshots of the incorrect state Visual evidence, not just a verbal description
The reproducibility status (yes / no / intermittent) Defines what to do next (see What next?)
A date- and time-stamped report, without overwriting previous reports A versioned history of each reproduction attempt is kept
A "technical" evidence file next to the execution report If later you want to cover the bug with automated tests, another playbook can reuse it as is

5. What next? — What to do based on the result

Situation Next action
✅ Bug confirmed, no automated tests for that module Run the Bug Fix Cycle playbook on the cart: it leaves a test that fails with the bug and confirms it passes once the fix is ready
✅ Bug confirmed, and the fix is already deployed Run the post-deploy smoke test to verify the fix worked
❌ Bug not reproducible It isn't discarded — it's still documented in the ticket and escalated with the context of what was tried
⚠️ Intermittent bug It's marked as such in the report and the conditions of each attempt are added (device, time of day, connection type)

6. Common problems (and what to do)

What you see Why it can happen What to do
The assistant can't access the flow Login is required and no credentials were given Get them from the technical team or from whoever reported the bug
In Step 2 the assistant says "no scenarios were found" The with-story mode was used, but the story has no test plan Repeat Step 2 in no-story mode, with the bug ID
In Step 2 the assistant doesn't start and asks for an ID The no-story mode was used without providing the bug ID Add the ID (for example, BUG-42) to the Step 2 prompt
The bug doesn't reproduce in staging It only happens in production, with real data that staging doesn't have Document that difference in the report, don't force the reproduction
The screenshots come out blank Rendering problem in the test environment Confirm that the environment loads properly in a regular browser before retrying
The report doesn't mention the affected acceptance criterion (AC) Step 1 was skipped without giving context by hand Add the AC manually when asking for the report in Step 3

7. Visual summary of the flow

flowchart TD
    A([🐞 Bug reported<br/>e.g.: Apply coupon does not respond<br/>in the TechStore cart]) --> B{Is there a<br/>related story?}
    B -->|Yes| C["<b>Step 1</b> — Read the story<br/>and identify the affected AC"]
    B -->|No| D
    C --> D["<b>Step 2</b> — Exploration focused<br/>on the bug: steps and screenshots"]
    D --> E["<b>Step 3</b> — Minimal report<br/>with date and time"]
    E --> F["<b>Step 4</b> — Update the ticket<br/>by hand"]
    F --> G{Was it reproduced?}
    G -->|Yes| H([✅ Bug confirmed:<br/>regression or smoke test])
    G -->|No| I([❌ Not reproducible:<br/>document and escalate])
    G -->|Sometimes| J([⚠️ Intermittent:<br/>record conditions])