Skip to content

Feature - New End-to-End

This guide is meant to help you understand what the AI assistant does when it runs this playbook. The idea is that you can read it, understand what happens at each step, and know what information you need to give the assistant so the work turns out well — no programming knowledge required.

In short

  • What it's for: getting a new feature fully tested end to end, from the acceptance criteria all the way to a final report and a pull request with all the evidence.
  • When to use it: when a new user story enters the sprint and doesn't have any tests yet.
  • What you need: the user story with its acceptance criteria, the application URL, and the project's qa-stack.yaml already set up.
  • What you get: an approved test plan, a working automated test suite, a final report, and a pull request ready for review.
  • Steps: 7 — read the story, build the plan, explore the app, generate the suite, run and repair, report, and open the pull request (with two human review pauses along the way).
What is a playbook? (if this is your first time using one)

Think of it as a cooking recipe for the AI assistant. The playbook is the recipe you follow, interacting with the assistant step by step, in order, without skipping anything, so the result is always the same no matter who asks for it.

Each step of the recipe uses a specific tool (in technical jargon they're called "skills"), a bit like a kitchen utensil: one is for peeling, another for cutting, another for baking. You don't need to know how each utensil works inside — you only need to know which ingredient goes into each step (for example, the user story, or the application URL) so the final dish turns out well.

A playbook = an ordered guide of steps. A skill = the specific tool that carries out each step. You interact with the playbook — the assistant takes care of using the right tool at each moment.


Also available as an automated plan

This same flow can run without the assistant, from the CI pipeline, with the Feature - New End-to-End plan: the NEO engine runs it on its own, without pauses, when an MR is opened with the feature label.


1. What does this playbook do?

It's the playbook you use when a new feature arrives in the project and it needs to be tested end to end, from scratch: from the acceptance criteria to a final report and a pull request (a proposal of changes that the development team reviews before accepting it) with all the evidence.

Unlike other playbooks designed to run everything in one go without pauses, this one stops twice so a person can review and approve before moving on:

  1. After building the test plan (to confirm that the right things are going to be tested).
  2. After repairing the tests that failed (to confirm that the repair was real and not just cosmetic).

Those two pauses are the main reason to choose this playbook over another one: it gives you visibility and control over the intermediate results, which is key when the team needs that evidence before approving the merge (integration) of the code.

When it's done, the new feature ends up with:

  • A complete test plan, already reviewed and approved by a person.
  • An automated test suite, generated and working.
  • A final report with the result.
  • A pull request with all those artifacts, ready for the team to review.

2. When should I use this playbook?

Use it when you're facing any of these situations:

Situation Business example
A new user story enters the sprint and there are no previous tests for that feature The team adds the ability to apply a discount coupon in the shopping cart, something that has never been tested before
You want to go from "acceptance criteria" to "working automated tests" in a single work session The Product Owner (PO) defines 3 acceptance criteria for the discount coupon and, on the same day, you need automated tests covering them
You need complete evidence before approving the release to production (plan + tests + report, as part of the Definition of Done) The team can't close the discount coupon feature without attaching the test plan and the report to the pull request

Important

This playbook assumes the project already has automated QA set up (the file with the project's technical configuration, qa-stack.yaml, and the test pipeline already exist). If the project is brand new and doesn't have any of that yet, you first need to run the project setup playbook — this playbook is for adding a new feature, not for getting a project ready from scratch.


3. Walkthrough example: "TechStore" adds a new feature

We'll continue with the same business case: TechStore, the online electronics store with a shopping cart. TechStore has already gone through the bootstrap playbook — it already has automated QA working for everything that existed until now.

Now the development team adds a new feature: the ability to apply a discount coupon in the shopping cart. Nobody has tested this feature yet — it was just finished. That's exactly the scenario this playbook is for.

Throughout the following steps, we'll see what you would answer the assistant at each moment, using the discount coupon as an example.

3.1 Before you start: what you need to have on hand

You don't need to know how to program, but you do need to have this information on hand:

Information TechStore example
A user story in a file or in the Jira/Confluence URL where it lives, with its acceptance criteria, the app URL, and the credentials to log in "As a customer I want to be able to apply a discount coupon in the shopping cart", saved in user-stories/US-045.md or wherever you defined it in the file that contains the project's technical configuration, qa-stack.yaml
The application URL where the feature will be tested https://techstore-staging.com
The project's qa-stack.yaml file already set up That's where the assistant finds the folders, the testing tool, and the project's language on its own — you don't need to tell it

Original playbook

The general playbook is in the NEO-releases repository.

3.2 How to start the conversation

Before you start, tell the assistant which story you're going to work on and in which environment, something like this:

💬 Prompt for the assistant
Here's some information, I just want you to save it as context:
I want to run the full flow for the discount coupon story
(user-stories/US-045.md). The app is at https://techstore-staging.com.

The assistant will remember that context throughout all the following steps — you don't need to repeat it every time.

Where each piece of information comes from

  • The story: you mention it in this message (the path to the file).
  • The app URL: the assistant takes it from the story in Step 1. If the story doesn't include it, it leaves it empty (it doesn't make it up) and, when building the plan, it uses the environment URL configured in qa-stack.yaml.
  • Folders, testing tool, and language: the assistant reads them on its own from qa-stack.yaml.

3.3 Step by step

Note

This is an example: you need to follow the original playbook.

Step 1 of 7 — Read the user story

Skill used: lippia-neo-read-user-story

What happens: the assistant opens the story file and extracts the summary, the acceptance criteria, the app URL, and the credentials needed to log in. With that, it builds the base context that all the following steps will use. You see the result right there, in the conversation with the assistant.

Example with the discount coupon: the assistant identifies acceptance criteria such as "a valid coupon takes 10% off the total", "an expired coupon shows an error message", and "a non-existent coupon is not applied".

Checkpoint — Complete criteria

Check that the criteria the assistant extracted are complete and really match the story. If something is ambiguous or incomplete, it's better to sort it out with the Product Owner before moving on.


Step 2 of 7 — Build the test plan

Skill used: lippia-neo-create-test-plan

What happens: the assistant turns each acceptance criterion into concrete test scenarios — not just the "happy path" (the coupon works correctly), but also the edge cases (expired coupon, non-existent coupon, already-used coupon, empty cart, etc.). All of this is saved in a test plan, with numbered cases, in the project's plans folder (by default test-plans/), for example: test-plans/US-045-test-plan.md.


⏸ Human review pause 1 — Validate the test plan

Mandatory pause — action required before continuing to Step 3

Before the assistant keeps going, someone has to review the plan and confirm:

  • There is at least one test case for each acceptance criterion
  • The most important cases and the most critical edge cases are covered
  • The test cases are actually possible to carry out in the test environment
  • There are no repeated test cases or ones that contradict each other

If you find something odd, ask the assistant to fix it directly, for example:

💬 Prompt — Adjust the plan
Adjust the plan: case 04 assumes a coupon that doesn't exist in the staging environment,
replace it with one that does exist.

Only once the plan is approved do you continue with Step 3.


Step 3 of 7 — Manual exploration of the feature, guided by the test plan from the previous step

Skill used: lippia-neo-run-manual-testing

What happens: the assistant runs the plan's scenarios by actually navigating the application in a browser, the way a person would. While doing so, it notes down the names of the buttons and fields it uses (so the automated tests can find them later), saves screenshots as evidence, and documents any odd behavior it finds as a possible defect.

With the discount coupon: the assistant goes into the TechStore cart, enters a valid coupon, and confirms that the discount is applied correctly; then it tries an expired one and confirms that the expected error message appears — and so on with each case in the plan.

The exploration only goes through screens

This exploration only navigates the interface (the app's screens). It doesn't send direct requests to an API. If the feature is API-only, or combines screens and API, the API part isn't explored here: its automated tests are built directly from the plan in Step 4.

What you have at the end of this step: an exploration report in the execution/US-045/ folder (one readable by people and another that the assistant uses in the next step), along with the screenshots.

Checkpoint — Well-documented exploration

Check that the exploration was well documented, with clear evidence for each tested case and with the buttons and fields used clearly identified — it's the input used to build the automated tests in the next step.


Step 4 of 7 — Generate the automated test suite

Skill used: lippia-neo-generate-test-suite

What happens: with the plan from Step 2 and what was discovered by navigating the app in Step 3, the assistant generates the corresponding automated test files, using the testing tool the project already has configured, which is specified in the file with the project's technical configuration, qa-stack.yaml.

What you have at the end: one automated test file for each case in the plan (for example, one for "valid coupon", another for "expired coupon", another for "non-existent coupon"). They're all together in a folder named after the story ID, inside the project's tests folder: with the default configuration, tests/playwright/US-045/. Each file is named with the case number and a short description, for example tc-001-cupon-valido.spec.ts.

Checkpoint — One file per case

Check that there is a generated file for each test case in the plan — if one is missing, something was left uncovered.


Step 5 of 7 — Run and repair the tests

Skill used: lippia-neo-run-and-heal

What happens: the assistant runs the entire generated test suite. If any test fails, it analyzes why and tries to fix it on its own — it repeats this attempt up to 3 times, until all the tests pass or until there are no more improvements from one attempt to the next, whichever comes first. At the end, it leaves a summary of what was repaired and how many attempts it used, also saved as a file in the reports folder (by default test-results/reports/).

To limit the run to this story's tests, give it this full message (replacing US-045 with your story's ID):

💬 Prompt for the assistant
/lippia-neo-run-and-heal
Run with max_iterations=3 and include_dirs=["US-045"]

At the end of this step, one of these situations can happen:

Result What it means
✅ All green All the tests passed (some maybe after a repair)
⚠️ It stopped because it stopped improving Between one attempt and the next, no more tests were fixed, so the assistant stopped before the 3 attempts
❌ The 3 attempts ran out Some tests are still failing — you need to check whether it's a real defect in the app or a limitation of the environment

Checkpoint — Write down the repair summary

Before moving on, write down the result (which of the three in the table), how many tests passed and how many failed at the end, and how many of the 3 attempts were used.


⏸ Human review pause 2 — Review the repair

Mandatory pause — action required before continuing to Step 6

Before accepting the repair, someone has to confirm:

  • The tests that failed at first now pass for the right reason (the real problem was fixed, the check wasn't "turned off" so it would stop failing)
  • No test was disabled without a documented reason
  • If any test is still failing: it's documented whether it's a real defect in the app or a limitation of the test environment

If a repair doesn't convince you, tell the assistant:

💬 Prompt — Revert a repair
The expired coupon case was repaired by changing the selector, but the expected
behavior isn't the correct one — revert the fix and mark it as a known defect.

Only once the repair is approved do you continue with Step 6.


Step 6 of 7 — Generate the final report

Skill used: lippia-neo-write-test-report

What happens: the assistant gathers everything that happened in the manual exploration (Step 3) and in the repair (Step 5) into a single final report, and compares the result against the acceptance criteria it took from the story in Step 1. Each time this step runs, a new report is generated — a previous report is never overwritten, so the complete history is kept.

Where it ends up: in the project's reports folder (by default test-results/reports/), with the date and time in the name, for example: test-results/reports/US-045-test-report-2026-07-17-1015.md.

Checkpoint — Complete report

Check that the report includes a summary of the final result, the list of defects found, the details of what was repaired (and in how many attempts), and which acceptance criteria were actually validated.

Why might the coverage say \"not assessable\"?

Coverage is measured against the story's acceptance criteria. If the assistant doesn't have them on hand (for example, because the conversation was resumed in another window), that section is marked as "not assessable" instead of making it up. In that case, ask it to read the story again before generating the report.


Step 7 of 7 — Create the pull request with the evidence

Skill used: none — it's a manual step.

Manual step

This is the only completely manual step: the assistant can help you write the text, but a person is the one who uploads the changes and opens the pull request.

What you need to do:

  1. Create a new branch for the test files, starting from the main branch (for example, qa/US-045-tests). If you don't know how, ask the development team so you follow the standard way your team has defined.
  2. Upload the test plan (test-plans/US-045-test-plan.md), the generated suite (the tests/playwright/US-045/ folder), and the final report (in test-results/reports/).
  3. Open the pull request, with the description linked to the original user story.

Before considering the work done, check:

  • The test plan is uploaded
  • The test suite is green (or the red cases are documented as known defects)
  • The final report is uploaded
  • The pull request is open and linked to the user story

3.4 What do I have after completing the seven steps?

What you get What it's for
A reviewed and approved test plan Evidence of what was decided to test and why
A record of the plan-guided exploration with screenshots Visual evidence of how the new feature behaved
An automated test suite So the feature keeps being tested on its own from now on
A test repair summary A record of what was fixed and how
A final report Consolidated evidence to attach to the Definition of Done
A pull request with all of the above The complete package, ready for the team to review and approve

From here on, the discount coupon feature in TechStore has automated test coverage — ready to join the set of tests that run on their own every time new code is uploaded.


4. Common problems (and what to do)

What you see Why it might happen What to do
The test plan ends up with incomplete or very generic cases The story's acceptance criteria are ambiguous Complete the story with the Product Owner and repeat Step 2
The manual exploration fails when navigating the app The URL is wrong, or the test environment is down Check that the TechStore URL is accessible (and that it's the correct one in qa-stack.yaml)
"Empty" or useless automated tests are generated The manual exploration in Step 3 wasn't well documented Repeat Step 3, focusing on properly documenting each screen and button used
The test repair doesn't converge after 3 attempts The tests depend on data that changes between runs (for example, a coupon that was already used) Ask for fixed test data to be added and repeat from Step 5
The assistant warns that it can't generate the suite because the testing tool is "in STUB state" The testing tool configured in the project isn't fully supported yet Let the technical team know — you need to use another supported tool or wait until it's ready
The tests fail at login The project's test credentials (.env.tests file) are empty or incomplete Ask someone on the technical team to fill in the real values — they're never written by hand inside the tests — and repeat Step 5
The pull request doesn't pass the team's review The tests turned out very fragile (for example, fixed wait times that don't match reality) Review the generated test files and adjust them together with the technical team

5. Visual summary of the flow

flowchart TD
    A([🆕 New user story<br/>e.g.: discount coupon]) --> B
    B["<b>Step 1</b> — Read the user<br/>story"] --> C
    C["<b>Step 2</b> — Build the test<br/>plan"] --> D{Plan approved<br/>by a person?}
    D -->|No, adjust| C
    D -->|Yes| E["<b>Step 3</b> — Explore<br/>the feature"]
    E --> F["<b>Step 4</b> — Generate the automated<br/>test suite"]
    F --> G["<b>Step 5</b> — Run and repair<br/>the tests"]
    G --> H{Repair approved<br/>by a person?}
    H -->|No, revert or fix| G
    H -->|Yes| I["<b>Step 6</b> — Generate the<br/>final report"]
    I --> J["<b>Step 7</b> — Create the pull request<br/>with the evidence"]
    J --> K([✅ New feature with<br/>automated coverage])