Skip to content

Feature - New End-to-End

This guide explains what the NEO engine does when it runs this plan. The idea is that you can read it, understand what happens at each step without anyone stepping in, and know what the MR needs to include for the work to turn out well.

In short

  • What it's for: getting a new feature tested end to end, with no human intervention: from the user story to a working automated test suite and a final report.
  • When it runs: when an MR is opened with the feature label. It's triggered by a CI rule (Model 4), not by a person.
  • What it needs: the MR description must contain the path or the link (Jira / Azure DevOps) of the user story, and the project's qa-stack.yaml must already be set up.
  • What it leaves: a test plan, the evidence from the exploration, an automated test suite that has been run and repaired, and a final report.
  • Steps: 6 — read the story, build the test plan, explore the app, generate the suite, run and repair, report. No pauses.
What is a plan? (if this is the first time you see one)

A plan is a recipe that a machine runs instead of a person. Like a playbook, it chains several skills in a fixed order. The difference is that a playbook is followed by the AI assistant together with you, with pauses for you to decide; a plan is run by the NEO engine on its own, end to end, usually from the CI pipeline.

See Plans for the full comparison between plan and playbook.


1. What does this plan do?

It's the automated version of the Feature - New End-to-End playbook: when a new feature arrives, it gets it tested end to end without anyone having to ask the assistant.

Its intent is that no new feature reaches review without tests. As soon as the team marks the MR as feature, the pipeline:

  1. Finds the user story (in a file in the repo, in Jira, or in Azure DevOps) and extracts its acceptance criteria.
  2. Builds a test plan that covers all of those criteria.
  3. Walks through the real application following that plan, the way a person would, and records what it finds.
  4. Turns that exploration into automated tests, in the project's framework.
  5. Runs them and repairs the ones that fail for minor reasons (for example, a button that was renamed).
  6. Writes a final report that brings it all together.

When it's done, the MR has all the QA evidence for the feature, ready for the team to review.


2. When does it run?

Situation Business example
An MR is opened with the feature label The TechStore team finishes building the discount coupon in the cart and opens the MR with the feature label
You want every new feature to reach review with its tests without depending on someone remembering to ask TechStore's QA lead wants every new-feature MR to come with a plan, tests, and a report, no exceptions

It's triggered by a CI rule (Model 4 — Pure declarative). The decision of which plan to run lives in the pipeline, not in the plan or in the engine. The CI templates generated by lippia-neo-ci-bootstrap already include this rule (see Set up CI).

Important

  • The MR description must contain the path or the link of the user story (for example, user-stories/US-045.md or the URL of the Jira ticket). Free text describing it won't work: the plan doesn't make up the story.
  • If the story is in Jira or Azure DevOps, the project must have that MCP configured in qa-stack.yaml (see Set up the MCPs).
  • The plan assumes the project already has automated QA set up (qa-stack.yaml). If not, you first need to run the project bootstrap playbook.
  • The application must be deployed and reachable at the URL listed in the story: Step 3 walks through it live.

3. How does it differ from the playbook?

The plan does the same work as the Feature - New End-to-End playbook, but with nobody on the other side:

Playbook Plan
Who runs it The AI assistant, talking with you The engine, on its own, from CI
How it starts You ask the assistant to test the story An MR is opened with the feature label
Pauses for review Two: validating the test plan and reviewing the repair None. Human review happens afterwards, on the MR, with the report
If the story isn't enough to test The assistant asks you and waits The plan stops and records the reason
Pull request with the evidence Step 7 creates it It doesn't create one: the plan already runs on an open MR, and CI can comment the result there with neo report

4. Guiding example: "TechStore" adds a discount coupon

We continue with the same business case as in the playbook: TechStore, the online electronics store, adds the ability to apply a discount coupon in the shopping cart. This time nobody asks the assistant anything: the developer opens the MR with the feature label and the pipeline does the rest.

4.1 What the plan needs to receive

The plan has a single required piece of data. In CI, the neo event command fills it in on its own from the MR:

Data What it is TechStore example
description The reference to the user story: a path to a .md in the repo, or the URL of a Jira or Azure DevOps ticket user-stories/US-045.md (it's the MR description)

Everything else (folders, test framework, language, app URL, credentials) comes from the story and from the project's qa-stack.yaml.

4.2 How it's triggered

In the pipeline, the CI rule builds the input from the MR and runs the plan:

rules:
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_LABELS =~ /feature/'
script:
  - neo event --provider gitlab --out input.json
  - neo run plans/feature-new-end-to-end.yaml --input-file input.json --workspace . --out result.json

To try it from your computer, without CI:

neo run plans/feature-new-end-to-end.yaml --workspace . \
  --data '{"description":"user-stories/US-045.md"}'

4.3 Step by step

Step 1 of 6 — Find and read the user story

Skills used: lippia-neo-resolve-user-story-source, lippia-neo-fetch-user-story (fetches the ticket from Jira or Azure DevOps, depending on where it is), and lippia-neo-read-user-story

What happens: first the plan decides where the story is: if it's a path, it reads it from the repo; if it's a Jira or Azure DevOps link, it fetches it from there (only the matching step runs; the other one is skipped). Then it reads it and extracts the essentials: the summary, the acceptance criteria, the app URL, and the test credentials.

With TechStore: it reads user-stories/US-045.md and extracts the discount coupon's criteria (for example, "when a valid coupon is applied, the total is recalculated").

First stopping point

If the story doesn't have the minimum needed to test it (for example, it has no acceptance criteria), the plan stops here with status stopped. It's not an engine error: it's an expected stop, so that no tests are generated on an incomplete story.


Step 2 of 6 — Build the test plan

Skill used: lippia-neo-create-test-plan

What happens: from the acceptance criteria, it builds an exhaustive test plan: one scenario for each case to test, with its steps and its expected result.

What you get: the test plan in test-plans/US-045-test-plan.md.

Second stopping point

If the test plan comes out with no scenarios at all, the plan stops here (stopped): there's no point exploring or generating tests with nothing to test.


Step 3 of 6 — Explore the application following the test plan

Skill used: lippia-neo-run-manual-testing

What happens: the engine opens a browser and walks through the real application following each scenario in the test plan, the way a person would: it applies the coupon, checks the total, tries an expired coupon. It records what happened in each scenario, the findings (things that don't behave the way the story says), screenshots, and the real selectors on the screen.

What you get: an exploration report and a "handoff" file (execution/US-045/US-045-scripted-execution-report.json) that the next step uses.


Step 4 of 6 — Generate the automated test suite

Skill used: lippia-neo-generate-test-suite

What happens: it turns what was explored in Step 3 into automated tests, written in the project's framework (Playwright, Lippia, Cypress, or Pytest, according to qa-stack.yaml). It uses the real selectors found during the exploration, so the tests are born pointing at the actual screen.

What you get: the new test files, in the project's tests folder.


Step 5 of 6 — Run and repair the tests

Skill used: lippia-neo-run-and-heal

What happens: it runs the suite and, if a test fails for a minor reason (a fragile selector, a short wait), it repairs it and runs it again. It repeats that cycle up to 10 times or until everything passes.

It repairs tests, not the application

If a test fails because the application really doesn't meet an acceptance criterion, this step doesn't "fix" it so that it passes: it stays as a failure in the report. Repairing means correcting the test, never hiding a bug.


Step 6 of 6 — Write the final report

Skill used: lippia-neo-write-test-report

What happens: it brings everything together in a single report: the story's acceptance criteria, which scenarios were explored and what was found, which automated tests were generated, and how their run ended. It includes a matrix that maps each acceptance criterion to the tests that cover it.

What you get: the report, with date and time, in the project's reports folder (by default test-results/reports/).


4.4 How does it end?

How it ended Plan status What it means What to do
✅ All steps ran success The feature ended up with a plan, tests, and a report The team reviews the report and the tests in the MR
⏹ The story wasn't enough to test, or the test plan came out empty stopped Expected stop at Step 1 or Step 2 Complete the story (acceptance criteria, URL) and run the pipeline again
❌ A skill returned something malformed, or something broke failed / error The step failed, not the application Check the job log and result.json

In CI, stopped exits with code 0 and failed/error with code 1 (see How to run it). Everything each step wrote is in result.json.


5. Common problems (and what to do)

What you see Why it can happen What to do
The plan ends stopped at Step 1 The story has no acceptance criteria, or the MR description doesn't contain the story's path or link Complete the story, or put its path/link in the MR description, and run again
Step 1 can't find the Jira or Azure DevOps story The MCP isn't configured in qa-stack.yaml, or the credentials are missing from the CI variables See Set up the MCPs and Set up CI
Step 3 can't open the application The app isn't deployed, or the story's URL isn't reachable from CI Confirm the URL responds from the runner and run again
Some tests keep failing after Step 5 The application doesn't meet some acceptance criterion, or the failure couldn't be repaired in 10 cycles Check the report: if it's a real bug, continue with the Bug Fix Cycle playbook
Tests fail at login The test credentials (.env.tests) are empty or incomplete in CI Fill in the CI variables; they're never hard-coded inside the tests

6. Visual summary of the flow

flowchart TD
    A([🏷 MR with feature label<br/>e.g.: TechStore, discount coupon]) --> B["<b>Step 1</b> — Find and read<br/>the user story"]
    B --> C{Is the story<br/>enough to test?}
    C -->|No| X([⏹ stopped])
    C -->|Yes| D["<b>Step 2</b> — Build the test plan"]
    D --> E{Does it have scenarios?}
    E -->|No| X
    E -->|Yes| F["<b>Step 3</b> — Explore the app"]
    F --> G["<b>Step 4</b> — Generate the suite"]
    G --> H["<b>Step 5</b> — Run and repair"]
    H --> I["<b>Step 6</b> — Final report"]
    I --> J([✅ MR with plan, tests, and report])

Original plan

The plan file is plans/feature-new-end-to-end.yaml, in the engine repository (crowdar-engine-mvp). It's commented step by step; the format reference is in Anatomy of a plan.