Frontend UI Change PR
This guide explains what a quality assurance (QA) playbook is and what exactly the "Frontend UI Change PR" playbook does. The idea is that you can read it, understand what happens at each step, and know what information you need to give the assistant so the review of the change turns out well — no programming knowledge required.
In short
- What it's for: confirming that a screen change submitted as a Pull Request didn't break anything before approving it.
- When to use it: when a frontend PR modifies buttons, forms, or screens that already existed, or adds a new interaction.
- What you need: the PR number, the ticket (or the list of changes), the staging URL with the PR deployed, and the project's
qa-stack.yaml. - What you get: the existing suite repaired, new tests for whatever is new (if anything), and a report detailing each fixed identifier, ready to attach to the PR.
- Steps: 6 — read the ticket, go through the screen, repair the suite, generate tests for what's new, run everything, and report (with 2 pauses for you to decide).
What is a playbook? (if this is your first time using one)
It's like, for example, the checklist a pilot follows before takeoff: they don't improvise every time, they follow a proven sequence of steps. A playbook is that same idea applied to testing an application: a written recipe that tells the AI model, step by step, what to do using different elements/tools called "Skills".
What is a "Skill"? A skill is the specific tool that carries out each step of the Playbook. You interact with the playbook — the assistant takes care of using the right tool at each moment.
Part of this flow is also available as an automated plan
Repairing the suite (Step 3) can run without the assistant, from the CI pipeline, with the UI Selector Repair plan: it fixes only the selectors that the change's code renamed, without exploring the app or generating new tests.
1. What does this playbook do?
It's the playbook you use when a developer has already modified a screen and submitted that change as a Pull Request (PR), and you need to confirm that it didn't break anything before approving it.
Think of it this way: a Pull Request (PR) is a code change that a developer has already put together, but that hasn't reached real users yet — it's kept aside, waiting for someone to review it and say "this is fine, it can be published". That "someone" is this playbook.
Unlike other playbooks, here you don't start from a user story — you start from a screen that already exists, already built. The assistant goes through that new screen the way a real user would, and does three things with what it finds:
- If something stopped working just because an internal identifier changed (the button is still there and does the same thing, but its "name" changed internally), it repairs it on its own.
- If it finds something that simply didn't exist before (a new field, a new step), it asks you whether it needs to be covered with a new test.
- If the behavior really changed (it's not a naming issue, the screen doesn't do what the ticket says it should do), it doesn't hide it: it records it as a real problem.
What triggers this playbook
This playbook is triggered by a Pull Request that has already been submitted — the starting point is the change the developer already made, not a QA decision to review something.
2. When should I use this playbook?
Use it when you're facing any of these situations:
| Situation | Business example |
|---|---|
| A Pull Request modifies components, forms, or buttons on a screen that already existed | The TechStore development team submitted a PR that moves the "Checkout" button to the very top and adds a new field to apply a discount coupon in the cart |
| Tests that already existed start failing after the change | TechStore's automated cart tests, which had been passing fine, now fail — you need to confirm whether it's because something really broke or because only an internal identifier changed |
| QA evidence is needed before approving the merge | TechStore's tech lead doesn't want to approve the PR "on someone's word" — they want a report confirming the cart still works |
| The PR adds a new interaction that didn't exist | The discount coupon field is a new feature within the cart and needs its own test, not just a check that the old stuff still works |
Important
- If the change starts from a blank user story, with no previously built screen, this is not the right playbook — for that you use Feature New End to End.
- This playbook needs the project to already have the file with the technology stack (
qa-stack.yaml) set up beforehand. If this is the first time automated QA is being done on this application, you first need to run New Project Bootstrap. - It needs the change to be deployed to a test environment (staging), accessible through a URL — undeployed code isn't enough.
3. Walkthrough example: "TechStore", an online store with a shopping cart
We'll follow the same example business case used in other guides: TechStore, an online electronics store with the typical parts of any store you'd recognize from having shopped online at some point: login, catalog, shopping cart, and checkout.
Let's assume this scenario: the TechStore development team finished programming two changes in the cart and submitted them together as Pull Request #142:
- They moved the "Checkout" button from the bottom of the page to the very top.
- They added a new field to apply a discount coupon.
The PR is already deployed to a test environment, waiting for review before approving the merge to production. That's exactly the scenario this playbook is for.
Throughout the following steps, we'll see what you would answer the assistant at each moment, using this TechStore PR as an example.
3.1 Before you start: what you need to have on hand
You don't need to know how to program, but you do need to have this information on hand:
| Information | TechStore example |
|---|---|
| The Pull Request number or name | PR #142 (the assistant uses it as PR-142, without spaces or symbols, to name the folders and files in every step) |
| The ticket or the description of the changes | Ticket TS-US-14 (or, if there's no ticket, the written list: "button moved" + "new coupon field") |
| The URL of the environment where the PR is deployed | https://techstore-staging.com/preview/142 |
| The existing test suite for that screen (if there is one) | tests/playwright/carrito/ — it lives inside the project's tests folder (paths.tests_dir in qa-stack.yaml, by default tests/playwright/). If it doesn't exist yet, that's not a problem: you tell the assistant at Checkpoint 1 and new tests are generated directly for everything, skipping the Step 3 repair |
Not sure where to get some of this information?
That's normal, you don't need to be a technical person to get it:
- The PR number is given to you by the developer when they say "it's ready to test", or you can see it yourself by going into the repository (GitLab or GitHub) → "Merge requests" / "Pull requests" section → it's listed with a number next to the title.
- The ticket is usually given to you by the developer as a link (Jira or Azure DevOps). If there's no ticket, no problem: it's enough for the developer to tell you in two or three sentences what changed.
- The staging URL is almost always written as an automatic comment inside the Pull Request itself ("Preview deployed at..."), or the developer gives it to you directly.
- The existing suite: you don't need to know this — if you have no idea, write "doesn't exist yet" and the assistant checks it on its own in the project configuration.
Original playbook
The general playbook is in the NEO-releases repository.
3.2 How to start
Before asking for the first step, tell the assistant the PR details, all together in a single message. Following the TechStore example:
Here's the information about the changes implemented in the frontend, I want
you to just take it as context, I don't want you to take any additional action.
Work details:
- PR: PR #142 — use the identifier PR-142 as <PR-ID>, since it's used to
name files and folders in every step
- Ticket / story: TS-US-14
- URL of the environment with the PR: https://techstore-staging.com/preview/142
- Existing suite: carrito (inside tests/playwright/)
UI changes declared by the dev:
- "Checkout" button moved to the very top
- New field to apply a discount coupon
The assistant will use that context throughout all the following steps — you don't need to repeat it every time.
Where each piece of information comes from
- The ticket or the declared changes → you provide them in that message (the "UI changes" are optional if there's a ticket).
- The app URL → the assistant takes it from the ticket if it's there; if not, you mention it in the message.
- The tests folder and the rest of the configuration → the assistant reads them on its own from
qa-stack.yaml.
3.3 Step by step
Step 1 of 6 — Read the ticket or the description of the change
Skill used: lippia-neo-read-user-story
What happens: if there's a ticket, the assistant reads it with this command and extracts what changed, which URL to test it on, and with which credentials. If there's no ticket, this step is skipped entirely — the assistant already has everything it needs from the information you gave it in the starting message (section 3.2), you don't need to ask for anything extra: it uses the list of declared changes as what needs to be reviewed, and notes in the final report that there was no reference ticket. In both cases, if something doesn't match what the developer said in the PR, it points it out to you before moving on.
Checkpoint — Does the summary match the PR?
The summary the assistant shows you has to match what you already knew about the PR. If it mentions something you didn't expect (undeclared features, a broader scope), it's time to clear it up with the developer before continuing.
Step 2 of 6 — Go through the new screen the way a real user would
Skill used: lippia-neo-run-manual-testing
What happens: the assistant actually goes into TechStore from the browser, navigates the cart, clicks the "Checkout" button in its new position, tries the coupon field — and notes down everything it finds, along with the real identifiers of each button and field it touched and screenshots of each modified component.
Since a PR doesn't have a test plan put together, the assistant explores based on the list of changes from Step 1. For that, it absolutely needs the PR identifier (PR-142): without it, it won't start, because it uses it to name the folder where it saves everything. Give it this full message:
/lippia-neo-run-manual-testing
Use us_id: PR-142 and free_exploration (description = UI changes from Step 1,
expected = ACs / declared changes, url = URL from Step 1).
Focus on the PR's UI changes.
Capture the current selectors (data-test, aria-label), screenshots of the modified
components, and document any new navigation flow.
In the notes of each finding, indicate whether it's broken-selector, new-flow, or unexpected-behavior.
Each finding is classified into one of these three categories (the classification is requested by the message above and is written in the notes of each finding):
| Category | What it means | What the assistant does with it |
|---|---|---|
| Broken selector | The button still works the same for the user, but its internal identifier changed | It repairs it in Step 3, without needing your intervention |
| New flow | An interaction that simply didn't exist before (the coupon field) | You're asked at Checkpoint 1 whether it needs to be covered with a new test |
| Unexpected behavior | The screen doesn't respond the way the ticket says it should | It's documented as a real defect |
What this step does NOT do
The exploration only navigates the screen, the way a person would with the browser. It doesn't directly call internal services (APIs) or review the code.
How to know if it went well: you end up with a report with screenshots of each touched component, the list of captured identifiers, and the findings already classified into one of the three categories above. With TechStore, everything ends up in the project's explorations folder (by default execution/):
| File | What it's for |
|---|---|
execution/PR-142/PR-142-scripted-execution-report.md |
The readable report, for you to review |
execution/PR-142/PR-142-scripted-execution-report.json |
The same information in a format for the assistant, used in Step 4 |
execution/PR-142/screenshots/ |
The screenshots as evidence |
⏸ Checkpoint 1 — Is there anything new to cover?
Mandatory pause before Step 3
This is the first moment in the playbook where the assistant stops to wait for your decision before continuing — so it's a good idea to review the Step 2 report (execution/PR-142/PR-142-scripted-execution-report.md) carefully before answering.
| Exploration result | What it means | What you answer the assistant |
|---|---|---|
| At least one new flow was found (in TechStore: the coupon) | A specific test needs to be generated for it | "There are new flows: 'Apply discount coupon'. Continue with healing and generate new tests." |
| Only broken selectors were found, nothing new (or it's a purely visual change, or the new part already had tests from before) | Repairing what already existed is enough | "Only broken selectors, no new flows. Continue with healing only." |
| There was no previous suite for this screen (you said "doesn't exist yet" in section 3.2) | There's nothing to repair — it must be treated as if everything were new, even if no specific new flow was detected | "There's no previous suite. Treat it as a new flow and generate tests for everything from scratch." |
Step 3 of 6 — Repair the existing suite
Skill used: lippia-neo-run-and-heal
What happens: the assistant runs the existing cart tests against the new screen. Before the first run, it fills in any pending data left in the tests (values marked as TODO) and replaces hand-written credentials with the ones from the project's credentials file. Then, for each test that fails, it finds out the reason: whatever fails only because an internal identifier changed (for example, the "Checkout" button now has a different technical name because it was moved), it fixes by looking it up on the current screen. Whatever fails because the behavior really changed, it doesn't force it to pass — it marks it as a real problem that needs a person to look at it.
Give it this message, indicating the existing suite's folder (relative to the project's tests folder):
/lippia-neo-run-and-heal
Run with max_iterations=3 and include_dirs=["carrito"]
In fix_applied for each repaired selector, indicate the line, the previous selector, and the new selector.
If the project uses Lippia instead of Playwright
With Lippia, the run can't be limited to a folder: the assistant runs all the project's tests and notes that it couldn't limit it.
If there was no previous suite
If you answered "doesn't exist yet" and marked it as a new flow at Checkpoint 1, this step is skipped entirely, there's nothing to repair. You go straight to Step 4, where everything is generated from scratch.
What can and can't be repaired:
| It can be repaired like this | It's never repaired like this — it's marked as a defect |
|---|---|
| Updating the internal identifier of a button or field whose name changed | Changing what was expected to happen so it matches incorrect behavior |
| Updating a URL or navigation path that the PR modified | Marking a test as "skip" without writing down the reason |
| Adjusting the wait time if the new screen takes longer to load | Deleting the check instead of fixing the identifier |
Watch out for changed checks
If a test fails because the expected result doesn't match, the assistant may update that check when it believes the screen change was intentional. Review those fixes at Checkpoint 2 against the right-hand column of the table: if any of them "bends" the test to fit incorrect behavior, it's a defect, not a repair.
How to know if it went well: you end up with a summary of how many tests were failing at the start, how many passed at the end, how many attempts were needed (out of a maximum of 3), how many identifiers were fixed, and how many tests were marked as defects. The assistant stops when everything passes, when an attempt doesn't improve anything compared to the previous one, or when it reaches 3 attempts. The details of each run are saved in the project's reports folder (file run-and-heal-<date>.json).
⏸ Checkpoint 2 — Was too much repaired?
Mandatory pause before continuing
Fixing a lot of things isn't a problem with the tests — it's a sign that the PR's real change is bigger than what the developer declared.
| Warning sign | What it implies |
|---|---|
| ⚠️ More than 5 fixes in the same file | The component was rebuilt, not touched up |
| ⚠️ 30% or more of the suite needed fixing | The real scope is larger than declared |
| ⚠️ Flows the ticket doesn't mention were fixed | The PR touched more screens than announced |
| ⚠️ More than 2 attempts were needed and it still wasn't passing | The new screen may have inconsistencies of its own |
If you see any warning sign, reply to the assistant:
There's excessive repair in the cart module — [details of what caught your attention].
Pause and check with the dev whether the PR's scope is correct.
If the number of fixes seems reasonable for the described change:
Repair within expectations. Continue with [generating new tests / the report].
Step 4 of 6 — Generate tests for what's new (only if you said at Checkpoint 1 that there was something new)
Skill used: lippia-neo-generate-test-suite
What happens: the assistant writes specific tests for the discount coupon, using the real identifiers it captured in Step 2 — without touching or overwriting the cart tests that already existed. For that, you point it to the results file that Step 2 left (the .json, exactly as the assistant returned it — you don't need to put it together by hand):
/lippia-neo-generate-test-suite
Use manual_execution_report_path: execution/PR-142/PR-142-scripted-execution-report.json
Generate tests ONLY for the new flows identified in Step 2.
Use the selectors captured during the exploration. Don't regenerate existing tests.
If something failed during the exploration
If any walkthrough in Step 2 ended with an error, the assistant also generates a specific test for that error (this is mandatory), so it's monitored in the future.
How to know if it went well: a new tests folder appears, separate from the original suite, with at least one test for each new flow identified. With TechStore, it ends up in tests/playwright/PR-142/ (for example, tc-001-aplicar-cupon.spec.js). The assistant confirms the actual folder in its response — use that one in Step 5.
Step 5 of 6 — Run everything again until it passes (only if Step 4 was run)
Skill used: lippia-neo-run-and-heal
What happens: the assistant runs the repaired suite from Step 3 together with the new tests from Step 4, all together, and repairs whatever is still pending in the new tests (up to 2 attempts). If something really keeps failing, it's documented as an application defect — it's not hidden. The original suite should already be passing since Step 3: if it fails again here, it's documented as a side effect of the new tests.
/lippia-neo-run-and-heal
Run with max_iterations=2 and include_dirs=["carrito", "PR-142"]
Always specify both folders
Don't run the entire tests folder: always point to the cart suite and to the new tests folder. That way you don't drag in tests from other modules or applications that may live in the same repository.
Step 6 of 6 — Final report with evidence
Skill used: lippia-neo-write-test-report
What happens: the assistant gathers everything done in the previous steps (the exploration from Step 2 and the runs from Steps 3 and 5) and writes a final report with screenshots, and a line-by-line detail of each identifier that was fixed (from which name to which), ready to attach to the PR as evidence. That detail is added because you ask for it in the message:
/lippia-neo-write-test-report
Use us_id: PR-142
Include the audit trail of repaired selectors in the report
How to know if it went well: the report includes the base suite passing, the details of each fix, the new tests (if any), and it's saved with the date and time without overwriting previous reports. With TechStore, it ends up in the project's reports folder (by default test-results/reports/), for example: PR-142-test-report-2026-07-17-1015.md.
Why does the report say the coverage is \"not assessable\" or that the source reference is missing?
Coverage is measured against the ticket's acceptance criteria. If the PR had no ticket (or the ticket had no criteria), the report has nothing to measure against and marks it as "not assessable"; and if there was no ticket, it indicates this in the header as a missing source reference. This is expected and doesn't indicate any problem.
3.4 What do I have after completing all the steps?
| What you get | What it's for |
|---|---|
| An exploration report with screenshots | Evidence that someone (the assistant) actually tested the PR's new screen |
| The existing suite repaired and passing | Confirmation that the change didn't break the cart |
| The details of each fixed identifier, from what to what | Traceability so the PR reviewer can confirm each selector change is correct |
| New tests for what the PR added (if there was something new) | Coverage for the discount coupon, ready to run in the future |
| A final report with date and time | The document you attach to the PR as backing to approve — or stop — the merge |
From here on, the TechStore team can approve the merge of PR #142 with real evidence that the cart still works and that the new coupon was tested.
4. What if the pipeline runs it on its own? — Smoke branch or full regression
In the future, this playbook will also be able to run on its own, inside the pipeline, without anyone asking for it. In that mode there are no pauses for you to decide: instead, before starting, a classifier tool (lippia-neo-classify-ui-scope) reads the PR's code change and its description, and decides how big the change is:
| Classifier decision | What it runs |
|---|---|
| Smoke (small, isolated change) | Just a quick pass of the existing suite, repairing whatever it can (up to 2 attempts). No exploration or report |
| Full regression (broad or risky change) | The 6 steps of this guide, one after another, without the checkpoint pauses |
Some criteria it uses to decide:
| Signal in the PR's change | Scope it chooses |
|---|---|
| Change in a single isolated component | Smoke |
| Change in styles or colors | Smoke |
| Change in the overall structure of the screens or in navigation | Full regression |
| Change in forms or business flows | Full regression |
| Change in login or in screen access permissions | Full regression |
| More than 10 files modified | Full regression |
| A new page, route, or component, or a scope that isn't clear | Full regression (when in doubt, it always chooses the full one) |
In the meantime
Until this automatic mode is active, you decide the scope manually at Checkpoint 1.
5. Common problems (and what to do)
| What you see | Why it might happen | What to do |
|---|---|---|
| The Step 2 exploration doesn't find anything to click | The PR's staging environment hasn't finished deploying | Confirm with the technical team that the URL is active before retrying |
| The Step 3 repair reaches 3 attempts (or stops improving) without passing | The component was rebuilt from scratch, not touched up | Pause, review the Checkpoint 2 signs, and check the PR's real scope with the developer |
| The new tests from Step 4 use incorrect identifiers | The Step 2 exploration didn't capture all the elements of the new flow | Repeat Step 2 with an explicit focus on the elements of that flow |
| The base suite fails again in Step 5 (it was green in Step 3) | The new tests left data behind that interferes with the others | Add state cleanup between tests and repeat Step 5 |
| The final report shows no fixes | It's the best possible scenario: the PR didn't break any existing identifier | It's not an error — continue normally |
| The assistant stops saying the testing tool is "in STUB state" | The project's testing tool isn't fully supported yet | It's an issue for the technical team, not something you can solve on your own |
| The tests fail at login | The project's test credentials (.env.tests file) are empty or incomplete |
Ask someone on the technical team to fill in the real values — they're never written by hand inside the tests — and repeat Step 3 or 5 |
6. Visual summary of the flow
flowchart TD
A([🔀 Frontend PR in staging<br/>e.g.: PR #142 in TechStore,<br/>button moved + new coupon]) --> M{Who runs it?}
M -->|You, manually| P1
M -->|The pipeline, on its own| CL{How big<br/>is the change?}
CL -->|Small and isolated: smoke| SM["Quick pass of the existing<br/>suite, no exploration<br/>or report"]
CL -->|Broad or risky: full| FU["Steps 1 to 6<br/>without pauses"]
SM --> Z
FU --> Z
P1["<b>Step 1</b> — Read the ticket<br/>or the description"] --> P2
P2["<b>Step 2</b> — Go through the new screen<br/>and classify findings"] --> C1{"⏸ Checkpoint 1<br/>Anything new?"}
C1 -->|Yes, there is a new flow| P3A
C1 -->|No previous suite existed| P4
C1 -->|No, only broken selectors| P3B
P3A["<b>Step 3</b> — Repair the existing<br/>suite"] --> C2A{"⏸ Checkpoint 2<br/>Was too much repaired?"}
P3B["<b>Step 3</b> — Repair the existing<br/>suite"] --> C2B{"⏸ Checkpoint 2<br/>Was too much repaired?"}
C2A -->|Yes| X([⚠️ Pause and check<br/>with the developer])
C2B -->|Yes| X
C2A -->|No, it is proportional| P4
C2B -->|No, it is proportional| P6
P4["<b>Step 4</b> — Generate tests<br/>for what's new"] --> P5
P5["<b>Step 5</b> — Run everything again<br/>until it passes"] --> P6
P6["<b>Step 6</b> — Final report<br/>with evidence"] --> Z
Z([✅ PR ready to approve<br/>or stop the merge,<br/>with real evidence])