UI Selector Repair
This guide explains what the NEO engine does when it runs this plan. The idea is that you can read it, understand what happens at each step without anyone stepping in, and know what to expect when a frontend change reaches the pipeline.
In short
- What it's for: when a frontend change renames buttons, fields, or screens, automatically fixing the tests that used them, before they start failing.
- When it runs: when a frontend code change reaches the pipeline, before deploying it.
- What it needs: the change's "diff" (the list of code lines that changed) and the project's
qa-stack.yaml. - What it leaves: the affected tests fixed and run, plus the list of what couldn't be fixed on its own, for a person to review.
- Steps: 3 — find the affected tests, fix their selectors, and run them. No pauses.
What is a plan? (if this is the first time you see one)
A plan is a recipe that a machine runs instead of a person. Like a playbook, it chains several skills in a fixed order. The difference is that a playbook is followed by the AI assistant together with you, with pauses for you to decide; a plan is run by the NEO engine on its own, end to end, usually from the CI pipeline.
See Plans for the full comparison between plan and playbook.
What is a selector?
An automated test needs to find on the screen the button or field it's going to use. To do that it uses a selector: an "address" for the element, such as its identifier or its text (for example, data-test="checkout-btn"). If the developer renames that identifier, the test can no longer find the button and fails, even though the application works perfectly. This plan fixes those addresses.
1. What does this plan do?
It keeps a design change from breaking the automated tests. When the frontend team renames identifiers on the screen, the plan reads exactly what changed in the code and updates the tests that used them, without waiting for them to fail.
Its intent is to separate two kinds of failure that are easy to mix up: "the application has a bug" and "the test is out of date". This plan only deals with the second one, and it does so with the most reliable information there is: the change's own code. To do that:
- It finds which tests touch the part of the application that changed.
- In those tests, it fixes only the selectors whose replacement is clear in the change's code. Anything ambiguous it doesn't guess: it leaves it for a person to review.
- It runs the tests it fixed, to confirm they work.
Since it works by reading the change's code, it doesn't need the application to be deployed to make the fixes: it's meant to run before the deploy.
2. When does it run?
| Situation | Business example |
|---|---|
| A frontend change reaches the pipeline | The TechStore team pushes a change to the cart: the "Checkout" button gets a new identifier |
| You want to keep tests from failing because of a rename | TechStore's QA lead doesn't want to see the suite go red every time the design team reorganizes a screen |
It runs on every frontend change, big or small: there's no earlier step deciding whether it's worth it. The lippia-neo-ci-bootstrap CI templates include a ui-selector-repair job that runs it on every push to develop that modifies files in src/components/ or src/pages/.
Important
- The plan only fixes selectors. It doesn't generate new tests for new features or explore the application: that's what the Frontend UI Change PR playbook is for.
- It only fixes what the change's code makes clear: if an identifier was deleted with no replacement, or there's more than one possible candidate, it leaves it untouched and reports it.
- If the project doesn't have a
qa-stack.yamlyet, you first need to run the project bootstrap playbook.
3. How does it differ from the playbook?
The plan is derived from the Frontend UI Change PR playbook, but it keeps only the part about repairing the suite:
| Frontend UI Change PR playbook | This plan | |
|---|---|---|
| Who runs it | The AI assistant, talking with you | The engine, on its own |
| Where it learns what changed | From the ticket and from walking through the deployed screen | From the change's code (the diff) |
| Needs the app deployed | Yes, to walk through it | Not to make the fixes. It only uses it when running the fixed tests |
| Tests for new things | Generates them, if you decide there's something new | Doesn't generate new tests |
| Pauses | Two: deciding whether there's something new and reviewing whether it repaired too much | None: anything doubtful is left untouched and reported |
| Report | Final report with evidence | Doesn't write a report; the result is in result.json and, in CI, in an MR with the changes |
4. Guiding example: "TechStore" renames the checkout button
We continue with the Frontend UI Change PR playbook's case. The TechStore team reorganized the cart: they moved the "Checkout" button and, along the way, changed its identifier from data-test="checkout-btn" to data-test="finish-purchase-btn". There are 3 automated checkout tests that use the old identifier.
4.1 What the plan needs to receive
| Data | What it is | TechStore example |
|---|---|---|
commit_id |
The identifier of the change's commit. It's used to name the folder where the results end up | a1b2c3d |
code_diff_path |
The path to the file with the change's diff (the output of git diff) |
diffs/a1b2c3d.diff |
In CI, the job generates the diff and builds this data on its own.
4.2 How it's triggered
In CI, the job computes the push's diff, runs the plan and, if something was fixed, opens an MR with the fixed tests:
rules:
- if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "develop"'
changes:
- src/components/**/*
- src/pages/**/*
To try it from your computer:
git diff develop > diffs/a1b2c3d.diff
neo run plans/ui-selector-repair.yaml --workspace . --out result.json \
--data '{"commit_id":"a1b2c3d","code_diff_path":"diffs/a1b2c3d.diff"}'
4.3 Step by step
Step 1 of 3 — Find the tests affected by the change
Skill used: lippia-neo-find-ui-tests-by-code-diff
What happens: it reads the diff and pulls out what changed: files, added or removed selectors, routes, classes, and visible text. Then it searches the tests folder for the ones that use any of that. It only reads: it doesn't run or modify anything.
With TechStore: it finds the 3 checkout tests that use checkout-btn.
A test matching doesn't mean it's broken
A match means "this test touches the part that changed". If no test matches, the following steps are skipped: there's nothing to fix.
Step 2 of 3 — Fix the selectors
Skill used: lippia-neo-repair-selectors-by-diff
Only runs if Step 1 found tests
What happens: in the tests Step 1 found, it replaces only the selectors whose change is clear in the diff ("it used to be this, now it's this"). It doesn't open the browser or run anything: it's a fix on the files.
Anything that isn't clear (an identifier deleted with no replacement, or with more than one candidate) isn't guessed: it goes into a list of pending items for a person to review.
With TechStore: it replaces checkout-btn with finish-purchase-btn in the 3 tests.
Step 3 of 3 — Run the fixed tests
Skill used: lippia-neo-run-and-heal
Only runs if Step 2 fixed some test
What happens: it runs only the tests fixed in Step 2, never the whole suite, to confirm they work. If one fails for another minor reason, it tries to repair it and runs it again, up to 2 times.
With TechStore: it runs the 3 checkout tests with the new selector and confirms they pass.
4.4 How does it end?
| How it ended | What it means | What to do |
|---|---|---|
| ✅ Tests were fixed and they pass | The frontend change no longer breaks the suite | In CI, review and approve the MR with the fixed tests |
| ⚠️ Some selectors are pending | The diff didn't make the replacement clear for some selectors | Check the pending list in result.json and fix them by hand |
| ➖ No test matched the change | The change doesn't touch anything the tests use | Nothing: the plan ends with no changes |
| ❌ A fixed test keeps failing | The problem wasn't just the selector | Check whether the screen changed its behavior, or whether it's a bug |
In all these cases the plan ends in success: the details of what was found, what was fixed, and what's pending are in result.json.
5. Common problems (and what to do)
| What you see | Why it can happen | What to do |
|---|---|---|
| The plan found no tests, but some tests are failing | The tests use selectors that don't appear in the diff (for example, a text that changed in another file) | Review those tests by hand, or with the Frontend UI Change PR playbook |
| Many selectors were left pending | The change deleted elements or replaced them with several | That's expected: the plan doesn't guess. Fix them by hand |
| Step 3 fails to open the app | The application isn't reachable at the URL in qa-stack.yaml |
Confirm the URL; Step 2's fixes are still made |
| The diff is empty | The job couldn't compute the push's diff | Check that the job has the full git history (GIT_DEPTH: "0") |
6. Visual summary of the flow
flowchart TD
A([🎨 Frontend change<br/>e.g.: TechStore renames the checkout button]) --> B["<b>Step 1</b> — Find the tests<br/>affected by the diff"]
B --> C{Did it find<br/>tests?}
C -->|No| X([➖ Nothing to fix])
C -->|Yes| D["<b>Step 2</b> — Fix the selectors<br/>with a clear replacement"]
D --> E{Did it fix<br/>any?}
E -->|No| P([⚠️ Only pending items to review])
E -->|Yes| F["<b>Step 3</b> — Run the fixed tests"]
F --> G([✅ Tests up to date + pending list])
Original plan
The plan file is plans/ui-selector-repair.yaml, in the engine repository (crowdar-engine-mvp). The format reference is in Anatomy of a plan.