Skip to content

Smoke Test Post-Deploy

This guide is meant to help you understand what the AI assistant does when it runs this playbook. The idea is that you can read it, understand what happens at each step, and know what information you need to give the assistant so the check turns out well — no programming knowledge required.

In short

  • What it's for: quickly confirming, after a deploy, that nothing critical broke.
  • When to use it: right after a deploy, a hotfix, or a rollback finishes, or as an automatic gate in the pipeline.
  • What you need: a set of tests that already exists, the environment URL, and the project's qa-stack.yaml.
  • What you get: a dated report that says what worked and what didn't after the deploy.
  • Steps: 2 — run the test set just once, without repairing it, and generate the report.
What is a playbook? (if this is your first time using one)

Think of it as a cooking recipe for the AI assistant. You tell it "I want to do this" (for example, "I want to confirm that the deployment that just happened didn't break anything") and the playbook is the recipe the assistant follows step by step, in order, without skipping anything, so the result is always the same no matter who asks for it.

Each step of the recipe uses a specific tool (in technical jargon they're called "skills"), a bit like a kitchen utensil: one is for running the tests, another for writing the final report. You don't need to know how each utensil works inside — you only need to know which ingredient goes into each step (for example, which environment the deployment happened in) so the final result is useful.

A deploy (or deployment) is when a new version of the application is released.

A playbook = an ordered guide of steps. A skill = the specific tool that carries out each step. You interact with the playbook — the assistant takes care of using the right tool at each moment.


1. What does this playbook do?

It's the playbook you use as soon as a new version of the application has finished being released (a deploy), to quickly confirm that nothing critical broke — before accepting that version.

Think of it as the quick check you do on a car that just came out of the repair shop: you don't take the whole engine apart again, but you do make sure it starts, the lights turn on, and the brakes respond before handing the keys back to the owner. If those basic things work, you trust that the shop did a good job. If any of them doesn't work, you don't keep driving — you go back and check what happened.

This playbook does exactly that, but with the application: it takes the set of tests that already exists from before (it doesn't create new tests) and runs it just once, exactly as it is, without stopping to fix anything that fails along the way. At the end, it leaves a document with the date and time that clearly says: "after this deploy, this is what worked and this is what didn't".

What this playbook does NOT do

It doesn't check the application in depth or test new things, and it doesn't generate or repair tests. It also doesn't fix anything it finds broken — it only detects it and documents it. It's a "vital signs" check, not a full medical checkup. If something fails and needs to be repaired, that's a different job (we cover it in What next?).

The only exception: before running, the assistant automatically fills in some pending data in the tests (values marked as TODO or hand-written credentials). It's not a repair of failures, but it may touch a test file.


2. When should I use this playbook?

Use it when you're facing any of these situations — they're all typical starting points for this playbook:

Situation (starting point) Business example
A deploy to production or to a test environment has just finished The TechStore team has just released a new version of the store to production, and before telling everyone "it's done" they want to confirm that the cart and payment still work
It's an automatic check inside the pipeline, before letting a version move from one environment to another Every time TechStore uploads code to the test environment, the system runs this check on its own, without anyone asking for it, and if something fails it doesn't let that version move on to production
A hotfix or a rollback has just been applied TechStore detected that the "Checkout" button wasn't responding, they released an urgent fix, and now you need to quickly confirm that the fix didn't break something else (for example, the login)
Someone asks for concrete evidence that the environment is responding properly TechStore's tech lead wants a dated document confirming that, after this morning's deploy, the system is still healthy

Important

  • This playbook needs the set of tests to already exist from before. It doesn't create new tests. If TechStore doesn't have tests set up for its shopping cart yet, you first need to run the project setup playbook or the full QA flow.
  • This playbook is not a full regression. It's fast and lightweight on purpose, designed to confirm what's critical after a deploy — not to recheck the whole application end to end.

3. Walkthrough example: "TechStore", an online store with a shopping cart

We'll follow the same example business case as in other guides: TechStore, an online electronics store with a shopping cart (add products, view the cart, pay). It's the kind of application anyone recognizes from having shopped online at some point.

Let's assume this scenario: it's 10 in the morning and the TechStore development team has just finished deploying a new version to production, with several improvements to the payment screen. Before announcing that the deploy went well, someone on the team says: "wait, let's first confirm that the cart and payment still work like before".

That's exactly the scenario for this playbook: there's already a set of tests built earlier for TechStore (it was generated at some previous point, with another playbook), and what's needed now is to run it once, quickly, and keep a record of the result — not redo the testing work from scratch.

Throughout the following steps, we'll see what you would tell the assistant at each moment, using this TechStore deploy as an example.

3.1 Before you start: what you need to have on hand

You don't need to know how to program, but you do need to have this information on hand:

Information TechStore example
That the set of tests already exists from previous work TechStore already has automated tests for the cart and payment, generated when the project was set up (they live in the project's tests folder, paths.tests_dir in qa-stack.yaml)
The URL of the environment where the deploy was done, and that it's accessible https://techstore.com (production) or https://techstore-staging.com, depending on where the deploy was done
A name to identify this check Something simple that says which deploy it's about: staging, prod, or hotfix-login if it was a specific fix
The project's qa-stack.yaml file already set up If TechStore doesn't have it yet, you first need to run the project setup playbook

Before you start, confirm that the application responds

If the TechStore URL doesn't load or throws an error, there's no point running the check yet — you need to resolve that first, because otherwise the result won't say anything real about the deploy.

Original playbook

The general playbook is in the NEO-releases repository.

3.2 How to start the conversation

Before asking for anything specific, tell the assistant which deploy this is about, so it keeps it in mind throughout the whole process. For example:

💬 Prompt for the assistant
A TechStore deploy to production (https://techstore.com) has just been completed.
I want to run the post-deploy smoke test to confirm that the cart and payment
still work properly, without repairing the tests that fail.
Call this check "prod" (that's how it will appear in the report name).

The assistant will use this description throughout all the following steps, so you don't have to repeat it every time.


3.3 Step by step

Step 1 of 2 — Run the existing set of tests, just once, without repairing it

Skill used: lippia-neo-run-and-heal

What happens: the assistant takes TechStore's tests that already exist (the ones for the cart, the login, the payment) and runs them all, once, against the environment where the deploy was done. Unlike other playbooks, here the assistant doesn't try to fix anything that fails along the way — if a test doesn't pass, it records it as failed and keeps going. The idea is for the result to reflect exactly the real state of the deploy, without anyone "touching" it in between.

What the assistant gathers during this step:

Information What it's for
How many tests passed and how many failed The key number to know whether the deploy is healthy
How long it took to run everything A time reference for future checks
The details of each test that failed, if any A starting point to investigate if needed

At the end of this step, one of these situations can happen:

Result What it means
✅ Everything passed The deploy didn't break anything critical — a good sign
❌ Something failed It may be a real regression caused by the deploy — you still move on to Step 2, which will document it. The assistant warning that "some tests were left unrepaired" is expected here: this playbook doesn't repair
⚠️ The environment didn't respond (for example, the page doesn't load, or throws a connection error) This isn't a smoke test result — it means the deploy environment isn't available. It's not the same as "a test failed": you need to resolve access to the environment first, and only then run this step again

Why this distinction matters

If the TechStore cart "fails" because the page didn't even load, that doesn't say anything about whether the deploy broke the cart — it says the environment is down. Confusing these two things can lead the team to chase a bug that doesn't actually exist.


Step 2 of 2 — Generate the check report

Skill used: lippia-neo-write-test-report

What happens: the assistant gathers everything it found in Step 1 and writes a tidy document, with date and time, which is saved as a historical record of that specific deploy — without deleting or overwriting reports from previous deploys.

The only thing it might ask you in this step is the name you want to give the check (for example, prod, staging, or hotfix-login), so it can name the report file, and which environment it ran in. It already has everything else, because it comes from Step 1.

What you have at the end of this step:

Report condition What to expect
Report file created, with date and time In the project's reports folder (by default test-results/reports/), for example: smoke-prod-test-report-2026-07-17-1015.md
Total count of tests, passed and failed A clear number, with no ambiguity
List of failures, if any, with the reason for each A starting point to investigate or reprioritize
Overall status of the deploy "Passed" or "Failed", clearly visible at the beginning of the document
Why does the report say the coverage is \"not assessable\"?

A smoke test doesn't start from a user story with acceptance criteria, so the report has nothing to measure coverage against. This is expected and doesn't indicate any problem.


3.4 What do I have after completing the two steps?

What you get What it's for
A dated report of the deploy's status Concrete evidence that the deploy was checked, not just "on someone's word"
The count of passed and failed tests A simple number to decide whether the deploy stays or is rolled back
The details of what failed, if anything failed A starting point to decide the next step (investigate, repair, escalate)
The certainty that no failed test was repaired in the process The check is faithful to the real state of the deploy, with no automatic repairs to disguise it

4. What next? — What to do based on the result

Smoke test result What to do
✅ Everything passed The deploy is confirmed as healthy. The report is saved as evidence and you carry on as usual (for example, announcing that the deploy is OK)
❌ Something failed, and it looks like a real regression Investigate it with the bug reproduction playbook, or cover it with an automated test using the Bug Fix Cycle playbook — this playbook, by design, doesn't fix anything on its own
⚠️ The environment didn't respond (it's not a real failure, it's an infrastructure block) Resolve access to the environment (find out why the URL isn't responding) and run Step 1 again from scratch
🩹 It was a hotfix or a rollback Besides this check, it's worth letting the team know that the fix was confirmed (or not) with dated evidence, so the incident can be closed with peace of mind

This playbook, on its own, doesn't repair anything — its job ends with confirming and documenting the state of the deploy. If something went wrong, the work of fixing it or covering it with more tests continues in another playbook, using the report generated here as a basis.


5. Common problems (and what to do)

What you see Why it might happen What to do
The assistant says it can't find the tests to run No set of tests has been generated for TechStore yet, or they're outside the project's tests folder First run the project setup playbook or the full QA flow, so there's something to check
All the tests fail with a timeout or connection error The deploy environment isn't available (it's not a problem with the application itself) Check that the TechStore URL responds in a normal browser before running the check again
The tests fail at login The project's test credentials (.env.tests file) are empty or aren't the ones for this environment Ask someone on the technical team to fill in the real values — they're never written by hand inside the tests
The assistant says it started "repairing" tests that failed It was run without the right options and went into full repair mode Explicitly clarify that you want "smoke, no repair" mode before running it again

6. Visual summary of the flow

flowchart TD
    A([🚀 Deploy just completed<br/>e.g.: new payment version in TechStore]) --> B
    B["<b>Step 1</b> — Run the existing test set<br/>just once, without repairing it"] --> C
    C["<b>Step 2</b> — Generate the dated report"] --> D{Result?}
    D -->|Everything passed| E([✅ Deploy confirmed OK])
    D -->|Something failed| F([❌ Failures documented,<br/>ready to investigate])