Skip to content

Exploratory Testing Portfolio

This guide is meant to help you understand what the AI assistant does when it runs this playbook. The idea is that you can read it, understand what happens at each step, and know what information you need to give the assistant so the exploration turns out well — without needing to know how to code.

In short

  • What it's for: exploring an entire application (or a large part of it) by splitting it into small, manageable exploratory sessions, and ending up with a report that brings together everything that was learned.
  • When to use it: when you're asked to "explore the whole app" or a broad scope, and you want to review what's going to be explored before spending the time of each session.
  • What you need: the project's qa-stack.yaml, access to the application (or enough information about it) and, if needed, the test credentials.
  • What you get: a map of the app's areas, a portfolio of charters, a report for each session with its evidence and a consolidated final report.
  • Steps: 4 — discover the areas, build the charters, run one session per charter and consolidate the report (with two human review pauses in between).
What is a playbook? (if this is your first time using one)

Think of it as a cooking recipe for the AI assistant. You tell it "I want to do this" (for example, "I want to explore the whole online store before the release") and the playbook is the recipe the assistant follows step by step, in order, without skipping anything, so the result is always the same no matter who asks for it.

Each step of the recipe uses a specific tool (in technical jargon they're called "skills"), something like a kitchen utensil: one is for mapping the application, another for setting up the sessions, another for exploring and another for writing the final report. You don't need to know how each utensil works inside — you only need to know which ingredient goes into each step (for example, the maximum number of sessions you want or which part of the app matters most to you) so the final result is useful.

A playbook = an ordered guide of steps. A skill = the specific tool that carries out each step. You interact with the playbook — the assistant takes care of using the right tool at each moment.


1. Basic concepts

Before you start: here are some concepts specific to exploratory testing that are worth understanding for this playbook. Open the one you need.

Exploratory testing

It's like walking around a city you don't know without a fixed itinerary: you have a goal ("I want to get to know the harbor district"), but you decide where to go next based on what you find. In exploratory testing there's no list of steps written in advance: whoever is testing learns how the application behaves while using it, and uses that to design the next test. It's the opposite of a "scripted" test case, where each step is already defined.

Area

It's a part of the application that can be explored separately: for example, the product catalog, the cart or the payment. The assistant doesn't use a fixed list of areas: it discovers them from what actually exists in the application and in the project documentation. Something like "the whole app" or "general" never counts as an area — it's too broad to explore well.

Charter

It's the "roadmap" of an exploratory session. It says which area will be explored, what you want to learn or discover (the mission), some starting points to investigate (the focus), why that area was chosen and how much time will be spent on it. It's not a list of steps: it's a goal with boundaries. Each charter has a sequential number: CHARTER-001, CHARTER-002, etc.

SBTM session and time-box

SBTM (Session-Based Test Management) is an organized way of doing exploratory testing: you explore in sessions, each one with a single charter and a fixed maximum time (the time-box). It's like a meeting with a start and end time: you know in advance how long it will last and what topic it covers. In this playbook the time-box for each session ranges from 15 to 90 minutes (60 by default).

Portfolio

It's the set of charters that will be run to cover the broad request. If you're asked to "explore the whole app", instead of one giant charter a portfolio of several small charters is built (by default, up to 5). The areas left out of the portfolio aren't hidden: they're listed with the reason they weren't chosen.

Finding

It's anything relevant that comes up during the session. Odd behavior isn't automatically a bug: the assistant classifies each finding as a possible defect (there's a clear basis to say it's wrong), an observation (useful information, without concluding it's a defect), a false positive (something was suspected, but ruled out), a change request (it looks more like a product decision than an error) or pending (information is missing to decide).

T/B/S distribution

It's how the session time was split: T is the time actually exploring, B the time investigating findings and S the setup time (logging in, preparing data, configuring). It helps you understand the session — for example, if almost everything went into setup, that tells you something about how hard it is to test that part. There's no "correct" percentage.

Checkpoint and resume (save point)

It's like the save point in a video game. During the session the assistant saves its progress every few minutes (what it did, what it found, how much time it has used). If the conversation is cut off before it finishes, the session stays "in progress" and can be resumed from the last save point, without starting from scratch or creating a new session.

Don't confuse it with this playbook's human review pauses: those are moments when a person reviews and approves before continuing.

Portfolio identifier (artifact_scope_id)

It's the name of the folder where everything from this exploration is saved (area map, charters, sessions and final report). If you don't give it one, the assistant generates it on its own from the date and time, for example EXP-20261002-101500, and uses it in every step so nothing ends up scattered around.

Persistent mutations

These are actions that leave a real change saved in the system: creating, deleting, confirming or modifying data (for example, confirming a purchase or deleting an account). By default this playbook doesn't perform them: if a line of exploration needs them, the assistant notes it as a limitation and goes another way, unless you authorize it and there's safe test data.


2. What does this playbook do?

It's the playbook you use when you're asked to explore an entire application or a large part of it, and you want to do it in an organized way: first see which areas exist, then choose which ones to explore, explore each one in its own session and, at the end, bring everything together in a single report.

Think of it as a team of inspectors who have to check an entire building. They don't start walking around aimlessly: first they go through the building quickly to draw a plan of the floors and sections, then they decide which sections to check thoroughly and how much time to give each one, they check each section separately noting everything they see, and at the end they hand in a general report saying what was checked, what was found and what was left pending.

Unlike the automatic version (see below), this playbook stops twice so a person can review and approve:

  1. After building the portfolio of charters (to confirm that the right things will be explored before spending the sessions' time).
  2. After running all the sessions (to confirm that the findings and the coverage are properly documented before the final report).

The playbook doesn't need a fixed risk model to decide what to explore. If you or the project provide useful context (business priorities, recent changes, existing test coverage, critical parts), the assistant uses it to choose and order the charters — but it doesn't turn it into a score or make up priorities nobody gave.

What this playbook does NOT do

It doesn't create automated tests or scripted test cases: it explores. It doesn't report bugs during the initial mapping of the application, and it doesn't treat odd behavior as a defect without a clear basis to say it's wrong. It doesn't perform actions that leave changes saved in the system without your authorization. And it doesn't guarantee that an area was "100% tested": running out of time in a session doesn't mean that part was explored exhaustively.

Do you want to run it all at once, without pauses?

There's an automatic version that does the same four steps in a row, without stopping for human review: the meta-skill lippia-neo-meta-run-exploratory-testing. It's a good fit when you don't need to review the portfolio before running it. If you want to control what gets explored and how the findings end up, use this playbook.


3. When should I use this playbook?

Use it when you're facing any of these situations:

Situation Business example
You're asked to explore the whole application or a broad scope, without a specific goal Before a big release, TechStore's QA lead asks for "a general exploration of the whole store"
A large scope needs to be split into manageable sessions Exploring all of TechStore in a single session would be unmanageable; it's better to split it into catalog, cart, payment, etc.
You want evidence organized by area and also a general summary The TechStore team wants to see what was found in the cart and in the payment separately, and also a single report for the rest of the team
You want to review the charters before spending the sessions' time Before spending an hour on each area, the QA person wants to confirm that the right areas were chosen
Actions that leave saved changes, or environment restrictions, need to be controlled In TechStore, confirming a purchase creates a real order; nobody wants that to happen without approval and without safe test data

Important

  • This playbook needs the project to already have its qa-stack.yaml file. If TechStore doesn't have it yet, you first need to run the project setup playbook.
  • If you already know exactly what you want to verify (for example, the acceptance criteria of a story), what you need is a test plan, not a broad exploration: see the new feature end to end playbook.

4. Guide example: "TechStore", an online store with a shopping cart

We'll continue with the same example business case as in other guides: TechStore, an online electronics store with a shopping cart (search for products, add them to the cart, pay).

Let's assume this scenario: an important release is coming and the QA lead asks for "a general exploration of the whole store before going live". There's no specific user story or list of cases: you need to understand what parts the store has, choose which ones to explore with the time available and leave evidence of what was found. Also, the latest version brings changes to the payment, so it's worth giving it priority.

Throughout the following steps we'll see what you'd tell the assistant at each moment, using this TechStore exploration as the example.

4.1 Before you start: what you need to have on hand

You don't need to know how to code, but you do need to have this information on hand:

Information TechStore example
The project's qa-stack.yaml file already set up That's where the assistant finds the folder where it saves the exploration (by default exploration/) and, if configured, the app URL
Access to the application or enough information about it https://techstore-staging.com. If there's no URL, the assistant can discover the areas from the documentation, but to run the sessions it needs to be able to use the app
Test credentials, if the app requires login They go in the project's credentials file (.env.tests); they're never written in the message
The maximum number of charters you want and how much time per session Maximum 4 charters, 45 minutes each (if you don't say anything: 5 charters of 60 minutes)
Which parts matter most to you (optional) "Give priority to the payment, which changed in the latest version, and to the main purchase flow"
Whether actions that leave saved changes are allowed In TechStore: don't confirm purchases without approval (this is the default behavior)
Existing user stories and test plans (optional) If the project has them, they help the assistant discover the areas better

Default values

If you don't tell it anything, the assistant uses: up to 5 charters, 60 minutes per session, report in Spanish, no actions that leave saved changes, and up to 8 resumes per session if the conversation is cut off. The portfolio identifier isn't mandatory: if you don't give one, it's generated automatically.

Original playbook

The general playbook can be found in the NEO-releases repository.

4.2 How to start the conversation

Before asking for anything specific, tell the assistant what you want to explore and with what limits, so it keeps it in mind throughout the whole process. For example:

💬 Prompt for the assistant
I want to do a portfolio exploration of the whole TechStore store
(https://techstore-staging.com).
Maximum 4 charters, 45 minutes per charter, report in English.
Give priority to the payment, which changed in the latest version, and to the main purchase flow.
Don't perform actions that leave saved changes (like confirming a purchase) without my approval.

The assistant will use this description throughout all the following steps, so you don't have to repeat it every time.


4.3 Step by step

Note

This is an example: you need to follow the original playbook.

Step 1 of 4 — Discover the application's areas

Skill used: lippia-neo-discover-exploratory-areas

What happens: the assistant builds a "map" of the real parts of TechStore that can be explored. To do so, it combines whatever is available: the user stories and the project documentation, any test plans that already exist and, if it can get into the app, a quick, surface-level walk through the screens (just to get to know them, not to test them thoroughly). It merges repeated areas into one and notes, for each one, where the evidence that it exists came from.

TechStore example: the assistant identifies areas such as product catalog, search, cart, payment, login and customer account and order tracking. It doesn't use a predefined list: if TechStore didn't have search, that area wouldn't appear.

No bugs are reported in this step

If something odd shows up during the walk-through, it isn't reported as a bug: this step only builds the map. It also doesn't perform actions that leave saved changes just to get to know the application, and if the tool fails, that isn't taken as a problem with the store.

What you have at the end of this step: the area map in the exploration folder, for example exploration/EXP-20261002-101500/portfolio/discovery.md (one readable by people and another .json that the assistant uses in the next step).

At the end of this step, one of these situations can happen:

Result What it means
✅ At least one area was found The assistant can continue to Step 2
❌ There isn't enough information to identify real areas The assistant stops instead of making up areas. You need to give it more sources (documentation, stories) or access to the app
⚠️ The app doesn't respond or there's no way to interact with it It stops as well: the access needs to be resolved before trying again

Checkpoint — There are areas to continue with

The assistant does this check on its own: to continue, there has to be at least one area discovered with evidence. Even so, it's worth looking at the map and confirming that the areas correspond to real parts of TechStore.


Step 2 of 4 — Build the portfolio of charters

Skill used: lippia-neo-create-exploratory-charters

What happens: with the map from Step 1 (and the same portfolio identifier), the assistant builds one charter for each chosen area. If there are more areas than the maximum number of charters you asked for, it chooses which ones go in using only the available context: what you asked for, the priorities you gave, the recent changes you mentioned, the existing test coverage. Your instructions carry more weight than any criterion of the assistant's own. If there's no information to tell areas apart, it keeps the order of the map and tries to represent the whole scope well.

Each charter states: which area is explored, what you want to learn or discover (the mission, not a list of steps), where to start investigating, why that area was chosen and how much time the session has (between 15 and 90 minutes).

TechStore example: with a maximum of 4 charters and the priority on payment, the portfolio could look like this:

Charter Area Reason for selection
CHARTER-001 Payment You asked for it as a priority and it changed in the latest version
CHARTER-002 Cart It's part of the main purchase flow you asked to prioritize
CHARTER-003 Product catalog It's the entry point to the main purchase flow
CHARTER-004 Login and customer account It represents a part of the scope that isn't in the purchase flow

And the areas that were left out, each with its reason (for example: search and order tracking, because the maximum of 4 charters was reached and they weren't among the stated priorities).

What you have at the end of this step: the portfolio (exploration/EXP-20261002-101500/portfolio/portfolio.md) and one file per charter (exploration/EXP-20261002-101500/charters/CHARTER-001.md, etc.).


⏸ Human control pause 1 — Validate the portfolio

Mandatory pause — action required before continuing to Step 3

Before spending the sessions' time, someone has to review the portfolio and confirm:

  • Each charter has a specific area (nothing like "the whole app" or "general")
  • The mission says what you want to learn or discover, not a list of steps
  • The focus of each charter makes sense for that area and for what you asked for
  • The reason for selecting each charter is based on real information
  • The time for each session is between 15 and 90 minutes
  • The set of charters reasonably represents what you asked to explore
  • The areas that were left out are listed and their reason is understandable
  • It's clear which actions that leave saved changes are allowed (and which aren't) and what restrictions the environment has

If you find something odd, ask the assistant directly to correct it, for example:

💬 Prompt — Adjust the portfolio
Adjust the portfolio: CHARTER-003 for the catalog is too broad,
split it into "product listing" and "product detail".
Remove CHARTER-004 for login so the new charter fits.

Only once the portfolio is approved do you continue with Step 3.


Step 3 of 4 — Run one exploratory session per charter

Skill used: lippia-neo-run-exploratory-session

What happens: the assistant runs the charters one at a time, in order: each charter is a session, with its own time. In each session it really uses the application (in a browser, if it's a web app), and learns how it behaves while it designs and runs tests on the fly: it observes, asks itself a question, tries something, looks at what happened, writes it down and decides where to go next. The charter's focus is a starting point, not a mandatory checklist.

Meanwhile it keeps a session log (what it did and when), notes the findings with their evidence (for example, screenshots), calculates how the time was split (T/B/S) and saves its progress every few minutes.

TechStore example (CHARTER-001, payment): the assistant goes through the payment with different payment methods and amounts, tries what happens when going back from payment to the cart and notes that an error message in the card form appears in English. When a line of investigation requires confirming a real purchase, it doesn't do it (it isn't authorized): it notes it as a limitation and continues with other safe lines.

What the assistant collects in each session:

Information What it's for
Session log Reconstructing what was done and why each decision was made
Classified findings, with evidence Knowing what came up and with what degree of certainty (possible defect, observation, etc.)
Expected vs. observed result For possible defects, whenever there's a clear basis for what should happen
Time distribution (T/B/S) Understanding how much was really explored and how much went into setup or investigation
Coverage summary What was explored, what was left uncovered and what limited it, described in words (without "high/medium/low")
Opportunities and open questions New lines of exploration, with a suggested charter when appropriate

At the end of each session, one of these situations can happen:

Result What it means
✅ Completed The time was reached or the mission was reasonably accomplished. You continue with the next charter
⚠️ In progress (needs to be resumed) The conversation was cut off before the time ran out. The session was saved and is resumed from its save point (see below)
❌ Blocked It couldn't be run or continued (for example, there's no access to the app, credentials are missing or the charter depends on an unauthorized action). See below what to do depending on the type of block

If the session is left "in progress": ask the assistant to resume it from the report it left. For example:

💬 Prompt — Resume a session
Resume the CHARTER-001 session from its save point
(exploration/EXP-20261002-101500/sessions/SESSION-001/SESSION-001-report.md).

When it's resumed, the same session continues (a new one isn't created) and the time already used is respected. A maximum of 8 resumes per charter is recommended. If two resumes show no progress, it's best to stop and check what's going on with the tool, instead of continuing to try endlessly.

If the session is left "blocked":

Type of block What to do
General: there's no access to the app, no way to use it, or login fails for everything Stop the portfolio: no other session will be able to run until it's resolved
Specific to that area Record it and continue with the next charter, if it's safe
The charter needs an action that leaves saved changes and it isn't authorized Don't perform it: continue with safe lines or leave the charter marked. If you want, approve it with safe test data

A tool error isn't an app bug

If the browser freezes or the conversation is cut off, that isn't a TechStore problem. The assistant doesn't note it as an application finding — and in the review in the next step it's worth confirming that this was respected.

What you have at the end of this step: one folder per session, for example exploration/EXP-20261002-101500/sessions/SESSION-001/, with the session report (SESSION-001-report.md and its .json), the save point (checkpoint.json) and the evidence (evidence/ folder).


⏸ Human control pause 2 — Review the sessions

Mandatory pause — action required before continuing to Step 4

Before the final report, someone has to review the sessions and confirm:

  • Each charter has its session report
  • There are no sessions left "in progress" without an explicit decision (resume it or consider it cut off)
  • The important findings have enough evidence
  • The possible defects distinguish what was expected from what was observed, when there's a clear basis to know it
  • Tool errors don't appear as application defects
  • The blocks, the limitations and what couldn't be covered are stated explicitly
  • Coverage is described in words, without assuming that everything was tested
  • The exploration opportunities have a suggested charter when appropriate

If you want, at this point you can add a follow-up charter before the final report, for example:

💬 Prompt — Add a follow-up charter
Before consolidating, add a follow-up charter for the error message
in English in the card form that came up in CHARTER-001, and run it.

Only once the sessions have been reviewed do you continue with Step 4.


Step 4 of 4 — Consolidate the portfolio's final report

Skill used: lippia-neo-write-exploratory-testing-report

What happens: the assistant gathers the portfolio, the actual results of each session and the chosen language, and writes a single final report. It doesn't use the application again or repeat tests: it only consolidates what already happened. The report explains what was explored, what was learned and observed, what limitations there were and what new lines of exploration came up — without making up risks, severities or coverage that don't exist. If a finding is missing its severity, it's left as "unclassified" instead of being filled in by guesswork.

What the report includes:

Section What to expect
Executive summary and scope What you asked to explore and what was actually explored
Status per charter Which ones were completed, which ones were blocked or left unfinished
Findings by area Separating possible defects from observations; never as "confirmed defects"
Blocks and uncovered areas Includes the areas that weren't chosen in Step 2, with their reason
Open questions and opportunities With suggested follow-up charters
Evidence index Links to each session's reports

Overall portfolio status:

Status What it means
✅ Completed All the charters finished complete
⚠️ Partial At least one was completed and at least one other was blocked or left unfinished. What wasn't finished doesn't count as covered
❌ Blocked No charter could be completed

Where it's saved: in the exploration folder, with date and time in the name, for example exploration/EXP-20261002-101500/EXP-20261002-101500-portfolio-report-<date-and-time>.md (and its .json). Each time this step runs, a new report is generated — a previous one is never overwritten.

Checkpoint — Report in a single language

Check that the report is written entirely in the language you asked for. If it mixes languages, ask the assistant to repeat the consolidation with the language fixed.


4.4 What do I end up with after the four steps?

What you get What it's for
A map of the application's real areas, with evidence Knowing what parts the app has and where that information comes from
A reviewed and approved portfolio of charters Evidence of what was decided to explore, why, and what was left out
A report for each session, with log, findings and evidence The details of what happened in each area
A consolidated final report A single view to share with the team: what was explored, what was found and what was left pending
Suggested follow-up charters The starting point for the next exploration

Everything is kept together in a single folder: exploration/EXP-20261002-101500/.


5. What next? — What to do based on the result

Portfolio result What to do
✅ Completed, with no possible defects Share the final report with the team as evidence of the exploration
🐞 Possible defects came up Confirm them with the bug reproduction playbook before sending them to the development team — this playbook documents them, but doesn't confirm them or open tickets
⚠️ Partial: there are blocked or unfinished charters Resolve the block (access, credentials, safe test data) and explore those areas again in a new run
🧭 Suggested follow-up charters or unchosen areas remained Use them as the starting point for the next exploration, giving those areas priority
❌ Blocked: no charter was completed Check the access to the application and the available tools before running the playbook again

6. Common problems (and what to do)

What you see Why it can happen What to do
Step 1 ends with no areas There isn't enough information: sources are missing, the app doesn't respond or there's no way to use it Check the sources, the access and the available tools. The assistant won't make up areas
A charter is too broad The chosen area covers too much for a single session Split it before running, in human control pause 1
The reason for selecting a charter doesn't hold up There wasn't enough context to tell areas apart Give the assistant more context (priorities, recent changes) or accept a stable order that represents the whole scope well
The session is cut off early The conversation or the tool was interrupted Resume it from its save point, respecting the maximum number of resumes
A large part of the time goes into setup The app is hard to prepare for testing (data, access, environment) Record it as a limitation of the environment or of testability (how easy it is to test that part)
A large part of the time goes into investigating findings One finding took up the whole session Record the observation and consider a follow-up charter for that topic
The assistant stops a line because it needs an unauthorized action The action leaves saved changes (for example, confirming a purchase) and is disabled by default Don't perform it, or approve it using safe test data
The final report mixes languages The language wasn't fixed Repeat the consolidation specifying the language

7. Visual summary of the flow

flowchart TD
    A([🧭 Broad exploration request<br/>e.g.: the whole TechStore store]) --> B
    B["<b>Step 1</b> — Discover the areas<br/>of the application"] --> C{Was at least<br/>one area found?}
    C -->|No| Z([⛔ Stopped: missing sources<br/>or access to the app])
    C -->|Yes| D["<b>Step 2</b> — Build the portfolio<br/>of charters"]
    D --> E{Portfolio approved<br/>by a person?}
    E -->|No, adjust| D
    E -->|Yes| F["<b>Step 3</b> — Run the session<br/>for a charter"]
    F --> G{How did the<br/>session end?}
    G -->|In progress| H["Resume from the<br/>save point"]
    H --> F
    G -->|General block| Y([⛔ Portfolio stopped:<br/>resolve the access])
    G -->|Completed or<br/>specific block| I{Any charters<br/>left?}
    I -->|Yes| F
    I -->|No| J{Sessions reviewed<br/>by a person?}
    J -->|No, add follow-up| F
    J -->|Yes| K["<b>Step 4</b> — Consolidate the<br/>final report"]
    K --> L([✅ Portfolio report<br/>ready to share])