How to run it
The engine has four commands. The first two are the ones you will use almost all the time:
| Command | What for |
|---|---|
neo run <plan.yaml> |
Model 4: run a plan you have already chosen |
neo run --input "<texto>" |
Model 2: let the engine choose the plan from a free-text request |
neo event |
In CI: turn the MR/PR event into the plan's input |
neo report |
In CI: post the execution result as a comment on the MR/PR |
neo matrix |
Compare a skill across several AI models (skill quality control) |
1. neo run — Model 4 (fixed plan)
neo run plans/feature-new-end-to-end.yaml \
--input-file input.json \
--workspace ../mi-proyecto \
--out result.json
The input is a JSON file with the keys the plan asks for in required_keys. For feature-new-end-to-end:
{ "description": "user-stories/US-010.md" }
You can also pass it inline with --data:
neo run plans/ui-selector-repair.yaml --workspace . \
--data '{"commit_id":"commit-1","code_diff_path":"diffs/PR-1.diff"}'
PowerShell
If you generate input.json with Out-File -Encoding utf8, PowerShell adds an invisible mark (BOM) at the beginning. The engine tolerates it: you don't need to convert the file.
2. neo run --input — Model 2 (the engine chooses the plan)
neo run --input "retest the payment flow, something changed there" \
--plans-dir plans \
--workspace ../mi-proyecto
The engine gathers all the plans in --plans-dir, asks the classifier skill to choose one, and runs it. If the confidence is below 0.7, it runs nothing: it escalates to a human. The details are in Orchestration models in the engine.
To pass a file as the request:
neo run --input "$(cat pedido.txt)" --plans-dir plans --workspace .
3. neo run options
| Option | Model | What it does | Default |
|---|---|---|---|
<plan.yaml> |
M4 | The plan to run | — |
--input-file <json> |
M4 | JSON file with the plan's input | — |
--data '<json>' |
M4 | The plan's input, inline | — |
--input "<texto>" |
M2 | Free-text request to classify | — |
--plans-dir <dir> |
M2 | Folder with the candidate plans | plans |
--overlay-dir <dir> |
M2 | Folder with plans that replace the ones in --plans-dir with the same name |
— |
--threshold <n> |
M2 | Minimum confidence to run without a human | 0.7 |
--workspace <dir> |
Both | Folder of the test repository. Skills can only write there | Current folder |
--out <json> |
Both | Saves everything each step produced to a JSON file | — |
--model <modelo> |
Both | Overrides the plan's model (or the classifier's model in M2) | The plan's |
--stack <framework/ci> |
Both | Overrides what qa-stack.yaml says, for example playwright/github or just lippia |
— |
--db <archivo> |
Both | Audit database | audit.db |
4. What it returns
4.1 On screen
When it finishes, the engine prints a summary line:
run feature-new-end-to-end-3f9a1c2e: success (cost $0.4182, steps 8, cached False)
While it runs, it also prints each step with its input and its status ([step 2] lippia-neo-create-test-plan status=success ...).
4.2 Possible statuses
| Status | What happened |
|---|---|
success |
All the steps ran (or were skipped because of their when) |
stopped |
A step with stop_unless stopped the plan on purpose. It is a normal stop, not an error |
failed |
A step with guard: true returned something that doesn't match its schema |
error |
Something broke: a mistyped reference, a skill that doesn't exist, a network error |
4.3 Exit codes
They matter in CI, because they decide whether the job turns green or red:
| Code | When |
|---|---|
0 |
success or stopped |
1 |
failed, error, or a configuration problem (invalid plan, nonexistent skill) |
2 |
Model 2 only: confidence was low and it was escalated to a human. Nothing was run |
4.4 Files
| File | What it contains |
|---|---|
result.json (with --out) |
Everything each step wrote, under its output_to. This is what neo report reads |
audit.db |
The full audit in SQLite: executions, steps, model used, tokens, cost, duration, and every piece of context data |
| The skills' artifacts | Test plans, tests, reports, and screenshots, in the folders defined by qa-stack.yaml inside the --workspace |
To review the audit, any SQLite viewer will do. The tables are runs, steps, and context.
5. neo event and neo report — in CI
These two commands are the boundary with the CI platform. The plan never knows whether it is running on GitLab or on GitHub:
neo event → neo run → neo report
(reads the MR/PR) (agnostic) (comments the result)
5.1 neo event
It reads the variables injected by the platform and writes a JSON file that always has the same keys:
neo event --provider gitlab --out input.json
{
"event": "merge_request",
"project": "grupo/mi-proyecto",
"commit_sha": "a1b2c3...",
"branch": "feature/carrito",
"pr_id": "42",
"title": "Agregar producto al carrito",
"description": "user-stories/US-010.md",
"source_branch": "feature/carrito",
"target_branch": "develop",
"labels": ["feature"]
}
The plan reads them as ${input.description}, ${input.labels}, etc.
5.2 neo report
It takes result.json and posts the summary key as a comment on the MR/PR:
neo report --provider gitlab --result result.json
| Option | What it does | Default |
|---|---|---|
--provider |
gitlab or github |
gitlab |
--result |
The JSON file left by neo run --out |
— |
--summary-key |
Which key of the result to post | summary |
On GitLab it needs CROWDAR_CI_TOKEN with api permission: the job's default token cannot comment on MRs.
Report even if it fails
In the pipeline, it's a good idea to capture the exit code of neo run, run neo report anyway, and only then exit with that code. That way the MR gets the comment whether it passes or fails. The lippia-neo-ci-bootstrap templates already do it this way.
6. neo matrix — quality control for a skill
It runs a skill against a reference ("golden") case on several models at once, validates each output, and shows a grid:
neo matrix lippia-neo-create-test-plan --golden golden \
--models anthropic/claude-haiku-4-5-20251001,openai/gpt-4o-mini
PASS anthropic/claude-haiku-4-5-20251001 cost=$0.0123 ...
FAIL openai/gpt-4o-mini cost=$0.0041 missing required key 'scenarios'
matrix lippia-neo-create-test-plan: DRIFT DETECTED (1/2 cells, $0.0164)
It exits with code 1 if any cell fails. It helps you detect when a change to a skill broke its behavior on some provider. The reference cases live in the golden/ folder of the engine repo, one subfolder per skill.
7. Common problems (and what to do)
| What you see | Why it happens | What to do |
|---|---|---|
cached True and the plan did nothing |
The same plan with exactly the same input already finished successfully before (idempotency) | Change the input, use another database with --db otra.db, or delete audit.db |
UnresolvedRefError: ... required input(s) reference keys nothing wrote |
A ${...} in the plan points to data that nobody wrote, and the skill absolutely needs it |
Check that the input has that key (--data / --input-file) or fix the reference in the plan |
cannot read skill file: .../SKILL.md |
The engine can't find the skill in the skills folder | Check CROWDAR_SKILLS_DIR, or the plan's stack: field, which takes priority |
ignoring unknown tools: ... in the log |
The step asks for a tool that isn't in qa-stack.yaml (mcp_servers) and isn't provided by the adapter either |
Add the server to mcp_servers with that exact name, or fix the name in the plan |
| Fails when fetching a Jira or Azure DevOps ticket | The MCP environment variables are missing | See Requirements and installation |
escalated to human (code 2) |
In Model 2, confidence was below the threshold, or the request didn't include the data the plan needs | Rephrase the request with more detail, or run the plan directly with Model 4 |
Forbidden control character detected in headers |
An API key with odd characters, on an old version of the engine | Update the engine; current versions clean up the key automatically |
In CI, neo report fails with 401/403 |
CROWDAR_CI_TOKEN doesn't have api permission |
Generate a project/group access token with the api scope |