Skip to content

How to run it

The engine has four commands. The first two are the ones you will use almost all the time:

Command What for
neo run <plan.yaml> Model 4: run a plan you have already chosen
neo run --input "<texto>" Model 2: let the engine choose the plan from a free-text request
neo event In CI: turn the MR/PR event into the plan's input
neo report In CI: post the execution result as a comment on the MR/PR
neo matrix Compare a skill across several AI models (skill quality control)

1. neo run — Model 4 (fixed plan)

neo run plans/feature-new-end-to-end.yaml \
  --input-file input.json \
  --workspace ../mi-proyecto \
  --out result.json

The input is a JSON file with the keys the plan asks for in required_keys. For feature-new-end-to-end:

{ "description": "user-stories/US-010.md" }

You can also pass it inline with --data:

neo run plans/ui-selector-repair.yaml --workspace . \
  --data '{"commit_id":"commit-1","code_diff_path":"diffs/PR-1.diff"}'

PowerShell

If you generate input.json with Out-File -Encoding utf8, PowerShell adds an invisible mark (BOM) at the beginning. The engine tolerates it: you don't need to convert the file.


2. neo run --input — Model 2 (the engine chooses the plan)

neo run --input "retest the payment flow, something changed there" \
  --plans-dir plans \
  --workspace ../mi-proyecto

The engine gathers all the plans in --plans-dir, asks the classifier skill to choose one, and runs it. If the confidence is below 0.7, it runs nothing: it escalates to a human. The details are in Orchestration models in the engine.

To pass a file as the request:

neo run --input "$(cat pedido.txt)" --plans-dir plans --workspace .

3. neo run options

Option Model What it does Default
<plan.yaml> M4 The plan to run —
--input-file <json> M4 JSON file with the plan's input —
--data '<json>' M4 The plan's input, inline —
--input "<texto>" M2 Free-text request to classify —
--plans-dir <dir> M2 Folder with the candidate plans plans
--overlay-dir <dir> M2 Folder with plans that replace the ones in --plans-dir with the same name —
--threshold <n> M2 Minimum confidence to run without a human 0.7
--workspace <dir> Both Folder of the test repository. Skills can only write there Current folder
--out <json> Both Saves everything each step produced to a JSON file —
--model <modelo> Both Overrides the plan's model (or the classifier's model in M2) The plan's
--stack <framework/ci> Both Overrides what qa-stack.yaml says, for example playwright/github or just lippia —
--db <archivo> Both Audit database audit.db

4. What it returns

4.1 On screen

When it finishes, the engine prints a summary line:

run feature-new-end-to-end-3f9a1c2e: success (cost $0.4182, steps 8, cached False)

While it runs, it also prints each step with its input and its status ([step 2] lippia-neo-create-test-plan status=success ...).

4.2 Possible statuses

Status What happened
success All the steps ran (or were skipped because of their when)
stopped A step with stop_unless stopped the plan on purpose. It is a normal stop, not an error
failed A step with guard: true returned something that doesn't match its schema
error Something broke: a mistyped reference, a skill that doesn't exist, a network error

4.3 Exit codes

They matter in CI, because they decide whether the job turns green or red:

Code When
0 success or stopped
1 failed, error, or a configuration problem (invalid plan, nonexistent skill)
2 Model 2 only: confidence was low and it was escalated to a human. Nothing was run

4.4 Files

File What it contains
result.json (with --out) Everything each step wrote, under its output_to. This is what neo report reads
audit.db The full audit in SQLite: executions, steps, model used, tokens, cost, duration, and every piece of context data
The skills' artifacts Test plans, tests, reports, and screenshots, in the folders defined by qa-stack.yaml inside the --workspace

To review the audit, any SQLite viewer will do. The tables are runs, steps, and context.


5. neo event and neo report — in CI

These two commands are the boundary with the CI platform. The plan never knows whether it is running on GitLab or on GitHub:

neo event       →   neo run       →   neo report
(reads the MR/PR)   (agnostic)        (comments the result)

5.1 neo event

It reads the variables injected by the platform and writes a JSON file that always has the same keys:

neo event --provider gitlab --out input.json
{
  "event": "merge_request",
  "project": "grupo/mi-proyecto",
  "commit_sha": "a1b2c3...",
  "branch": "feature/carrito",
  "pr_id": "42",
  "title": "Agregar producto al carrito",
  "description": "user-stories/US-010.md",
  "source_branch": "feature/carrito",
  "target_branch": "develop",
  "labels": ["feature"]
}

The plan reads them as ${input.description}, ${input.labels}, etc.

5.2 neo report

It takes result.json and posts the summary key as a comment on the MR/PR:

neo report --provider gitlab --result result.json
Option What it does Default
--provider gitlab or github gitlab
--result The JSON file left by neo run --out —
--summary-key Which key of the result to post summary

On GitLab it needs CROWDAR_CI_TOKEN with api permission: the job's default token cannot comment on MRs.

Report even if it fails

In the pipeline, it's a good idea to capture the exit code of neo run, run neo report anyway, and only then exit with that code. That way the MR gets the comment whether it passes or fails. The lippia-neo-ci-bootstrap templates already do it this way.


6. neo matrix — quality control for a skill

It runs a skill against a reference ("golden") case on several models at once, validates each output, and shows a grid:

neo matrix lippia-neo-create-test-plan --golden golden \
  --models anthropic/claude-haiku-4-5-20251001,openai/gpt-4o-mini
  PASS  anthropic/claude-haiku-4-5-20251001  cost=$0.0123  ...
  FAIL  openai/gpt-4o-mini  cost=$0.0041  missing required key 'scenarios'
matrix lippia-neo-create-test-plan: DRIFT DETECTED (1/2 cells, $0.0164)

It exits with code 1 if any cell fails. It helps you detect when a change to a skill broke its behavior on some provider. The reference cases live in the golden/ folder of the engine repo, one subfolder per skill.


7. Common problems (and what to do)

What you see Why it happens What to do
cached True and the plan did nothing The same plan with exactly the same input already finished successfully before (idempotency) Change the input, use another database with --db otra.db, or delete audit.db
UnresolvedRefError: ... required input(s) reference keys nothing wrote A ${...} in the plan points to data that nobody wrote, and the skill absolutely needs it Check that the input has that key (--data / --input-file) or fix the reference in the plan
cannot read skill file: .../SKILL.md The engine can't find the skill in the skills folder Check CROWDAR_SKILLS_DIR, or the plan's stack: field, which takes priority
ignoring unknown tools: ... in the log The step asks for a tool that isn't in qa-stack.yaml (mcp_servers) and isn't provided by the adapter either Add the server to mcp_servers with that exact name, or fix the name in the plan
Fails when fetching a Jira or Azure DevOps ticket The MCP environment variables are missing See Requirements and installation
escalated to human (code 2) In Model 2, confidence was below the threshold, or the request didn't include the data the plan needs Rephrase the request with more detail, or run the plan directly with Model 4
Forbidden control character detected in headers An API key with odd characters, on an old version of the engine Update the engine; current versions clean up the key automatically
In CI, neo report fails with 401/403 CROWDAR_CI_TOKEN doesn't have api permission Generate a project/group access token with the api scope