Skip to content

Performance testing

These five skills cover the full path of a load test: reading an API collection (Postman or Bruno), generating the load scripts in k6 or JMeter, running them profile by profile (smoke, load, stress, spike) and writing the final report with the numbers of each run. Each skill does a single thing and talks to the next one through the shape of its data: flows and authentication, scripts, run folders, report.

CI mode: no questions

None of these skills ask the person invoking them any questions. Each value is resolved in this order: explicit input, then the performance block of qa-stack.yaml (together with environment.base_url and paths.*) and, lastly, a documented default. Whatever is resolved by default is recorded. Only what is impossible to infer (for example, no collection and no flows) returns status: needs_input or blocked, with the detail in missing. The performance block is described in STACK.md of the skills repository.

Limits worth keeping in mind

  • The write test creates real data. lippia-neo-read-api-collection only runs reads; the exception is the write test (POST/PUT/PATCH, and DELETE only with include_mutations: all), which repeats each write with its body to detect single-use data. It only runs if include_mutations is not none, mutation_probe_iterations is greater than 0 and the probe is enabled; against a non-local host it also requires performance.confirm_remote: true.
  • Nothing gets installed. The runner does not install k6, JMeter or Java: if it cannot find them, it returns blocked (k6) or refused (JMeter) and runs nothing.
  • No secrets in files. No skill writes credentials, tokens or keys into scripts, reports, logs or warnings: they are masked (********) and values arrive through the process environment or through data files that the skills never read or copy.
  • JMeter does not measure iterations. With JMeter, iterations and everything derived from them appear as n/a in the report (not as 0 or as an estimate).

lippia-neo-read-api-collection

When to use it

When there is a Postman or Bruno collection and you need to understand its flows, how it authenticates (bearer, session cookie, API key, basic or none) and which requests depend on that authentication, before generating any test. Also to confirm against the environment how the login responds and what the read requests return with that session, and to know which values a request stores for the following ones in its flow. It proposes test flows and writes a structured report (api-collection-<name>.json). It does not generate scripts, does not run loads and does not write secrets. It tries the login only once and without retries, so the test account does not get locked.

Invocation example
{
  "input": {
    "collection": "collections/projects-api.postman_collection.json",
    "environment": "collections/local.postman_environment.json",
    "base_url": "http://localhost:8105",
    "probe": "yes",
    "include_mutations": "none"
  }
}
Input Type Description
— Required No mandatory inputs: whatever is missing is filled in from qa-stack.yaml or by autodetection.
input.qa_stack_path Optional qa-stack.yaml by default.
input.collection / input.environment Optional Postman collection (.json) or Bruno folder (with .bru) and its environment. If absent: performance.collection.* from qa-stack.yaml or autodetection.
input.base_url Optional Overrides the base URL from the environment and from qa-stack.yaml.
input.probe Optional yes (default) or no. With no, nothing is called.
input.include_mutations Optional none / writes / all: which data-modifying methods are tested and enter the load. If absent: performance.mutations.include, then writes.
input.mutation_probe_iterations Optional Integer from 0 to 5: how many times each write is repeated. If absent: performance.mutations.probe_iterations, then 3.
input.login_request Optional Name or path of a collection request; overrides login detection.
input.data_files / input.credential_env Optional CSV files with credentials (from performance.data_dir, data/ by default) or, if there are none, environment variables holding the probe credentials.
input.output_dir Optional Report folder (otherwise: performance.scripts_dir, then paths.reports_dir).

Output

Field Description
status read (collection read, even if not probed) / needs_input (no collection, or there is a login but no source provides credentials; with performance.on_missing: warn the latter stays read with a warning) / blocked (unreadable or unsupported collection, or no qa-stack.yaml with stack.test_framework k6/jmeter).
collection_name, source_kind, base_url Name, origin (postman / bruno) and resolved base URL.
auth A single authentication per collection: mode (bearer, cookie, api_key_header, api_key_query, basic, none) and its data.
login The full login request, with the user and password fields identified (without values), or null if there is none.
flows Flows with their requests: needs_auth, captures, observed_status, expected_statuses, mutates, read_only_post, destructive, plus covers_flows / requires_captures_from per flow.
probe Whether it was probed, and why or why not.
report_path Path of the written report (never overwrites an existing one: a date is added to the name).
missing / warnings What is impossible to infer, and notices (what was masked, conventions applied).
Chains with

Consumes a collection (with its environment and data CSVs) and delivers the report with auth, login and flows: exactly the data shape that the skills generating load scripts expect.


lippia-neo-generate-k6-script

When to use it

When there are already structured HTTP flows (with or without authentication) and you need to turn them into executable k6 load scripts, one per profile (smoke, load, stress, spike), or regenerate them after changing profiles, thresholds, load mix, authentication, data or flows (they are rendered again: generated files are not edited by hand). The code is not written by the agent: a deterministic renderer (same input, same output) generates it and computes the declared load. It does not parse collections, does not run the load, does not install k6 and does not interpret results.

Invocation example
{
  "input": {
    "flows_report": "performance/api-collection-projects-api.json",
    "profiles": ["smoke", "load"],
    "include_mutations": "none"
  }
}
Input Type Description
input.flows Required Non-empty array of flows { name, kind: login \| flow, requires_login, requests }; each request carries name, method and url (and optionally headers, body, checks, captures, needs_auth, etc.). At most one kind: login flow. It can be replaced by input.flows_report.
input.flows_report Optional Path to a JSON { base_url?, auth, login, flows }. If there is neither flows nor flows_report, it looks for the most recent api-collection-*.json with status: read in performance.scripts_dir (or paths.reports_dir).
input.auth / input.login Optional Authentication (mode and its data) and login request with each field marked as user, pass or literal. Without auth: none.
input.profiles Optional Subset of smoke, load, stress, spike (otherwise: performance.profiles, then all four).
input.profile_overrides, input.thresholds, input.mix, input.journeys, input.executor, input.pacing Optional Stages per profile, thresholds, load split across scenarios, business journeys, load model (vus / arrival_rate / sequential) and pause between iterations.
input.checks, input.unique_fields, input.on_exhaust Optional Per-request validations, body fields that must be unique, and what to do when the CSV rows run out (skip_writes by default, recycle, abort, synthesize).
input.include_mutations / include_destructive / include_probe_failed / include_write_without_body Optional Which requests enter the load. By default, destructive ones (DELETE), those a previous probe saw fail and those from a host other than the environment's are excluded.
input.data_files, input.variables, input.max_vus, input.base_url, input.output_dir, input.qa_stack_path Optional Data CSVs (only their columns are described, values are never read), non-sensitive placeholder values, open-model VU cap, base URL, output folder and path to qa-stack.yaml.

Output

Field Description
status generated (rendered and validated) / needs_input (flows missing and no other source) / blocked (no Node, invalid input or broken references, conflict with existing files that carry no marker, or a piece of data impossible to infer).
files, generation_report Files written and path of generation-report.json (warnings, exclusions, decisions, declared load and commands).
profiles, flows_count Profiles generated and scenarios included.
env_vars, required_env Environment variables the scripts read (names only) and those with no default that must be passed to run.
data_mapping, missing_data How each CSV column is used, and data that some flow needs and nobody provides.
excluded_requests, request_body_missing Requests that did not enter the load (with their reason) and writes without a body.
declared_load Declared load per profile, computed by the renderer. In the closed model it is an estimate.
decisions, defaults_applied Decisions taken and where they came from (input, qa-stack or default).
validate_command, run_command Commands to validate without generating load (k6 inspect) and to run one profile.
warnings Notices from the renderer and from the skill. If k6 is not available, the no_validado: k6 no disponible notice comes first.

Security rules of the generated scripts

Auth headers, sensitive-named fields and URLs with credentials are replaced by placeholders that are resolved against the environment, with no default value. A session that cannot be established never gives a green result: with bearer or cookie, a failed preparation login aborts the run.

Chains with

Consumes flows, authentication and login shaped as { base_url?, auth, login, flows }, and delivers the scripts folder (<profile>.js) together with generation-report.json, which execution and reporting use to know the declared load and the mandatory variables.


lippia-neo-generate-jmeter-script

When to use it

The same as the k6 skill, but for JMeter: when there are structured HTTP flows and you need to turn them into executable .jmx load plans, one per profile, or regenerate them after a change. The XML is written by a deterministic renderer. It validates each .jmx with its own structural check and, only if JMeter is available, with JMeter. It does not parse collections, does not run the load, does not install JMeter and does not interpret results.

Invocation example
{
  "input": {
    "flows_report": "performance/api-collection-projects-api.json",
    "profiles": ["smoke", "load"]
  }
}
Input Type Description
input.flows Required Same as in k6: non-empty array of flows with their requests (it can be replaced by input.flows_report).
input.flows_report, input.auth, input.login, input.profiles, input.profile_overrides, input.mix, input.journeys, input.executor, input.checks, input.unique_fields, input.on_exhaust, input.pacing, input.data_files, input.variables, input.max_vus, input.base_url, input.output_dir, input.qa_stack_path Optional Same meaning and same precedence (input, qa-stack.yaml, default) as in the k6 skill.
input.include_mutations / include_destructive / include_probe_failed / include_write_without_body Optional Same as in k6.
input.thresholds Optional Thresholds (http_req_failed, http_req_duration, throttled, server_errors, conflicts). checks and dropped_iterations do not exist in JMeter and are ignored with a warning. Thresholds are not plan assertions: they are written to generation-report.json and evaluated against the results.

Output

Same as in k6 (status, files, generation_report, declared_load, data_mapping, excluded_requests, decisions, validate_command, run_command, warnings), with these differences:

Field Description
tool Always jmeter.
declared_load basis is always estimate: with core JMeter even the open model is not exact.
validation { structural: ok \| failed \| not_run, jmeter: ok \| failed \| not_available }. The structural check is mandatory; the JMeter one only if it is available.
status blocked also if the structural check fails.

JMeter-specific limits

JMeter does not split the credentials CSV into separate user pools: if there is a measured login and flows with a session on the same CSV, credential_pools_not_disjoint is reported and the decision is left to whoever invokes the skill. Thresholds are not computed inside the plan but against the results.

Chains with

Same input contract as the k6 skill; it delivers the plans folder (scripts/<profile>.jmx, user.properties, README) and generation-report.json.


lippia-neo-run-performance-test

When to use it

When there is a directory with k6 scripts (<profile>.js) or JMeter plans (scripts/<profile>.jmx) per profile and you need to run them against an environment leaving reproducible evidence of each run, or repeat a run (for example with another time_scale) or run several profiles one at a time resetting the state of the system under test before each one. Profiles run always one at a time and in sequence, with a pause between them. It does not modify scripts or data, does not interpret results, does not install k6 or JMeter and does not execute anything outside the tool and the indicated reset_command.

Invocation example
{
  "input": {
    "tool": "k6",
    "profiles": ["smoke", "load"],
    "base_url": "http://localhost:8105",
    "env_var_names": ["LOGIN_USER", "LOGIN_PASSWORD"],
    "time_scale": 0.1
  }
}
Input Type Description
— Required No mandatory inputs: whatever is missing is filled in from qa-stack.yaml.
input.tool Optional k6 or jmeter. By default jmeter if stack.test_framework is jmeter; otherwise k6.
input.jmeter_path Optional Explicit path to the JMeter executable (without it: PATH, JMETER_HOME, standard locations and performance.tool_path).
input.project_dir, input.script_dir, input.results_dir Optional Project folder, scripts folder (performance.scripts_dir) and runs folder (performance.results_dir; results for k6 and target for JMeter by default).
input.profiles Optional smoke, load, stress, spike (by default performance.profiles).
input.base_url Optional Environment base URL (by default environment.base_url). Only its host is reported.
input.env_var_names, input.env_file, input.required_env Optional NAMES of the variables to inject (the value comes from the process environment or from a file outside the repo) and the script's mandatory variables.
input.time_scale, input.cooldown_s, input.max_duration Optional Duration scale, pause between profiles and per-profile duration cap.
input.reset_command / input.reset_args Optional Command that restores the state of the system under test; it runs before each profile.
input.on_concurrent Optional warn (default) or abort if other processes of the tool are already running on the machine.
input.confirm_remote Optional Explicit authorization to run stress or spike against a non-local host.
input.run_id Optional Prefix for each run's identifier.

Output

Field Description
status completed (all profiles were attempted, even if one ended with crossed thresholds) / partial (stopped early: the reset_command failed, or there are concurrent processes with on_concurrent: abort) / refused (mandatory variables missing or JMeter not found; no run was spent) / needs_input (a required input or the confirmation for a remote host is missing) / blocked (no Node, or k6 not installed or older than 0.49).
tool, tool_version Tool used and its version.
runs Per profile: run_dir, exit_code, exit_meaning (ok, thresholds_crossed, aborted, error, skipped, not_run), timed_out, timestamps and which files were left.
missing / warnings What is impossible to infer and notices (concurrent processes, expected files that are missing, scaled run).

Each run is left in <results_dir>/<profile>-<UTC yyyymmdd-HHmmss>/ with <profile>-load-check.json, console.utf8.txt, run-meta.json and, depending on the tool, summary.json and dashboard.html (k6) or results.jtl and dashboard/ (JMeter). With JMeter the load-check is converted from the JTL.

Authorization for remote hosts

If the base_url host is not local and the profiles include stress or spike, it only runs with confirm_remote: true in the input. Without it, it returns needs_input with missing: ["confirm_remote"] and launches nothing. An exit code of 99 (thresholds crossed) or 108 (aborted) is run data: it is reported, not retried or covered up.

Chains with

Consumes a scripts folder per profile (with its generation-report.json, from which it gets the mandatory variables) and delivers run folders with the raw evidence (run-meta.json, <profile>-load-check.json, tool results) plus runs.json in results_dir.


lippia-neo-write-performance-report

When to use it

When there are one or more run folders and you need a final report readable by a PO or dev lead; when there are runs of several profiles and you want a comparison table of offered load against latency to detect an inflection point; or when a run ended with crossed thresholds (99), aborted (108) or without executing iterations and it must be explained with its measurable causes. A deterministic script extracts only facts into report-data.json and the report is written using only those numbers. It does not run tests, does not modify scripts or results and does not invent metrics. The report is the same with k6 or JMeter.

Invocation example
{
  "input": {
    "project": "projects-api",
    "environment": { "simulated": false, "notes": "shared QA environment" },
    "format": "md"
  }
}
Input Type Description
— Required No mandatory inputs: whatever is missing is filled in from qa-stack.yaml.
input.run_dirs Optional Array of run folders. If absent: all runs of the latest invocation in performance.results_dir.
input.generation_report Optional Path or object with the declared (unscaled) load. By default <performance.scripts_dir>/generation-report.json; if declared_load is missing, it is declared as missing data.
input.environment Optional { simulated, notes }, declared by whoever requests the test (for example, that the service under test is a simulator). The skill never deduces it: without it, it stays as a missing input.
input.project Optional Project name used for the file name (by default, the folder of qa-stack.yaml).
input.reports_dir Optional Output folder (by default paths.reports_dir, then test-results/reports/).
input.format Optional md (default) or md+html.
input.qa_stack_path Optional qa-stack.yaml by default.

Output

Field Description
report_path, report_data_path The .md report and its data file, with a unique name per execution (never overwrites an existing one).
overall_status conclusive / conclusive_with_limits (there are limitations that do not block: simulated environment, generator on the same host, scaled run, single run) / non_conclusive (for example an aborted run with no load, session not established, requests that always return 4xx, saturated generator or rate limiting that invalidates the offered load) / blocked (no Node, nonexistent path or no run with data).
runs, flags Runs with their exit code and the flags present.
bottleneck_declared true only if there is cited evidence; otherwise the report states "saturation point not reached" (no se alcanzó el punto de saturación).
missing_inputs, missing, warnings Missing inputs, what is impossible to infer (for example, no runs) and notices (including what was masked).

How it is written

Every number comes from the data file, with its value and unit; an absent datum is written as "no data" (sin dato). A 429 is rate control and a 409 a state conflict: neither is presented as a system failure or as saturation. A load generator problem is never attributed to the system under test, and an exit 99 or 108 is never covered up. With JMeter, iterations and offered load appear as n/a and the load is described in served requests per second.

Chains with

Consumes run folders (tool results, <profile>-load-check.json and run-meta.json) and, optionally, the generation-report.json with the declared load; delivers the final report and its data file in reports_dir.