Configure CI
This section explains how to make NEO run on its own in the repository's pipeline, on GitLab, GitHub, Bitbucket, or Jenkins, without anyone having to open the AI assistant.
| Platform | Status | Guide |
|---|---|---|
| GitLab CI | ✅ The most complete: it triggers on its own, saves the results in a new MR, and comments on the original MR | Configure GitLab CI |
| GitHub Actions | ⚠️ It works, but it doesn't open the results PR automatically | Configure GitHub Actions |
| Bitbucket Pipelines | ⚠️ The pipeline is there, but today it doesn't trigger on its own from a label: you have to start it by hand | Configure Bitbucket Pipelines |
| Jenkins (+ Bitbucket Cloud) | ⚠️ Runs on every PR and comments the result; no results branch, not tested end to end yet | Configure Jenkins |
1. What does NEO do in CI?
In CI, NEO runs in Model 4 (declarative) (see The 4 orchestration models): there's no person chatting with the assistant, but a fixed plan that the Crowdar engine runs step by step.
The main flow is this:
Someone opens (or updates) an MR/PR and adds the label feature
│
▼
The pipeline runs the job feature-e2e inside the image crowdar-engine-mvp
1. Clones the NEO skills (crowdar-qa-skills)
2. Detects the framework from qa-stack.yaml and installs dependencies
3. Builds .env.tests with the app credentials (from CI variables)
4. Reads the MR/PR (neo event) and runs the plan feature-new-end-to-end
(read user story → plan → exploration → suite → run and heal → report)
5. Saves the results (plans, tests, reports) to a new branch
│
▼
publish-report comments the result on the MR/PR (pass or fail)
Why with a label: the plan uses an AI model and a real application, so it takes time and uses credits. Instead of running on every commit, it runs only on MRs/PRs that have the
featurelabel, and it runs again on every push as long as the label stays on.
2. How the files are generated: lippia-neo-ci-bootstrap
You don't need to write the pipeline by hand. The lippia-neo-ci-bootstrap skill generates it from qa-stack.yaml:
Generate the project's CI with lippia-neo-ci-bootstrap
Before running it:
qa-stack.yamlmust exist withstack.pipeline_provider(gitlab,github,bitbucket, orjenkins; Jenkins also needsstack.scm_provider). If it doesn't exist, the skill stops and asks you to runlippia-neo-bootstrap-stackfirst. It's part of the New Project Bootstrap playbook.- If the framework is Playwright,
@playwright/testmust be in the project'spackage.json. Otherwise, the runner installs an arbitrary version instead of the project's. - If the framework is Cypress, the generated jobs don't install its dependencies yet (on GitHub and Bitbucket it is treated as Playwright; on GitLab and Jenkins nothing is installed). Manually add
npm ciandnpx cypress installto the install step. See Configure Cypress.
The skill fills in the project data in the files it generates, based on qa-stack.yaml: test_framework, tests_dir, test_plans_dir, scripted_dir, and reports_dir.
It doesn't overwrite your pipeline
If the repository already has its own pipeline, lippia-neo-ci-bootstrap doesn't overwrite it. The NEO jobs always live in their own neo-ci-jobs/ folder:
| Platform | NEO files |
If you already have a pipeline |
|---|---|---|
| GitLab | .gitlab/neo-ci-jobs/ |
Adds one line include: to your .gitlab-ci.yml |
| GitHub | .github/workflows/neo-ci-jobs/ |
Creates a separate workflow (neo-ci-jobs.yml). It doesn't touch yours |
| Bitbucket | bitbucket-pipelines.yml (+ a reference copy in .bitbucket/neo-ci-jobs/) |
Bitbucket doesn't allow splitting the pipeline across several files: the skill shows you the proposed merge and asks for confirmation before writing |
| Jenkins | .jenkins/neo-ci-jobs/Jenkinsfile |
Nothing to merge: a separate Jenkins job points to that file, and your Jenkinsfile is untouched |
3. What you need to configure on the platform
The generated files aren't enough: you also need to add secret variables in the repository settings. The ones common to all three platforms:
| Variable | What it's for | Required? |
|---|---|---|
ANTHROPIC_API_KEY |
The engine uses Claude models | Yes (or OPENAI_API_KEY, depending on the plan) |
OPENAI_API_KEY |
If the plan uses OpenAI models | Depends on the plan |
CROWDAR_CI_TOKEN |
Commenting on the MR/PR and pushing the results branch | Yes |
APP_USER / APP_PASSWORD |
The app's test user; they are written to .env.tests |
If the app has a login |
APP_BASE_URL |
URL of the app under test | Yes |
On GitHub and Bitbucket you also need:
| Variable | What it's for |
|---|---|
CROWDAR_REGISTRY_USER / CROWDAR_REGISTRY_TOKEN |
Downloading the crowdar-engine-mvp image, which is in Crowdar's private GitLab registry |
SKILLS_REPO_TOKEN |
Cloning crowdar-qa-skills, which is in Crowdar's GitLab |
On GitLab these two things are handled automatically, because the image and the skills are on the same GitLab instance (
$CI_REGISTRYand$CI_JOB_TOKEN).
If your user stories come from Jira or Azure DevOps
.mcp.json isn't used in CI: that file is in .gitignore and never reaches the runner. The engine launches the MCP servers with the job's environment variables, so you need to add as CI variables the same keys you configured in Jira or Azure DevOps:
| Provider | Variables |
|---|---|
| Jira | JIRA_URL, JIRA_USERNAME, JIRA_API_TOKEN, CONFLUENCE_URL, CONFLUENCE_USERNAME, CONFLUENCE_API_TOKEN |
| Azure DevOps | AZURE_DEVOPS_ORG_URL, AZURE_DEVOPS_AUTH_METHOD (pat), AZURE_DEVOPS_PAT, AZURE_DEVOPS_DEFAULT_PROJECT (optional) |
For CI, it's best to use a service account (not your personal user), with read-only permissions.
⚠️ GitHub: the generated workflow currently doesn't pass these variables to the plan's job. See the GitHub guide.
4. Quick comparison
| GitLab | GitHub | Bitbucket | Jenkins | |
|---|---|---|---|---|
| Entry file | .gitlab-ci.yml |
.github/workflows/qa.yml |
bitbucket-pipelines.yml |
.jenkins/neo-ci-jobs/Jenkinsfile (Multibranch job) |
| How it's triggered | MR with the feature label |
PR with the feature label |
By hand (custom pipeline qa-feature-e2e) |
Every PR (Bitbucket has no labels) |
| Where secrets are stored | Settings → CI/CD → Variables | Settings → Secrets and variables → Actions | Repository settings → Repository variables | Manage Jenkins → Credentials |
| Comment with the result | ✅ On the MR | ✅ On the PR | ✅ On the PR | ✅ On the Bitbucket PR |
| Automatic MR/PR with the results | ✅ | ❌ Pushes the branch, but doesn't open the PR | ❌ Pushes the branch | ❌ Results stay as build artifacts |
| Engine image | $CI_REGISTRY/.../crowdar-engine-mvp |
The same, with registry user and token | The same, with registry user and token | The same, with the crowdar-gitlab credential |
5. Good to know
- The skills are cloned from the
mainbranch ofcrowdar-qa-skills(SKILLS_REFvariable). If you want to try a version from another branch, change that variable in the generated file. - The engine image is pinned to a version (
crowdar-engine-mvp:c4cd2740-playwright). To update it, change the tag in the generated file. - There are commented-out or example "legacy" jobs (
test-suite-chromium,test-suite-firefox,api-suite) from the previous design. They don't run, except for the Bitbucket example pipelines, which only doecho. - The results are kept as job artifacts for one week (
result.json, plans, tests, and reports).