Skip to content

Configure CI

This section explains how to make NEO run on its own in the repository's pipeline, on GitLab, GitHub, Bitbucket, or Jenkins, without anyone having to open the AI assistant.

Platform Status Guide
GitLab CI ✅ The most complete: it triggers on its own, saves the results in a new MR, and comments on the original MR Configure GitLab CI
GitHub Actions ⚠️ It works, but it doesn't open the results PR automatically Configure GitHub Actions
Bitbucket Pipelines ⚠️ The pipeline is there, but today it doesn't trigger on its own from a label: you have to start it by hand Configure Bitbucket Pipelines
Jenkins (+ Bitbucket Cloud) ⚠️ Runs on every PR and comments the result; no results branch, not tested end to end yet Configure Jenkins

1. What does NEO do in CI?

In CI, NEO runs in Model 4 (declarative) (see The 4 orchestration models): there's no person chatting with the assistant, but a fixed plan that the Crowdar engine runs step by step.

The main flow is this:

Someone opens (or updates) an MR/PR and adds the label  feature
            │
            ▼
The pipeline runs the job  feature-e2e  inside the image  crowdar-engine-mvp
   1. Clones the NEO skills (crowdar-qa-skills)
   2. Detects the framework from qa-stack.yaml and installs dependencies
   3. Builds .env.tests with the app credentials (from CI variables)
   4. Reads the MR/PR (neo event) and runs the plan  feature-new-end-to-end
      (read user story → plan → exploration → suite → run and heal → report)
   5. Saves the results (plans, tests, reports) to a new branch
            │
            ▼
publish-report  comments the result on the MR/PR (pass or fail)

Why with a label: the plan uses an AI model and a real application, so it takes time and uses credits. Instead of running on every commit, it runs only on MRs/PRs that have the feature label, and it runs again on every push as long as the label stays on.


2. How the files are generated: lippia-neo-ci-bootstrap

You don't need to write the pipeline by hand. The lippia-neo-ci-bootstrap skill generates it from qa-stack.yaml:

Generate the project's CI with lippia-neo-ci-bootstrap

Before running it:

  1. qa-stack.yaml must exist with stack.pipeline_provider (gitlab, github, bitbucket, or jenkins; Jenkins also needs stack.scm_provider). If it doesn't exist, the skill stops and asks you to run lippia-neo-bootstrap-stack first. It's part of the New Project Bootstrap playbook.
  2. If the framework is Playwright, @playwright/test must be in the project's package.json. Otherwise, the runner installs an arbitrary version instead of the project's.
  3. If the framework is Cypress, the generated jobs don't install its dependencies yet (on GitHub and Bitbucket it is treated as Playwright; on GitLab and Jenkins nothing is installed). Manually add npm ci and npx cypress install to the install step. See Configure Cypress.

The skill fills in the project data in the files it generates, based on qa-stack.yaml: test_framework, tests_dir, test_plans_dir, scripted_dir, and reports_dir.

It doesn't overwrite your pipeline

If the repository already has its own pipeline, lippia-neo-ci-bootstrap doesn't overwrite it. The NEO jobs always live in their own neo-ci-jobs/ folder:

Platform NEO files If you already have a pipeline
GitLab .gitlab/neo-ci-jobs/ Adds one line include: to your .gitlab-ci.yml
GitHub .github/workflows/neo-ci-jobs/ Creates a separate workflow (neo-ci-jobs.yml). It doesn't touch yours
Bitbucket bitbucket-pipelines.yml (+ a reference copy in .bitbucket/neo-ci-jobs/) Bitbucket doesn't allow splitting the pipeline across several files: the skill shows you the proposed merge and asks for confirmation before writing
Jenkins .jenkins/neo-ci-jobs/Jenkinsfile Nothing to merge: a separate Jenkins job points to that file, and your Jenkinsfile is untouched

3. What you need to configure on the platform

The generated files aren't enough: you also need to add secret variables in the repository settings. The ones common to all three platforms:

Variable What it's for Required?
ANTHROPIC_API_KEY The engine uses Claude models Yes (or OPENAI_API_KEY, depending on the plan)
OPENAI_API_KEY If the plan uses OpenAI models Depends on the plan
CROWDAR_CI_TOKEN Commenting on the MR/PR and pushing the results branch Yes
APP_USER / APP_PASSWORD The app's test user; they are written to .env.tests If the app has a login
APP_BASE_URL URL of the app under test Yes

On GitHub and Bitbucket you also need:

Variable What it's for
CROWDAR_REGISTRY_USER / CROWDAR_REGISTRY_TOKEN Downloading the crowdar-engine-mvp image, which is in Crowdar's private GitLab registry
SKILLS_REPO_TOKEN Cloning crowdar-qa-skills, which is in Crowdar's GitLab

On GitLab these two things are handled automatically, because the image and the skills are on the same GitLab instance ($CI_REGISTRY and $CI_JOB_TOKEN).

If your user stories come from Jira or Azure DevOps

.mcp.json isn't used in CI: that file is in .gitignore and never reaches the runner. The engine launches the MCP servers with the job's environment variables, so you need to add as CI variables the same keys you configured in Jira or Azure DevOps:

Provider Variables
Jira JIRA_URL, JIRA_USERNAME, JIRA_API_TOKEN, CONFLUENCE_URL, CONFLUENCE_USERNAME, CONFLUENCE_API_TOKEN
Azure DevOps AZURE_DEVOPS_ORG_URL, AZURE_DEVOPS_AUTH_METHOD (pat), AZURE_DEVOPS_PAT, AZURE_DEVOPS_DEFAULT_PROJECT (optional)

For CI, it's best to use a service account (not your personal user), with read-only permissions.

⚠️ GitHub: the generated workflow currently doesn't pass these variables to the plan's job. See the GitHub guide.


4. Quick comparison

GitLab GitHub Bitbucket Jenkins
Entry file .gitlab-ci.yml .github/workflows/qa.yml bitbucket-pipelines.yml .jenkins/neo-ci-jobs/Jenkinsfile (Multibranch job)
How it's triggered MR with the feature label PR with the feature label By hand (custom pipeline qa-feature-e2e) Every PR (Bitbucket has no labels)
Where secrets are stored Settings → CI/CD → Variables Settings → Secrets and variables → Actions Repository settings → Repository variables Manage Jenkins → Credentials
Comment with the result ✅ On the MR ✅ On the PR ✅ On the PR ✅ On the Bitbucket PR
Automatic MR/PR with the results ✅ ❌ Pushes the branch, but doesn't open the PR ❌ Pushes the branch ❌ Results stay as build artifacts
Engine image $CI_REGISTRY/.../crowdar-engine-mvp The same, with registry user and token The same, with registry user and token The same, with the crowdar-gitlab credential

5. Good to know

  • The skills are cloned from the main branch of crowdar-qa-skills (SKILLS_REF variable). If you want to try a version from another branch, change that variable in the generated file.
  • The engine image is pinned to a version (crowdar-engine-mvp:c4cd2740-playwright). To update it, change the tag in the generated file.
  • There are commented-out or example "legacy" jobs (test-suite-chromium, test-suite-firefox, api-suite) from the previous design. They don't run, except for the Bitbucket example pipelines, which only do echo.
  • The results are kept as job artifacts for one week (result.json, plans, tests, and reports).