Evaluation Pipeline

Understand what Divekit Evaluation Pipeline does, how it is configured, and how it links work and eval repositories.

What It Is

Evaluation Pipeline is Divekit’s two-repository mode for a single assignment UUID:

  • work repository: student-facing code repository
  • eval repository: evaluation repository

When enabled, distribution creates and links both repositories for each UUID.

What It Does

For each UUID, Divekit:

  1. Creates the work and eval repositories (depending on your target/group layout).
  2. Individualizes content per UUID (same UUID mapping across targets).
  3. Injects evaluation-pipeline CI assets.
  4. Links both repositories by setting cross-repository variables and tokens.
  5. Configures trigger/job-token scope so pipelines can communicate.
  6. Disables GitLab’s unique Pages domain setting on eval repositories when the provider supports project-level Pages settings.

Runtime Flow

flowchart LR
  A["Student pushes to work repo"] --> B["work .gitlab-ci.yml"]
  B --> C["Trigger eval pipeline"]
  C --> D["eval pipeline prepares test env"]
  D --> E["eval runs build/tests/plugins"]
  E --> F["reports/artifacts"]

CI Assets

Divekit keeps evaluation-pipeline authoring files under .divekit/eval-pipeline/. If custom files are missing, embedded defaults are used.

Project-level files:

  • .divekit/eval-pipeline/work/.gitlab-ci.yml
  • .divekit/eval-pipeline/eval/.gitlab-ci.yml
  • .divekit/eval-pipeline/work/.gitlab/ci/divekit/eval-trigger.yml
  • .divekit/eval-pipeline/materialization.json
  • .divekit/eval-pipeline/plugins.json
  • .divekit/eval-pipeline/eval/.gitlab/ci/divekit/plugins/<plugin-id>.yml

Distribution-specific files can be placed under .divekit/distributions/<distribution>/eval-pipeline/. Root-level .gitlab-ci_workrepo.yml and .gitlab-ci_evalrepo.yml still work as explicit overrides.

Legacy .divekit/evaluation-pipeline/ and secure-pipeline/ paths are still accepted for backward compatibility. The old setup_evalrepo.sh and test_evalrepo.sh hooks are no longer part of the current default flow.

In generated work repositories, .divekit/eval-pipeline/work/.gitlab-ci.yml becomes .gitlab-ci.yml. The default work CI then includes .gitlab/ci/divekit/eval-trigger.yml, which is generated from .divekit/eval-pipeline/work/.gitlab/ci/divekit/eval-trigger.yml.

You can scaffold/update these with:

divekit inject --scope ci

CLI Runtime Image

The default eval pipeline runs Divekit’s setup and report commands from the versioned CLI runtime image. DIVEKIT_CLI_VERSION is set to the CLI version that generated the template, and DIVEKIT_CLI_IMAGE normally resolves to the matching registry.git.nrw/divekit/divekit-cli image. This keeps the pipeline CLI aligned with its generated CI files and avoids downloading a binary in every run.

The GitLab runner must be able to pull that image. To test an unreleased build, override DIVEKIT_CLI_IMAGE with a compatible image. Alternatively, set DIVEKIT_CLI_URL to a raw binary, .zip, or .tar.gz; in that mode the chosen job image only needs wget and the pipeline downloads the explicit build. Remove development overrides before distributing a released course setup.

Pages URL Behavior

Divekit keeps eval Pages URLs predictable by disabling GitLab’s unique Pages domain setting during distribution and reuse flows. This is useful when course material links directly to eval Pages content.

For already distributed repositories, run:

divekit patch -d ST2M4 --disable-eval-pages-unique-domain

The patch flag only affects the eval target and can be combined with normal file patches when needed.

Teaching-Team Feedback

Course staff can add direct feedback to existing eval report pages with divekit feedback and divekit patch feedback. Use this for human feedback that should be visually separate from plugin-generated hints or FAQ entries.

# Author feedback locally in the origin repository
divekit feedback add -d ST2M4 --all --message "Remember to explain your design choices."

# Patch all authored feedback into existing eval repositories
divekit patch feedback -d ST2M4

Authored feedback is stored under .divekit/distributions/<distribution>/feedback/. Patched eval repositories receive .divekit/feedback/index.json, which divekit report renders as a teaching-team thread.

See Manual Feedback for the full workflow.

Work-To-Eval Materialization

Student edits happen in the work repository, while tests and hidden evaluation material live in the eval repository. When the eval pipeline starts, Divekit runs its internal eval setup, fetches the current work repository state, and applies .divekit/materialization.json in the eval repository.

For most projects you do not write that generated manifest by hand. Instead, mark files or directories in the origin repository with these suffixes:

  • _copy_to_eval: keep the path editable in the work repo and copy it into the eval repo during setup
  • _overwrite_to_eval: keep the path editable in the work repo and replace the matching eval-repo path during setup

The suffix is removed in the distributed work repository. These suffixes complement the existing target suffixes such as _workrepo, _evalrepo, and _norepo.

Example origin layout:

src/main/resources/E2_copy_to_eval/
src/main/java/example/solution_overwrite_to_eval/
src/main/resources/E2/E02minimal_evalrepo.uxf

Result:

  • work repo: students see src/main/resources/E2/ and src/main/java/example/solution/
  • eval repo: hidden static files stay in eval-only paths, then the eval setup copies or replaces the marked work paths before tests run

Use _copy_to_eval when eval-only files may already exist next to the copied path. Use _overwrite_to_eval when the work-repo path should completely replace the eval-repo path before evaluation.

Advanced projects can add explicit rules in .divekit/eval-pipeline/materialization.json, or in the matching distribution-specific eval-pipeline directory. Divekit merges those rules with the marker-derived rules and writes the generated eval-repo manifest to .divekit/materialization.json. See materialization.json for the full file format.

When updating existing repositories, patch the origin path you changed. Evaluation authoring paths and work-to-eval marker paths are handled by the same target rules as a normal distribution run.

Linking Variables (Auto-Managed)

When linking work -> eval, Divekit sets variables like:

  • In eval: WORK_REPO_URL, WORK_REPO_TOKEN
  • In work: EVAL_REPO_TRIGGER_URL, EVAL_REPO_TRIGGER_TOKEN

Backward-compatibility aliases are also set for existing templates (for example MAIN_REPO_URL, CODE_REPO_URL, and TEST_REPO_TRIGGER_URL).

If repository URL paths are changed afterwards with divekit patch name --scope path or --scope both, Divekit reconciles these linking variables again and re-patches origin files that contain Pages URL tokens.

Configuration Keys

Configure in .divekit/distributions/<distribution>/config.json:

{
  "evaluationPipeline": {
    "enabled": true,
    "groupId": 67890
  },
  "groupLayout": {
    "mode": "distribution-subgroups",
    "rootGroupId": 12345
  }
}

Key behavior:

  • evaluationPipeline.enabled: enables the paired eval target.
  • evaluationPipeline.groupId:
    • in distribution-subgroups: the eval distribution subgroup is created under this group
    • in legacy: the eval target uses this group directly
  • groupLayout.mode:
    • distribution-subgroups (default): <distribution>/work and <distribution>/eval style layout
    • legacy: old fallback behavior with an eval subgroup under the work group (unless explicit evaluationPipeline.groupId is set)
  • groupLayout.rootGroupId: optional fixed root for subgroup derivation.

evaluationPipeline is the canonical config key. Older configs using securePipeline are still read for compatibility. When using distribution-subgroups, Divekit now uses work and eval as subgroup names, but existing legacy student subgroups are still reused for compatibility.

Enable It

You can enable the evaluation pipeline during divekit init (interactive prompt), or afterwards:

divekit config origin set -d ST2M4 --key evaluationPipeline.enabled --value true
divekit config origin set -d ST2M4 --key evaluationPipeline.groupId --value 67890

Useful checks:

divekit config explain origin
divekit distribute -d ST2M4 --dry-run

Plugin-Driven CI Features

Evaluation-pipeline CI templates can include plugin toggles (for example PMD/Surefire). Manage them with:

divekit plugin
divekit plugin install
divekit plugin list
divekit plugin enable pmd
divekit plugin disable surefire
divekit plugin add pmd --source official --ref v0.3.0

For plugin setup and activation, see Plugins. For plugin authoring and contracts, see Plugin Development.

Security Notes

Evaluation Pipeline separates CI execution into a paired repository and automates token-based linking. It does not automatically define your final permission model. Always review group/project permissions in GitLab for your course setup.

Last modified August 11, 2026: docs(plugin): describe image-based runtime (3a21f3d)