Authoring Guide

How to structure and publish a Divekit Evaluation Pipeline plugin repository.

Authoring contracts are still experimental and may evolve.

This page is now only a stable pointer.

Source of Truth

Plugin authoring and CI conventions should be maintained in the plugin template repository:

That repository should define:

  • repository layout
  • required files such as ci.yaml or ci.yml, .gitlab-ci.yml, and plugin.json
  • the plugin image built from Dockerfile and implementation code under src/
  • optional reusable feedback assets under feedback/
  • release-template usage
  • plugin artifact-writing conventions
  • FAQ artifact conventions
  • copyable examples for new plugin repositories

Stable Summary

Every plugin repository should provide:

  • ci.yaml (preferred) or ci.yml
  • plugin.json
  • a maintainer-readable README.md

Plugins with implementation code should also provide a Dockerfile; production plugins should provide a release-oriented .gitlab-ci.yml. For stable SemVer tags, Divekit prefers the plugin’s GitLab release ZIP asset over reading files from the repository tree. The release asset should contain ci.yaml or ci.yml at the archive root and may include feedback/**. Legacy archives that still contain src/** remain readable, but Divekit ignores those entries. Branch and commit refs are still read directly from the repository tree for development.

Plugin implementation code lives under src/ and is copied into the released container image, commonly below /opt/divekit/src/:

FROM alpine:3.19

COPY src/ /opt/divekit/src/
COPY feedback/ /opt/divekit/feedback/

The execution job selects the image produced for the plugin release and invokes the implementation in place:

image: registry.git.nrw/divekit/plugins/divekit-plugin-example:__DIVEKIT_PLUGIN_VERSION__
script:
  - python /opt/divekit/src/main.py

The shared image-and-release template replaces __DIVEKIT_PLUGIN_VERSION__ with the release tag. Keep dependency installation in the image build instead of running apt, pip, or equivalent installers in every evaluation pipeline.

Optional feedback/ files may still be packaged in the release asset and materialized below .gitlab/ci/divekit/plugins/<plugin-id>/feedback/. DIVEKIT_PLUGIN_DIR points to that materialized plugin directory. It is not a code-delivery mechanism; src/** is deliberately not copied into evaluation repositories.

If the plugin wants to show feedback on the generated GitLab Page, write it to the plugin result artifacts:

plugin-results/<plugin-id>/feedback.md
plugin-results/<plugin-id>/feedback/index.json
plugin-results/<plugin-id>/feedback/**/*.md

Static Markdown resources can live in the plugin repository under feedback/ and be copied or referenced by the plugin job when it builds run-specific feedback. Feedback is published through the eval repository’s GitLab Page; it is not written back to student repositories.

Items declared in feedback/index.json can set defaultOpen to control whether the generated Page expands them initially. If the field is omitted, test-specific feedback is expanded by default and global feedback stays collapsed by default.

Use report notices when an annotation belongs to one concrete report result instead of the broader feedback section. Notices are plain text and render above the report body on the generated Page.

For execution problems that are not test results, publish issues instead of fake tests:

plugin-results/<plugin-id>/issues/index.json
plugin-results/<plugin-id>/issues/**/*.md
plugin-results/<plugin-id>/logs/**

Issue items can reference a Markdown explanation and an optional log file. Use affectsResult: true when the problem means the evaluation could not run successfully.

For the CLI-consumed semantic contract, see Contract Reference.