Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

8.5 KB
Raw Blame
---
title: "GitLab CI/CD"
description: "Add quality and security checks to GitLab with no model key required."
---

Bring quality and security checks to your repository. No model key required.
Optional LLM checks are available. Point the reusable GitLab CI job at your skills
and choose a quality threshold; merge requests evaluate changed skills
without installing scanners or adding evaluation code to your repository.
The shared container includes SkillSpector security scanning.

## Get started

The shared image applies one default validation policy to public and private
repositories. No `internal`/`external` variable is needed. Older image reports
label the same policy `external`; repository access still controls scan artifacts.

Copy [gitlab-skill-checks.yml](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/gitlab-skill-checks.yml)
from the [ready-to-use templates](https://github.com/NVIDIA/SkillEvaluator/tree/main/ci/bundles/1.0.1).
It includes the pinned shared configuration and scanner image. You do not need
a GitLab mirror or a local onboarding CLI.

<Accordion title="I prefer to inspect the source template">
  Browse the [GitLab source template](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/gitlab-skill-checks.yml)
  and [template catalog](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/README.md).
  Source files contain placeholders; use the filled-in copy when onboarding.
</Accordion>

<Steps>

### Add the reusable job

Copy `gitlab-skill-checks.yml` from the template folder using GitHub's **Raw** view. For a new pipeline,
use its contents as `.gitlab-ci.yml`. For an existing pipeline, merge its
`include`, variables and `skill-evaluation` rules into your root file.

### Choose your skills and score

Set `SKILL_EVAL_PATHS` to your skill directory and `SKILL_EVAL_MIN_SCORE` to the
required quality score from 0 to 100 (default 70). For multiple roots, use a
YAML multiline string.

### Open a merge request

The job evaluates changed skills, shows the result in the pipeline, and retains
reports as job artifacts, including when a check fails.

</Steps>

The generated file has this shape; the release fills in the immutable pins:

```yaml title=".gitlab-ci.yml"
include:
  - remote: https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/RELEASE_WORKFLOW_SHA/ci/gitlab/skill-evaluation.yml
variables:
  SKILL_EVAL_IMAGE: ghcr.io/nvidia/skillevaluator-tier1@sha256:RELEASE_IMAGE_DIGEST
  SKILL_EVAL_PATHS: "skills"
  SKILL_EVAL_MIN_SCORE: "70"
skill-evaluation:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
    - if: '$CI_PIPELINE_SOURCE == "web"'
```

A compatible runner must be able to pull the image and run Linux containers.
If it requires tags, add them to `skill-evaluation`. Static scanning needs no
model key, but dependency vulnerability checks may contact external services.

## See the results

Open the `skill-evaluation` job and download its `skill-evaluator-report`
artifact. Open `index.html` for the consolidated view, then select a skill for
its detailed HTML report. JSON, Markdown and SARIF are included.

<Frame caption="Real demo result: the skill scored 98.2 and failed a 100-point threshold. The score and the scan verdict are separate.">
  <img src="./assets/repository-ci-results-example.png" alt="Example report table showing a failed skill with a quality score of 98.2 and a link to its report" />
</Frame>

The shared `skill-evaluation` job writes JSON, Markdown and SARIF reports; it
does not produce JUnit XML yet. To show findings as a native GitLab
merge-request test report, run SkillEvaluator in your own job with `-r junit`
and attach the file it writes. `tier1` writes
`reports/skillevaluator-tier1.junit.xml`:

```yaml title="Run it yourself: JUnit test report"
skill-junit:
  script:
    - skillevaluator tier1 skills/foo -r junit -o reports
  artifacts:
    when: always
    reports:
      junit: reports/skillevaluator-tier1.junit.xml
```

`skillevaluator validate` on a folder of skills writes one file per skill
instead, as `reports/<skill>/skillevaluator-output-<timestamp>.junit.xml`;
point `junit:` at `reports/**/*.junit.xml` to include nested skill folders.

The [public examples](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/examples/README.md)
include this report preview and a log excerpt without requiring access to a
separate demo project.

<Tabs>
  <Tab title="Job log">

    The log identifies the skill, result and quality score. This shortened
    excerpt is from the real demo run shown above:

```text
Skill result: "skills/glossary-builder" — FAILED
  Quality: 98.2/100, grade "A"
  Check: "QUALITY" — "failed"
    Scanner error: "Quality score 98.2/100 is below the required 100/100 ..."
```

  </Tab>
  <Tab title="Report artifact">

    Download the artifact from the job to inspect the overview and per-skill
    findings. Artifacts expire according to the job's retention settings.

  </Tab>
</Tabs>

<AccordionGroup>
  <Accordion title="Author emails and reviewed security exceptions">
    Keep the `metadata.author` attribution. Extra email copies in
    contact fields or documents are checked separately; use a handle or a
    contact-directory link instead. See [author attribution](/skills/skillevaluator/tier1-reference#author-attribution-and-contact-emails).

    For reviewed SkillSpector exceptions, set
    `SKILL_EVAL_SECURITY_BASELINES: "true"` and add
    `.skillevaluator/skillspector-baselines.yaml` using a template and image
    version that include baseline support. CI reads the approved mapping
    and baseline files from the merge request's target revision. Regenerate and
    review fingerprint exceptions when the scanner version changes. Baselines
    do not waive PII or incomplete analysis. Follow the [baseline setup guide](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#reviewed-security-baselines).
  </Accordion>
  <Accordion title="When does it run?">
    `SKILL_EVAL_SCAN_MODE: auto` evaluates changed skills in detached
    merge-request pipelines and all configured skills on the default branch or
    a manual pipeline. Set it to `all` for a full scan each time. Changed-only
    selection needs the merge-request diff base and full Git history; missing
    history is an error, not a pass. Merged-results and merge-train pipelines
    must use `all` until changed-only selection is supported for them. Shared
    files outside skill directories do not select dependent skills; use `all`
    when those changes require a complete scan.
  </Accordion>
  <Accordion title="Fit the job into an existing pipeline">
    GitLab requires merge-request `rules` in the root `.gitlab-ci.yml`, even
    when a job is included. The generated file provides them. If your project
    has `workflow: rules`, ensure they allow merge-request pipelines while
    preserving your existing policy. The job uses the `test` stage by default;
    override its stage if your pipeline does not define it. A failed or
    incomplete job fails the pipeline; configure your GitLab merge policy if it
    should also block merging.
  </Accordion>
  <Accordion title="Scores, reports and privacy">
    The threshold applies to the quality score; security, schema and other
    required checks remain independent gates. The scanner runs quality and
    security checks and saves reports as job artifacts; it does not host reports
    or post merge-request comments automatically. Known values detected by
    credential patterns, NVIDIA API
    keys, and unquoted `KEY=value` and `Bearer` credentials are redacted,
    including parts of them that a scanner cuts or escapes, but not every
    secret or transformation is detectable. Raw scanner logs are
    withheld when a credential is present. Remove and rotate exposed
    credentials; previous artifacts and Git history are not rewritten.
  </Accordion>
</AccordionGroup>

For GitHub and other CI systems, see [Repository Integration](/skills/skillevaluator/repository-integration).

<div id="three-tier-checks-and-feedback">
<div id="merge-into-an-existing-root-configuration">

For the explicit all-tier opt-in and its prerequisites, see [Advanced CI](/skills/skillevaluator/advanced-ci).
The [GitLab setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#gitlab-checks-and-feedback) and [all-tier pipeline migration](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#merge-into-an-existing-root-configuration) references preserve the detailed contracts.

</div>
</div>