Garden Linux

Appearance

Test Coverage Markers ​

This guide explains how to create and use test coverage markers for establishing traceability between Garden Linux features and tests, as defined in ADR-0032. The following sections detail the tooling, marker syntax, categories and placement guidelines.

Coverage.py and Test Coverage Analysis ​

Tool Location: tests/util/coverage.py

What is coverage.py? ​

coverage.py is a static analysis tool that generates coverage reports for Garden Linux, implementing ADR-0032, by analyzing the relationship between test coverage markers (GL-TESTCOV-*) next to a feature's setting and their corresponding test cases. Unlike traditional code coverage tools that require test execution, this tool performs static analysis to determine which features have test coverage.

These markers follow a specific format (GL-TESTCOV-$feature-$category-$description - see Marker Schema for details) that enables automated parsing and analysis. The tool scans both feature definitions and test files to establish traceability between what's configured and what's tested.

Key Capabilities ​

  1. Static Coverage Analysis

    • Analyzes test coverage markers in features without running tests
    • Scans test files for @pytest.mark.testcov() decorators
    • Generates coverage reports showing which features are tested

  2. Coverage Reporting

    • Lists all test coverage markers defined in features
    • Shows which test coverage markers have corresponding tests
    • Identifies untested features and gaps in test coverage
    • Provides percentage coverage statistics

  3. Traceability

    • Establishes clear links between feature configurations and tests
    • Enables audit trail for compliance requirements
    • Supports version-independent testing strategies

How It Works ​

Step 1: Feature Scanning

  • Scans feature directories (features/*/) for test coverage markers
  • Extracts markers from inline comments in shell scripts (see Inline Comments)
  • Parses file.include.markers.yaml and initrd.include.markers.yaml files for file-based markers (see Split ID Files)
  • Builds a complete inventory of all defined test coverage markers across all marker categories (see Category Reference)

Step 2: Test Scanning

  • Scans test files (tests/**/test_*.py) for pytest markers
  • Extracts setting IDs from @pytest.mark.testcov() decorators (see Tests)
  • Maps each test function to its associated test coverage marker

Step 3: Coverage Analysis

  • Matches test coverage markers from features with test coverage markers in tests
  • Identifies covered and uncovered test coverage markers
  • Calculates coverage percentages and statistics

Step 4: Report Generation

  • Produces human-readable and machine-readable coverage reports
  • Highlights gaps in test coverage
  • Provides actionable insights for improving test coverage

Usage Example ​

bash
# Generate coverage report
python tests/util/coverage.py

...
Loading feature excludes...
✓ Excluding 19 features: _bfpxe, _build, _curl, _debug, _dev, _initrdDebug, _nopkg, _oci, _readonly, _secureboot, bluefield, capi, cisPackages, container, firecracker, nodejs, python, sssd, stigDev


================================================================================
MARKER COVERAGE REPORT
================================================================================

Summary:
  Total markers:              244
  Tested markers:              28
  Untested:                   216
  Orphaned markers:            91

  Coverage:                 11.5%

  Total test functions:       160
  Tests with markers:           0
  Tests without markers:      160

Coverage by feature:
--------------------------------------------------------------------------------
⚠ _fips: 0/4 (0.0%)
⚠ _fwcfg: 0/1 (0.0%)
...

Features without markers (10):
--------------------------------------------------------------------------------
○ _ephemeral: 0/0 (N/A - no markers defined)
○ _iso: 0/0 (N/A - no markers defined)
...

Untested markers by feature:
--------------------------------------------------------------------------------

_fips:
  Total markers:      7
  Tested:             3
  Untested:           4
  Untested markers list:
    - GL-TESTCOV-_fips-config-openssh-sshd-Ciphers
    - GL-TESTCOV-_fips-config-openssh-sshd-KexAlgorithms
    - GL-TESTCOV-_fips-config-openssl
    - GL-TESTCOV-_fips-config-openssl-fipsinstall
...

================================================================================

✓ JSON report written to: tests/coverage_report.json
✓ JUnit XML report written to: tests/coverage_report.xml

Integration with Development Workflow ​

  1. During Feature Development

    • Add test coverage markers to new features (see When to Create Markers for guidelines)
    • Place markers appropriately using inline comments or split ID files (see Where to Place Markers)
    • Use coverage.py to verify markers are correctly formatted and follow the Marker Schema
    • Identify which tests need to be created

  2. During Test Development

    • Check coverage reports to find untested features
    • Write tests with appropriate @pytest.mark.testcov() decorators (see Tests)
    • Ensure test coverage markers match the exact settings from features
    • Verify test coverage increases after adding new tests

  3. In CI/CD Pipelines

    • Run coverage.py as part of automated checks
    • Enforce minimum coverage thresholds
    • Block PRs that reduce test coverage
    • Generate audit reports for compliance tracking

Marker Schema ​

All test coverage markers follow this format:

GL-TESTCOV-$feature-$category-$description

Components ​

  1. Prefix: GL-TESTCOV- (fixed)
  2. Feature: Feature directory name (e.g., aws, ssh, cis, server)
  3. Category: Type of setting (see Category Reference)
  4. Description: Hyphenated description of what the setting does

Examples ​

GL-TESTCOV-aws-service-aws-clocksource-enable
GL-TESTCOV-ssh-config-sshd-config-permissions
GL-TESTCOV-base-config-no-hostname
GL-TESTCOV-server-service-systemd-timesyncd-enable
GL-TESTCOV-cisOS-config-crontab-permissions

Category Reference ​

Garden Linux uses four main categories for tes coverage markers:

1. config - Configuration Settings ​

Purpose: Configuration files, system settings, kernel parameters, security policies

Subcategories:

PatternMeaningExample
config-$path-$settingConfiguration file contentconfig-ssh-sshd-config-permitrootlogin
config-$path-permissionsFile/directory permissionsconfig-hosts-permissions
config-$path-linkSymbolic linkconfig-locale-link
config-no-$itemFile/directory absenceconfig-no-hostname
config-$component-$parameterConfiguration parameterconfig-audit-space-left-action

Why Test This:

  • ✅ Verify configuration files are correctly rendered
  • ✅ Ensure permissions are set for security
  • ✅ Validate kernel parameters are active
  • ✅ Confirm files are removed/present as expected

Example Markers:

  • GL-TESTCOV-base-config-os-release - /etc/os-release content
  • GL-TESTCOV-server-config-hosts-permissions - /etc/hosts permissions (0644)
  • GL-TESTCOV-server-config-locale-link - /etc/default/locale → /etc/locale.conf
  • GL-TESTCOV-base-config-no-hostname - /etc/hostname is removed
  • GL-TESTCOV-cisAudit-config-audit-space-left-action - Audit daemon config parameter

2. service - Systemd Services and Timers ​

Purpose: Systemd units (services, timers, mounts, sockets)

Subcategories:

PatternMeaningExample
service-$unit-enableService is enabledservice-systemd-timesyncd-enable
service-$unit-disableService is disabledservice-systemd-timesyncd-disable
service-$unit-maskService is maskedservice-google-guest-agent-manager-mask
service-$unit-overrideOverride service configservice-systemd-timesyncd-override
service-timer-$unit-enableTimer is enabledservice-timer-aide-check-enable

Why Test This:

  • ✅ Verify services are enabled/disabled as configured
  • ✅ Ensure service states for functionality

Example Markers:

  • GL-TESTCOV-server-service-systemd-networkd-enable - systemd-networkd.service enabled
  • GL-TESTCOV-azure-service-systemd-timesyncd-disable - systemd-timesyncd disabled (Azure uses chrony)
  • GL-TESTCOV-gcp-service-google-guest-agent-manager-mask - GCP agent masked
  • GL-TESTCOV-aide-config-service-timer-aide-check-enable - AIDE check timer enabled

3. mount - Filesystem Mounts ​

Purpose: Filesystem mount points and mount options

Subcategories:

PatternMeaningExample
config-mount-$path-enableMount unit enabledconfig-mount-tmp-enable
config-mount-$pathMount configurationconfig-mount-tmp

Why Test This:

  • ✅ Ensure partitions are mounted as specified
  • ✅ Validate mount flags (nosuid, noexec, nodev)

Example Markers:

  • GL-TESTCOV-server-config-mount-tmp-enable - tmp.mount unit enabled
  • GL-TESTCOV-server-config-mount-tmp - tmp.mount unit configuration

4. script - Executable Scripts ​

Purpose: Shell scripts, hooks, and executables

Subcategories:

PatternMeaningExample
script-$name-$actionScript executionscript-clocksource-setup
script-$component-$functionScript functionalityscript-profile-autologout

Why Test This:

  • ✅ Verify scripts are present and executable
  • ✅ Ensure scripts produce expected side effects
  • ✅ Validate script configurations

Example Markers:

  • GL-TESTCOV-aws-script-clocksource-setup - AWS clocksource setup script
  • GL-TESTCOV-cloud-script-profile-autologout - Auto-logout profile script

Special Patterns ​

Overview: -no- vs -disable ​

Markers use two distinct patterns to indicate negation, each with a specific semantic meaning:

PatternMeaningSystem StateTest Verification
-no-Absence (does not exist)Completely removedCheck file/directory doesn't exist
-disableDeactivation (exists but off)Present but turned offCheck state is "disabled"

Why This Distinction Matters:

  1. Different Test Assertions: Tests verify different conditions
  2. Different Security Guarantees: Absence is stronger than deactivation
  3. Different Reversibility: Removed items need reinstallation, disabled items can be re-enabled
  4. Clear Intent: Makes it obvious whether something is missing or just turned off

The -no- Pattern (Absence/Removal) ​

Meaning: Indicates something does not exist or was completely removed

Use For:

  • Files that must not exist (e.g., security-sensitive files)
  • Directories that were cleaned up
  • Configuration sections removed from files
  • Systemd override files that don't exist

Common Patterns:

PatternMeaningExample
config-no-$fileFile does not existconfig-no-hostname
config-no-$directoryDirectory does not existconfig-no-initrd-img
service-no-$unit-overrideNo systemd override existsservice-no-systemd-timesyncd-override
config-$component-no-$featureFeature removed from configconfig-cloud-no-growpart

Why Test -no- Patterns:

  • ✅ Verify cleanup operations succeeded
  • ✅ Ensure security-sensitive files are actually removed
  • ✅ Confirm features are not present (stronger guarantee than disabled)

The -disable Pattern (Deactivation) ​

Meaning: Indicates something exists but is turned off or deactivated

Use For:

  • Services that exist but are disabled
  • Features in configuration files set to "false" or "no"
  • Kernel modules that exist but are not loaded
  • Functionality that can be re-enabled without reinstalling

Common Patterns:

PatternMeaningExample
service-$unit-disableService exists but disabledservice-systemd-timesyncd-disable
config-$component-$feature-disableFeature set to disabled in configconfig-sshd-x11forwarding-disable

Why Test -disable Patterns:

  • ✅ Verify services are in correct disabled state
  • ✅ Ensure features are turned off in configuration
  • ✅ Confirm functionality is deactivated (but can be re-enabled if needed)

When to Create Markers ​

✅ DO Create Markers For: ​

Runtime-Verifiable Settings:

  • Configuration file content and structure
  • File/directory permissions and ownership
  • File/directory presence or absence
  • Service enable/disable/mask states
  • Systemd timer states
  • Mount point configurations
  • Symbolic links
  • Kernel parameters (sysctl, cmdline)
  • PAM configurations
  • Audit rules

Why: These settings define system behavior that must be verified on a running system.

❌ DO NOT Create Markers For: ​

Package Installation:

  • Packages in pkg.include
  • Packages in pkg.exclude
  • Package availability checks

:::note If a package in pkg.include installs an important service, it makes sense to add a $service-$unit-enable marker. Currently there is no "more elegant" way as we do not explicitly enable services. This is the Debian way of doing things. :::

Why:

  • Package installation failures cause the build to fail - already validated
  • If a package is missing, dependent configurations won't work (redundant testing)
  • Testing dpkg -l | grep package adds no value
  • As discussed in ADR-0013, testing for package presence adds no value as version-independent testing should focus on functionality, not package lists

Build-Time Operations:

  • File copying during build
  • Directory creation during build
  • Archive extraction during build

Why: These are build-time operations that cannot fail silently - build failures catch them.

Where to Place Markers ​

Features ​

Inline Comments (Preferred) ​

Files: exec.config, exec.early, exec.post, exec.late, pkg.exclude, file.exclude

Format:

  1. Add markers as simple inline comments.
  2. Multiple markers can be listed in multiple lines.
bash
# GL-TESTCOV-$feature-$category1-$description1
# GL-TESTCOV-$feature-$category2-$description2
command or configuration

Examples:

bash
# features/server/exec.config

# GL-TESTCOV-server-service-systemd-networkd-enable
systemctl enable systemd-networkd

# GL-TESTCOV-server-config-group-wheel
addgroup --system wheel

# GL-TESTCOV-server-config-hosts-permissions
chmod g-w / /etc/hosts
bash
# features/base/exec.post

# GL-TESTCOV-base-config-no-hostname
rm "$rootfs/etc/hostname"

# GL-TESTCOV-base-config-resolv-conf
if [ -f "$rootfs/etc/resolv.conf" ] && [ ! -L "$rootfs/etc/resolv.conf" ]; then
    rm "$rootfs/etc/resolv.conf"
fi

file.include.ids.yaml and initrd.include.ids.yaml (For File Includes) ​

Use For: Files in file.include/ and initrd.include/ directories

Why: Markers should not appear in the final image files

Files:

  • file.include.ids.yaml - for files in file.include/ directory
  • initrd.include.ids.yaml - for files in initrd.include/ directory

Format:

yaml
# features/$feature/file.include.ids.yaml
testcov:
  "path/to/file1":
    - GL-TESTCOV-$feature-$category-$description1
  "path/to/file2":
    - GL-TESTCOV-$feature-$category-$description2

Important:

  • Paths must NOT have a leading slash
  • Paths must be quoted strings

Example:

yaml
# features/firewall/file.include.ids.yaml
testcov:
  "etc/nftables.conf":
    - GL-TESTCOV-firewall-config-nftables-conf
  "etc/nftables.d/gl-default.nft":
    - GL-TESTCOV-firewall-config-nftables-default
yaml
# features/_usi/initrd.include.ids.yaml
testcov:
  "etc/repart.d/00-efi.conf":
    - GL-TESTCOV-_usi-config-initrd-repart-efi
  "usr/bin/repart-esp-disk":
    - GL-TESTCOV-_usi-config-initrd-script-repart-esp-disk

Tests ​

Use For: Referencing test coverage markers in test functions

Format:

  1. Use the @pytest.mark.testcov() decorator on test functions
  2. Pass a list of marker strings (even for a single marker)
  3. Multiple markers can be listed for tests that verify multiple settings
python
@pytest.mark.testcov(["GL-TESTCOV-$feature-$category-$description"])
def test_something(client):
    """Test implementation"""
    # test code here

Example:

python
# tests/test_cis.py
import pytest

@pytest.mark.testcov([
    "GL-TESTCOV-cisOS-config-crontab-permissions",
    "GL-TESTCOV-cisOS-config-cron-hourly-permissions",
    "GL-TESTCOV-cisOS-config-cron-daily-permissions"
])
def test_cron_permissions(client):
    """Verify CIS-compliant permissions on cron directories"""
    cron_paths = ["/etc/crontab", "/etc/cron.hourly", "/etc/cron.daily"]
    for path in cron_paths:
        result = client.execute(f"stat -c '%a' {path}")
        assert result.stdout.strip() == "700"

The test coverage marker system is based on: