Validation setup

Generalities

Setup files

Like profiles, setup configurations have nodes to describe different steps of the process. These nodes are split into subnodes to describe the course of the run.

The validation configuration is specified using setup files. These files can be in the yml format (pcvs.yml), or be an executable files (pcvs.setup) generating a yml configuration in stdout. The information of this configuration are crossed with the profile information to run the tests.

When PCVS is launched in a directory, it browses every subdirectory to find any pcvs.yml or pcvs.setup file and launches itself with the corresponding configuration.

Example

exampletree/
├── subdir1
│   └── pcvs.yml
└── subdir2
    └── pcvs.yml

Launching `pcvs run exampletree will generate tests for subdir1/pcvs.yml and for subdir2/pcvs.yml. There is no need to put a setup configuration in the root of exampletree, but it is possible to add a setup here.

Structure

The yml input must have one node per test. Each test can describe the following configurations :

build
run
validate
group
tag
artifact

Build

The build node describe how a binary file should be built depending on its sources. It contains the following subndoes :

build:
    files: path/to/the/file/to/build
    sources:
        binary: name of the binary to be built (if necessary)
        cflags: extra cflags
        ldflags: extra ldflags
    depends_on: ["list of test names it depends on"]
    cwd: directory where the binary should be built
    variants: [list of variants (CF Configuration basic blocks -> compiler
    node)]

    autotools:
        params: [list of options for autotools]
    cmake:
        params: [list of options for cmake]
    make:
        target: target for make command
        jobs: make -j n option

Run

The run node describes how a binary file should be launched. It contains the following nodes :

run:
    cwd: path to build directory
    depends_on: [list of tests on which it depends]
    package_manager:
        spack: [list of spack dependencies used by this test]
        module: [list of installed modules this test needs]
    program: name of the binary file

The run node owns the iterate subnode which can contain custom iterators described in the criterion node in the selected profile. Moreover, the run.iterate node can define custom iterators without defining them in criterion by writing them in the run.iterate.program node.

run:
    iterate:
        iterator_described_in_'profile.runtime.criterion':
            values: [list of values for the corresponding iterator]
        program:
            custom_iterator:
                numeric: true/false
                type: "argument" or "environment"
                values: [list of values taken by the iterator]
                subtitle: string chosen to identify this iterator

Validate

The validate node describes the expected test behaviour, including exit, time and matching output.

validate:
    expect_exit: expected exit code (integer)
    time:
        mean: expected time to compute the test (seconds / float) tolerance:
        standard deviation for expected time (seconds / float)
        hard_timeout: maximum time after which process has to be killed (seconds / float)
        soft_timeout: maximum time after which the test is considered failed (but can still finish, so you can run test correctness locally without testing optimisation (even on your slow computer))
    match:
        label:
            expr:
            expect:
    script:
        path: Path to a validating script
    method: name of the function in the plugin you  are using
        args: list of arguments for thefunction (see example below)
    method: 'not_longer_than_previous_runs' # check previous run in database
        args:
            history_depth: -1 # look at n previous run (-1 for all)
            tolerance: 10 # time can be 10% slower than the fastest of all previous runs

Group

Groups are described in profiles. They can contain build, run, tag, validate, and artifact subnodes. Once a group is defined in the used profile it can be called in the validation setup file.

group: name of the group defined in the profile

Tag

Tags get in the results and tests can be sorted tag-wise. A test can have multiple tags and tags do not have to be defined upstream.

tag:
    - tag1
    - tag2

Artifact

The artifact node contains anything the output should have in addition to the results of tests.

artifact:
    obj1: "path/to/obj1"
    obj2: "path/to/obj2"

Setup file

When using a pcvs.setup file, the file will be executed, and stdout should be a valid yaml file that respect pcvs.yml file structure as describe above.

When running the script, some environment variables are availables to ease configuration. The complete list of environment variables can be obtained by running pcvs -vvv run ....

The list is composed of:

Compilers environment variables starting with: PCVS_CMP.
Criterions environment variables starting with: PCVS_CRIT.
Other environment variables set in profile at compiler.compilers.<x>.envs.

Example of availables environment variables: (will depends on your profile file)

PCVS_CMP_CC='gcc'
PCVS_CMP_CC_ARGS=''
PCVS_CMP_CC_VAR_OPENMP='gcc'
PCVS_CMP_CC_VAR_OPENMP_ARGS='-fopenmp'
PCVS_CMP_CUDA='nvcc'
PCVS_CMP_CUDA_ARGS=''
PCVS_CMP_CXX='g++'
PCVS_CMP_CXX_ARGS=''
PCVS_CMP_CXX_VAR_OPENMP='g++'
PCVS_CMP_CXX_VAR_OPENMP_ARGS='-fopenmp'
PCVS_CMP_FC='gfortran'
PCVS_CMP_FC_ARGS=''
PCVS_CMP_FC_VAR_OPENMP='gfortran'
PCVS_CMP_FC_VAR_OPENMP_ARGS='-fopenmp'
PCVS_CMP_HIP='hipcc'
PCVS_CMP_HIP_ARGS=''

PCVS_CRIT_N_CORE='1'
PCVS_CRIT_N_MPI='1 2 4'
PCVS_CRIT_N_NODE='1 2'
PCVS_CRIT_N_OMP='1 4'
PCVS_CRIT_N_PROC='1 4 8'

PCVS_CC='gcc'
PCVS_CXX='g++'
PCVS_FC='gfortran'
PCVS_HIPCC='hipcc'
PCVS_NVCC='nvcc'