Manage remote scans
Prerequisites
- You have a remote installation (Certifai Pro or Enterprise) on one of the following platforms:
- You have imported the configuration into your Certifai toolkit.
CLI Commands
Certifai CLI management commands and parameters are defined in the CLI.
- Make sure you are in an active conda environment.
- Enter
certifai remote -h
to view a list of the available commands for managing Certifai in your remote environment. - To view the parameters and other options for a command by running
certifai remote <command> -h
.
Running a remote scan
See the corresponding instructions for running a scan for your remote installation type (Certifai Pro or Enterprise):
NOTE: Scan Definitions and Datasets in remote storage
Datasets must be located in a remote storage location accessible, with the storage credentials that were configured for your installation, in order to be used for a remote scan. However, scan definitions can either exist on your local drive or in a remote storage location accessible from your cluster.
Specifying an output path
The -o
flag scan be used to specify an output path when you run a remote scan, similar to when you run a local scan.
If the -o
flag is not specified, the following pathways are used for storing scan results in the order presented:
1. The Scan Directory that the Certifai Console is configured to read from in your remote installation
2. The output path specified within the scan definition
Regardless of how the output report path is specified for your remote scan, the remote storage location must be accessible with the storage credentials that were configured for the Operator.
Parallel Scan Execution
Remote scans can be executed using a parallel execution feature. This feature allows the user to submit a scan definition that contains many models and evaluation types, and have those reports generated in parallel. Each report for each model is generated independently. When these parallel scans are executed on a remote cluster that is made of up several compute nodes, this can result in improved completion times for scans that have many models and report types.
To execute a parallel scan, simply add the -p
parameter to the certifai remote scan
command. In addition to specifying -p
, you must specify a concurrency value, which is a number greater than 1. This concurrency value determines how many simultaneous reports are generated. For example:
certifai remote scan -p 4 -f ./scan_definition.yaml
This example tells the scanner to execute the scan in parallel, with a maximum of 4 simultaneous reports.
Concurrency Value
The value specified for the -p
parameter is dependent on how the target cluster is configured. If the target cluster has, for example,10 compute nodes, and you have a scan that contains 20 reports, specifying a value of -p 10
might be reasonable. This tells the scan controller
to utilize up to 10 simultaneous jobs for evaluating reports. Depending on how the Kubernetes cluster is configured, the Kubernetes runtime will
schedule these jobs optimally based on the available resources in the cluster.
Note also that in the case where a remote scan is executed on a cluster that has only one physical node, with limited CPU capability, (i.e.: a VM with 1 virtual CPU), a parallel scan is probably not going to be as fast to complete as running a non-parallel scan.
For more details: certifai remote scan -h
Required CPU and Memory
The following parameters can be added to a remote scan request to override operator settings to optimize pod resources for the scan.
The units for these values should correspond to Kubernetes resource unit conventions
NOTE
Run certifai remote scan -h
for help text about the limits for cpu and memory requests so a scan can be scheduled onto a node. You cannot request more resources than are available on each node in a cluster to ensure scans get scheduled and run. If you request too many resources, scans hang in a pending state.
--cpu-req <value>
Specifies cpu cores to be requested for each concurrent scanner pod. The value cannot exceed the amount of CPU cores available on each node of your cluster.
Default value: 1000m
--mem-req <value>
Specifies memory to be requested for each concurrent scanner pod. It is recommended this value not exceed 70-80 percent of the memory available on each node of your cluster.
Default Value: 500Mi
Example
certifai remote scan -f def.yaml --cpu-req 800m --mem-req 900Mi
When invalid values are provided for memory and CPU, an error message is displayed.
List remote scan jobs
To view a list of jobs that have been run on your Certifai installation run:
certifai remote list
Outputs job list:
Scan ID Total Reports Completed Reports Failed Reports Active Reports Progress State Scan Duration 49ae9469c1d5 24 24 0 0 100 Complete 4m26s 5f28fc373d8a 24 24 0 0 100 Complete 4m40s b5104a9d37fc 24 24 0 0 100 Complete 4m47s c199b1b8bab5 24 24 0 0 100 Complete 4m26s f5828a52c1b8 24 24 0 0 100 Complete 4m24s
NOTE:
- Deleted jobs are not listed.
Certifai Remote Scan Describer
The remote scan describer prints the definition or status of a scan. You must provide the Scan ID for the scan you wish to describe.
Usage: certifai remote describe <scan id>
Output:
(certifai) CS-0245:certifai_toolkit_1.3.1 smichalski$ certifai remote describe 49ae9469c1d5Scan ID Total Reports Completed Reports Failed Reports Active Reports Progress State Scan Duration 49ae9469c1d5 24 24 0 0 100 Complete 4m26s
For a full description of a Kubernetes job, add the --full
option.
Certifai Remote Scan Logger
The remote scan logger prints the logs for a scan. You must provide the Scan ID for the scan you wish to show logs for.
Usage: certifai remote logs <scan id>
To view the last <num> lines of the logs for a job, add the -t <num>
option to the logs command.
Certifai Remote Scan Deleter
The remote scan deleter removes all resources created for a scan. You must provide the Scan ID for the scan you wish to delete.
Usage: certifai remote delete <scan id>
Troubleshooting
Errors may occur while using the certifai remote [cmd]
commands. This section identifies some of them and provides instructions for how to resolve them.
Additional troubleshooting can be found at Remote Config Import
Alias not found
Failed to describe job - Alias "<-a option value>" does not exist in Certifai remote config
The alias set for use with the command does not exist in the Certifai remote configuration. Ensure that it is spelled correctly and try to import the configuration again: Remote Config Import
Job not found
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"<-n option>\" not found","reason":"NotFound","details":{"name":"<-n option>","group":"batch","kind":"jobs"},"code":404}
The job name set with the -n
option was not found on the remote installation. Ensure the job exists by running the
certifai remote list
command.