Risk and Policy Tool Basic Use
This page provides a description of how to use the Certifai AI Risk Assessment Questionnaire and the Policy Select tool.
For general information about what the Certifai AI Risk Assessment Questionnaire and the Policy Select tool is and what it does, visit the AI Risk Assessment Questionnaire and Policy Select information page.
Assess Risk and Configure Policy
Prerequisites:
AI Risk and Policy Select tool must be installed in your Enterprise cluster. Contact your CognitiveScale representative to obtain this toolset.
Steps
Open the AI Risk Assessment and Policy Select interface, and click
Assess Your Risk
to begin.Fill out an AI risk assessment questionnaire.
The questionnaire provides an assessment of risk levels associated with your models and data, based on your responses to the configured questions. For example the questionnaire may cover topics like the following:
The project context
Understanding and assessing impact
Assessing appropriate data use risk
Assessing accountability risk
Assessing third-party methodology risk
Assessing risk from historic bias
Assessing the risk from technical bias
Upon completion a report is produced (and may be downloaded) that provides recommended controls for AI risk assessment.
Configure your Policy selections by:
- Creating a policy by entering a name and description.
- Verifying/selecting the trust factors that are to be included in the policy by checking or unchecking trust factors in the list.
- Configuring the thresholds for each of the specified trust factors by entering the score that designates a model scan must receive to "pass". (Scans below the designated score "fail" the policy check.)
- Setting the violation severity for compliance failures by moving the slider.
Download the "active" policy (.yaml file) that is generated from the configured settings by clicking the
Download
button on the Policy summary page.Run the CLI command to make the
policy.yaml
accessible to Certifai.Example
certifai policy-load -f ~/Downloads/policy-definition.yaml -o ~/work/github/certifai/certifai_examples/examples/reports -u c12e_datasciencelab_german_creditWhere
-f
is where the policy definition file has been downloaded from the tool where it was created.-o
is the path to the reports directory.-u
is the name of the use case that you want to evaluate.
A success message is displayed that says the policy-definition.yaml will be moved into
examples/reports/<model-use-case>
folder (c12e_datasciencelab_german_credit
in this case)Open your Certifai Console, or if it is already open, refresh the browser, and the click the
Check Policy
option from the Scan List.The loaded policy is evaluated (policy check) and a compliance report is generated for the scan and displayed in Certifai Console
Compliance reports may be loaded into your integrated BI tool.
Additional CLI commands
Run the policy check from the CLI.
Example:
certifai policy-check -u c12e_datasciencelab_german_credit -s c46s22b4ece7 -o ~/work/github/certifai/certifai_examples/examples/reports
Where:
-u
is the use case that you want to evaluate.-s
is the id of the scan you want to check.-o
is the path to the reports directory.
Output: Policy evaluation results
Validate the policy-definition schema
Example:
certifai policy-validate -f ~/Downloads/policy-definition.yaml
Where:
-f
is where the policy definition file has been downloaded from the tool where it was created.
Output:
Policy definition validation succeeded
(or failed
)
Control Customization
Based on your use case, you may customize the policy control file when it's stored in the cloud. The location of the policy-control-config-file.yaml
file is set in the Certifai CR. If no location is set, the default policy control is loaded.
To configure the cloud location of the policy go to the policy
section of Certifai CR and add:
policy-control-config-file: s3://<path-to-policy-control-config-file.yaml>
NOTE
The Policy Control file must be a yaml
file.