Version: 1.3.16

Risk and Policy Tool Basic Use

This page provides a description of how to use the Certifai AI Risk Assessment Questionnaire and the Policy Select tool.

For general information about what the Certifai AI Risk Assessment Questionnaire and the Policy Select tool is and what it does, visit the AI Risk Assessment Questionnaire and Policy Select information page.

Assess Risk and Configure Policy

Prerequisites:

AI Risk and Policy Select tool must be installed in your Enterprise cluster. Contact your CognitiveScale representative to obtain this toolset.

Steps

  1. Open the AI Risk Assessment and Policy Select interface, and click Assess Your Risk to begin.

    AI-Risk-Assessment-and-Policy-Select-interface

  2. Fill out an AI risk assessment questionnaire.

    The questionnaire provides an assessment of risk levels associated with your models and data, based on your responses to the configured questions. For example the questionnaire may cover topics like the following:

    • The project context

    • Understanding and assessing impact

    • Assessing appropriate data use risk

    • Assessing accountability risk

    • Assessing third-party methodology risk

    • Assessing risk from historic bias

    • Assessing the risk from technical bias

      Risk-questionnaire-heat-map

    Upon completion a report is produced (and may be downloaded) that provides recommended controls for AI risk assessment.

  3. Configure your Policy selections by:

  • Creating a policy by entering a name and description.
  • Verifying/selecting the trust factors that are to be included in the policy by checking or unchecking trust factors in the list.
  • Configuring the thresholds for each of the specified trust factors by entering the score that designates a model scan must receive to "pass". (Scans below the designated score "fail" the policy check.)
  • Setting the violation severity for compliance failures by moving the slider.
  1. Download the "active" policy (.yaml file) that is generated from the configured settings by clicking the Download button on the Policy summary page.

    Policy-summary-page

  2. Run the CLI command to make the policy.yaml accessible to Certifai.

    Example

    certifai policy-load -f ~/Downloads/policy-definition.yaml -o ~/work/github/certifai/certifai_examples/examples/reports -u c12e_datasciencelab_german_credit

    Where

    • -f is where the policy definition file has been downloaded from the tool where it was created.
    • -o is the path to the reports directory.
    • -u is the name of the use case that you want to evaluate.

    A success message is displayed that says the policy-definition.yaml will be moved into examples/reports/<model-use-case> folder (c12e_datasciencelab_german_credit in this case)

  3. Open your Certifai Console, or if it is already open, refresh the browser, and the click the Check Policy option from the Scan List.

    Console-check-policy-option-in-scan-list

  4. The loaded policy is evaluated (policy check) and a compliance report is generated for the scan and displayed in Certifai Console

    Console-compliance-report

  5. Compliance reports may be loaded into your integrated BI tool.

Additional CLI commands

Run the policy check from the CLI.

Example:

certifai policy-check -u c12e_datasciencelab_german_credit -s c46s22b4ece7 -o ~/work/github/certifai/certifai_examples/examples/reports

Where:

  • -u is the use case that you want to evaluate.
  • -s is the id of the scan you want to check.
  • -o is the path to the reports directory.

Output: Policy evaluation results

Validate the policy-definition schema

Example:

certifai policy-validate -f ~/Downloads/policy-definition.yaml

Where:

  • -f is where the policy definition file has been downloaded from the tool where it was created.

Output: Policy definition validation succeeded (or failed)

Control Customization

Based on your use case, you may customize the policy control file when it's stored in the cloud. The location of the policy-control-config-file.yaml file is set in the Certifai CR. If no location is set, the default policy control is loaded.

To configure the cloud location of the policy go to the policy section of Certifai CR and add:

policy-control-config-file: s3://<path-to-policy-control-config-file.yaml>