Version: 1.3.16

AI Risk Assessment Questionnaire and Policy Select

This page provides a description of the Certifai AI Risk Assessment Questionnaire and the Policy Select tool.

The Certifai AI Risk Assessment Questionnaire allows Certifai Enterprise users to defines a trust scanning policy by selecting the controls they want to apply, and configuring them (e.g. specifying the minimum score required for fairness, listing the features to be scanned for fairness).

Policy Select tool allows Certifai Enterprise users to apply a selected policy to one or more model use cases, or download the policy. When a Policy is applied, the CLI or Scan Viewer UI run a policy check by applying the selected policy against the use case scan results to create a compliance report.

Why use the Risk Assessment and Policy Select toolset

The AI Risk Assessment and Policy Select toolset works with the Certifai scanner to help you setup policies that can be used in a number of ways, including:

  • Providing direction for the selection criteria during model development
  • Identifying release criteria as part of release process
  • Monitoring compliance and detecting drift following deployment

How the Risk Assessment and Policy Select toolset

The toolset works like this:

  1. Users fill out an AI risk assessment questionnaire that may be customized for different businesses and vertical markets.

    The questionnaire provides an assessment of risk levels associated with your models and data, based on your responses to the configured questions. For example the questionnaire may cover topics like the following:

    • The project context
    • Understanding and assessing impact
    • Assessing appropriate data use risk
    • Assessing accountability risk
    • Assessing third-party methodology risk
    • Assessing risk from historic bias
    • Assessing the risk from technical bias

    Upon completion a report is produced (and may be downloaded) that provides recommended controls for AI risk assessment.

  2. Users configure their policy selections by:

  • Creating a policy (name and description)

  • Verifying the trust factors that are to be included in the policy

  • Configuring the thresholds (scores) for each of the specified trust factors

  • Setting the violation severity for compliance failures

    An "active" policy (.yaml file) is generated from the configured settings that is saved to your model use case folder.

  1. The active policy is downloaded and using the CLI command it is copied to the model use case folder where it can be accessed by the Console.

  2. Compliance reports are created, and you may opt to have them displayed in the Certifai Console.

  3. Compliance reports may be loaded into your integrated BI tool.