AI Risk Assessment Questionnaire and Policy Select
This page provides a description of the Certifai AI Risk Assessment Questionnaire and the Policy Select tool.
The Certifai AI Risk Assessment Questionnaire allows Certifai Enterprise users to defines a trust scanning policy by selecting the controls they want to apply, and configuring them (e.g. specifying the minimum score required for fairness, listing the features to be scanned for fairness).
Policy Select tool allows Certifai Enterprise users to apply a selected policy to one or more model use cases, or download the policy. When a Policy is applied, the CLI or Scan Viewer UI run a policy check by applying the selected policy against the use case scan results to create a compliance report.
Why use the Risk Assessment and Policy Select toolset
The AI Risk Assessment and Policy Select toolset works with the Certifai scanner to help you setup policies that can be used in a number of ways, including:
- Providing direction for the selection criteria during model development
- Identifying release criteria as part of release process
- Monitoring compliance and detecting drift following deployment
How the Risk Assessment and Policy Select toolset
The toolset works like this:
Users fill out an AI risk assessment questionnaire that may be customized for different businesses and vertical markets.
The questionnaire provides an assessment of risk levels associated with your models and data, based on your responses to the configured questions. For example the questionnaire may cover topics like the following:
- The project context
- Understanding and assessing impact
- Assessing appropriate data use risk
- Assessing accountability risk
- Assessing third-party methodology risk
- Assessing risk from historic bias
- Assessing the risk from technical bias
Upon completion a report is produced (and may be downloaded) that provides recommended controls for AI risk assessment.
Users configure their policy selections by:
Creating a policy (name and description)
Verifying the trust factors that are to be included in the policy
Configuring the thresholds (scores) for each of the specified trust factors
Setting the violation severity for compliance failures
An "active" policy (.yaml file) is generated from the configured settings that is saved to your model use case folder.
The active policy is downloaded and using the CLI command it is copied to the model use case folder where it can be accessed by the Console.
Compliance reports are created, and you may opt to have them displayed in the Certifai Console.
Compliance reports may be loaded into your integrated BI tool.