Version: 1.3.15

Run sample scans

Prerequisites

  1. Download the Certifai toolkit
  2. Install the Certifai CLI
  3. Install the Certifai Reference Models

Run Reference Model server

To run a scan, you must first make one or more model implementations accessible to Certifai via REST endpoints.

  1. In another terminal, activate the certifai-reference-models conda environment

    conda activate certifai-reference-models
  2. Run the certifai reference model server by doing:

    startCertifaiModelServer

    If you are asked to allow an external connection, click 'Deny' as the models only need to be available locally.

    The output is similar to the example below:

    Adding route: /heart_disease_rf/predict using function certifaiReferenceModelServer.healthcare_heart_disease_prediction.randomForest.predict:predict_heart_disease_rf

Run a use case scan

  1. Leave the reference model server running in its own terminal.

  2. In another terminal, activate the certifai conda environment that you created when you installed the toolkit.

    conda activate certifai
  3. Go to the folder where you unzipped the toolkit, so you can use the provided examples. Run the certifai command specifying the scan definition file and an output path. (The following use case example scans four models that predict whether a person has type 2 diabetes.)

    ```
    certifai scan -f examples/definitions/diabetes_scanner_definition.yaml -o ./reports
    ```
    The scan may take 10 or more minutes to complete, depending on the CPU available on your system.
  4. When the scan completes, output similar to the following is displayed.

    ...
    Scan Completed
    ====== Report Summary ======
    Total number of evaluations performed: 20
    Number of successful reports: 20
    Number of failed reports: 0

    The reports are written into the ./reports folder as specified in the -o option. Reports are grouped into folders based on the model_use_case_id specified in the definition file. For example, the reports generated by the diabetes use case scan are in the ./reports/c12e_datascience_diabetes folder.

    In general scan results include up to 6 report files created for each model implementation:

    • robustness
    • fairness
    • explainability
    • explanations
    • performance
    • atx (AI Trust Index)

    The diabetes use case does not include a fairness report, so only 20 reports, 5 per model, are displayed. In addition to the reports, a usecase.yaml file that contains information about the scans model use case is written to the ./reports/c12e_datascience_diabetes/ folder.

Run a scan for a single use case model

Run a scan for a single model using the -m flag.

The following command runs a scan for the model with an id of svm in the definition file.

certifai scan -f examples/definitions/diabetes_scanner_definition.yaml -m svm -o ./reports

View scan results

  1. Start a local Certifai Console from the folder where you unzipped the toolkit (certifai/). Specify the ./reports output folder.

    certifai console ./reports
  2. The Console is available at http://localhost:8000. Copy this URL into a browser to view your scan result visualizations.