Version: 1.3.14.1

Run a scan using the Reference Model Server

Follow the instructions below to learn how to install and use the Reference Model Server to run a scan and view the visualizations in Console.

Prerequisites

  1. You have accepted the license agreement and downloaded the Toolkit .zip file to your local system.
  2. You have completed the Install Toolkit and View Example Reports tutorial.

NOTE: Windows users must run PowerShell as an administrator and Conda is required to install Toolkit.

Tutorial instructions

  1. Set your context to the folder where your Certifai Toolkit was unzipped.

    cd <toolkit-location>
  2. Create virtual environment for your Reference Model Server. (NOTE: This is different from the CLI environment you created in the Install Toolkit and View Example Reports tutorial.)

    conda create -n certifai-reference-models python=<version #>

    Example:

    conda create -n certifai-reference-models python=3.6
  3. Activate the Reference Model Server environment.

    conda activate certifai-reference-models
  4. Install the reference model package. (This may take a couple of minutes to run.)

    pip install reference_models/packages/*
  5. Start the Reference Model Server and leave it running in an open terminal window.

    startCertifaiModelServer
    • If you are asked to allow an external connection, click 'Deny' as the models only needs to be available locally.
    • The output is similar to the example below:
    Adding route: /heart_disease_rf/predict using function certifaiReferenceModelServer.healthcare_heart_disease_prediction.randomForest.predict:predict_heart_disease_rf
  6. Open a new terminal window and set your context to the folder where you unzipped your Certifai Toolkit.

    cd <toolkit-location>
  7. Activate Certifai CLI environment.

    conda activate certifai
  8. Run the following command which specifies the example scan definition file for this tutorial, examples/definitions/diabetes_scanner_definition.yaml and sets the output path for the scan report files. This path will be used by the Certifai Console to read the reports in step 9 below.

    The reports are written into the ./reports folder as specified in the -o option. Reports are grouped into folders based on the model_use_case_id specified in the definition file. For example The reports generated by the diabetes use case scan are in the ./reports/c12e_datascience_diabetes folder.

    In general scan results include up to 6 report files created for each model implementation:

    The diabetes use case does not include a fairness report, so only 20 reports, 5 per model, are displayed. In addition to the reports, a usecase.yaml file that contains information about the scans model use case is written to the ./reports/c12e_datascience_diabetes/ folder.

    certifai scan -f examples/definitions/diabetes_scanner_definition.yaml -o ./reports

    NOTE: The scan may take 10 or more minutes to complete, depending on the CPU available on your system. Output similar to the following is displayed when the reports are completed successfully:

    ...
    Scan Completed
    ====== Report Summary ======
    Total number of evaluations performed: 20
    Number of successful reports: 20
    Number of failed reports: 0

    You may optionally run a scan for a single use case model by specifying the model using -m argument in the command.

  9. Run the local Certifai Console and point to the reports location specified in step 8 above.

    certifai console ./reports
  10. The Console is available at http://localhost:8000. Copy this URL into a browser to view your scan result visualizations.

Learn more about navigating through the Console pages.