Run sample scans
Prerequisites
Run Reference Model server
To run a scan, you must first make one or more model implementations accessible to Certifai via REST endpoints.
In another terminal, activate the
certifai-reference-models
conda environmentconda activate certifai-reference-modelsRun the certifai reference model server by doing:
startCertifaiModelServerIf you are asked to allow an external connection, click 'Deny' as the models only need to be available locally.
The output is similar to the example below:
Adding route: /heart_disease_rf/predict using function certifaiReferenceModelServer.healthcare_heart_disease_prediction.randomForest.predict:predict_heart_disease_rfInfo
Your reference model server is running locally on port 5111. Each model has its own endpoint. For example, the Random Forest Heart Disease model is at
http://127.0.0.1:5111/heart_disease_rf/predict
.
Run a use case scan
Leave the reference model server running in its own terminal.
In another terminal, activate the
certifai
conda environment that you created when you installed the toolkit.conda activate certifaiGo to the folder where you unzipped the toolkit, so you can use the provided examples. Run the
certifai
command specifying the scan definition file and an output path. (The following use case example scans four models that predict whether a person has type 2 diabetes.)```certifai scan -f examples/definitions/diabetes_scanner_definition.yaml -o ./reports```The scan may take 10 or more minutes to complete, depending on the CPU available on your system.When the scan completes, output similar to the following is displayed.
...Scan Completed====== Report Summary ======Total number of evaluations performed: 20Number of successful reports: 20Number of failed reports: 0The reports are written into the
./reports
folder as specified in the-o
option. Reports are grouped into folders based on themodel_use_case_id
specified in the definition file. For example, the reports generated by the diabetes use case scan are in the./reports/c12e_datascience_diabetes
folder.In general scan results include up to 6 report files created for each model implementation:
- robustness
- fairness
- explainability
- explanations
- performance
- atx (AI Trust Index)
The diabetes use case does not include a fairness report, so only 20 reports, 5 per model, are displayed. In addition to the reports, a
usecase.yaml
file that contains information about the scans model use case is written to the./reports/c12e_datascience_diabetes/
folder.
Run a scan for a single use case model
Run a scan for a single model using the -m
flag.
The following command runs a scan for the model with an id
of svm
in the definition file.
certifai scan -f examples/definitions/diabetes_scanner_definition.yaml -m svm -o ./reports
View scan results
Start a local Certifai Console from the folder where you unzipped the toolkit (
certifai/
). Specify the./reports
output folder.certifai console ./reportsThe Console is available at
http://localhost:8000
. Copy this URL into a browser to view your scan result visualizations.