Run a scan using the Reference Model Server
Follow the instructions below to learn how to install and use the Reference Model Server to run a scan and view the visualizations in Console.
Prerequisites
- You have accepted the license agreement and downloaded the Toolkit .zip file to your local system.
- You have completed the Install Toolkit and View Example Reports tutorial.
NOTE: Windows users must run PowerShell as an administrator and Conda is required to install Toolkit.
Tutorial instructions
- MAC-LINUX
- Windows PowerShell
Set your context to the folder where your Certifai Toolkit was unzipped.
cd <toolkit-location>Create virtual environment for your Reference Model Server. (NOTE: This is different from the CLI environment you created in the Install Toolkit and View Example Reports tutorial.)
conda create -n certifai-reference-models python=<version #>Example:
conda create -n certifai-reference-models python=3.8Activate the Reference Model Server environment.
conda activate certifai-reference-modelsInstall the reference model package. (This may take a couple of minutes to run.)
pip install reference_models/packages/*Start the Reference Model Server and leave it running in an open terminal window.
startCertifaiModelServer- If you are asked to allow an external connection, click 'Deny' as the models only needs to be available locally.
- The output is similar to the example below:
Adding route: /heart_disease_rf/predict using function certifaiReferenceModelServer.healthcare_heart_disease_prediction.randomForest.predict:predict_heart_disease_rfInfo
Your reference model server is running locally on port 5111. Each model has its own endpoint. For example, the Random Forest Heart Disease model is at
http://127.0.0.1:5111/heart_disease_rf/predict
.For this tutorial, the models defined in the scan definition .yaml use endpoints provided by the Reference Model Server, so the server must be running in order for the scan to complete successfully.
Open a new terminal window and set your context to the folder where you unzipped your Certifai Toolkit.
cd <toolkit-location>Activate Certifai CLI environment.
conda activate certifaiRun the following command which specifies the example scan definition file for this tutorial,
examples/definitions/diabetes_scanner_definition.yaml
and sets the output path for the scan report files. This path will be used by the Certifai Console to read the reports in step 9 below.The reports are written into the
./reports
folder as specified in the-o
option. Reports are grouped into folders based on the model_use_case_id specified in the definition file. For example The reports generated by the diabetes use case scan are in the./reports/c12e_datascience_diabetes
folder.In general scan results include up to 6 report files created for each model implementation:
The diabetes use case does not include a fairness report, so only 20 reports, 5 per model, are displayed. In addition to the reports, a
usecase.yaml
file that contains information about the scans model use case is written to the./reports/c12e_datascience_diabetes/
folder.certifai scan -f examples/definitions/diabetes_scanner_definition.yaml -o ./reportsNOTE: The scan may take 10 or more minutes to complete, depending on the CPU available on your system. Output similar to the following is displayed when the reports are completed successfully:
...Scan Completed====== Report Summary ======Total number of evaluations performed: 20Number of successful reports: 20Number of failed reports: 0You may optionally run a scan for a single use case model by specifying the model using
-m
argument in the command.Run the local Certifai Console and point to the reports location specified in step 8 above.
certifai console ./reportsThe Console is available at http://localhost:8000. Copy this URL into a browser to view your scan result visualizations.
Learn more about navigating through the Console pages.