Version: 1.3.14

Preflight Scan Analysis

This page provides a description and examples of the preflight scan analysis feature.

Description

The preflight scan can be run for local scans using the CLI or notebooks to go beyond simple validation of the scan definition.

The preflight scan provides:

  • Warnings about potential issues with the scan
  • Errors about unexpected events that will prevent the scan from running
  • Observations, such as estimated scan times by model

The preflight check takes approximately 2-5 minutes/model for simple models and preflight scan progress is displayed in the terminal.

Run a preflight check

To run a preflight check on any local scan append the scan command with --preflight.

$ certifai scan -f certifai_examples/examples/definitions/german_credit_scanner_definition.yaml --preflight

Scan output

After you initiate the scan, progress of the preflight check is displayed as each model scan concludes.

Starting Preflight Scan
[--------------------] 2020-08-21 09:52:14.081083 - 0 of 16 checks (0.0% complete) - Running model_nondeterminism preflight check for model svm
[#-------------------] 2020-08-21 09:52:18.848352 - 1 of 16 checks (6.25% complete) - Running unknown_outcome_class preflight check for model svm
[##------------------] 2020-08-21 09:52:18.988502 - 2 of 16 checks (12.5% complete) - Running scan_time_estimate preflight check for model svm
2020-08-21 09:52:18,991 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)
[###-----------------] 2020-08-21 09:52:41.595360 - 3 of 16 checks (18.75% complete) - Running fairness_class_samples preflight check for model svm
[#####---------------] 2020-08-21 09:52:41.600087 - 4 of 16 checks (25.0% complete) - Finished all preflight checks for model svm
[#####---------------] 2020-08-21 09:52:41.600161 - 4 of 16 checks (25.0% complete) - Running model_nondeterminism preflight check for model logit
[######--------------] 2020-08-21 09:52:46.330665 - 5 of 16 checks (31.25% complete) - Running unknown_outcome_class preflight check for model logit
[#######-------------] 2020-08-21 09:52:46.462958 - 6 of 16 checks (37.5% complete) - Running scan_time_estimate preflight check for model logit
2020-08-21 09:52:46,466 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)
[########------------] 2020-08-21 09:53:00.454901 - 7 of 16 checks (43.75% complete) - Running fairness_class_samples preflight check for model logit
[##########----------] 2020-08-21 09:53:00.459405 - 8 of 16 checks (50.0% complete) - Finished all preflight checks for model logit
[##########----------] 2020-08-21 09:53:00.459474 - 8 of 16 checks (50.0% complete) - Running model_nondeterminism preflight check for model dtree
[###########---------] 2020-08-21 09:53:05.211731 - 9 of 16 checks (56.25% complete) - Running unknown_outcome_class preflight check for model dtree
[############--------] 2020-08-21 09:53:05.340272 - 10 of 16 checks (62.5% complete) - Running scan_time_estimate preflight check for model dtree
2020-08-21 09:53:05,342 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)
[#############-------] 2020-08-21 09:53:16.939798 - 11 of 16 checks (68.75% complete) - Running fairness_class_samples preflight check for model dtree
[###############-----] 2020-08-21 09:53:16.944578 - 12 of 16 checks (75.0% complete) - Finished all preflight checks for model dtree
[###############-----] 2020-08-21 09:53:16.944660 - 12 of 16 checks (75.0% complete) - Running model_nondeterminism preflight check for model mlp
[################----] 2020-08-21 09:53:21.915100 - 13 of 16 checks (81.25% complete) - Running unknown_outcome_class preflight check for model mlp
[#################---] 2020-08-21 09:53:22.063400 - 14 of 16 checks (87.5% complete) - Running scan_time_estimate preflight check for model mlp
2020-08-21 09:53:22,065 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)
[##################--] 2020-08-21 09:53:35.914853 - 15 of 16 checks (93.75% complete) - Running fairness_class_samples preflight check for model mlp
[####################] 2020-08-21 09:53:35.919279 - 16 of 16 checks (100.0% complete) - Finished all preflight checks for model mlp
Preflight Scan Completed

When all preflight checks are complete, success/failure of preflight checks per model are displayed, as well as errors (not in the following example), warnings, and observations.

====== Preflight Summary ======
Total number of results: 40
Number of observations: 28
Number of warnings: 12
Number of errors: 0
Observations for svm model:
Passed model non determinism check
Passed unknown outcome classes check
Expected time for robustness analysis is 12 seconds
Expected time for fairness analysis is 12 seconds
Expected time for explanation analysis is 28 seconds
Expected time for explainability analysis is 12 seconds
Expected time for performance analysis is 10 seconds
Warnings for svm model:
No counterfactual explanation can be found approximately 92% of rows
Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples)
Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)
Observations for logit model:
Passed model non determinism check
Passed unknown outcome classes check
Expected time for robustness analysis is 11 seconds
Expected time for fairness analysis is 11 seconds
Expected time for explanation analysis is 21 seconds
Expected time for explainability analysis is 11 seconds
Expected time for performance analysis is 10 seconds
Warnings for logit model:
No counterfactual explanation can be found approximately 92% of rows
Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples)
Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)
Observations for dtree model:
Passed model non determinism check
Passed unknown outcome classes check
Expected time for robustness analysis is 11 seconds
Expected time for fairness analysis is 11 seconds
Expected time for explanation analysis is 19 seconds
Expected time for explainability analysis is 11 seconds
Expected time for performance analysis is 10 seconds
Warnings for dtree model:
No counterfactual explanation can be found approximately 92% of rows
Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples)
Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)
Observations for mlp model:
Passed model non determinism check
Passed unknown outcome classes check
Expected time for robustness analysis is 11 seconds
Expected time for fairness analysis is 11 seconds
Expected time for explanation analysis is 21 seconds
Expected time for explainability analysis is 11 seconds
Expected time for performance analysis is 10 seconds
Warnings for mlp model:
No counterfactual explanation can be found approximately 92% of rows
Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples)
Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)