Preflight Scan Analysis
This page provides a description and examples of the preflight scan analysis feature.
Description
The preflight scan can be run for local scans using the CLI or notebooks to go beyond simple validation of the scan definition.
The preflight scan provides:
- Warnings about potential issues with the scan
- Errors about unexpected events that will prevent the scan from running
- Observations, such as estimated scan times by model
The preflight check takes approximately 2-5 minutes/model for simple models and preflight scan progress is displayed in the terminal.
Run a preflight check
To run a preflight check on any local scan append the scan command with --preflight
.
$ certifai scan -f certifai_examples/examples/definitions/german_credit_scanner_definition.yaml --preflight
Scan output
After you initiate the scan, progress of the preflight check is displayed as each model scan concludes.
Starting Preflight Scan[--------------------] 2020-08-21 09:52:14.081083 - 0 of 16 checks (0.0% complete) - Running model_nondeterminism preflight check for model svm[#-------------------] 2020-08-21 09:52:18.848352 - 1 of 16 checks (6.25% complete) - Running unknown_outcome_class preflight check for model svm[##------------------] 2020-08-21 09:52:18.988502 - 2 of 16 checks (12.5% complete) - Running scan_time_estimate preflight check for model svm2020-08-21 09:52:18,991 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)[###-----------------] 2020-08-21 09:52:41.595360 - 3 of 16 checks (18.75% complete) - Running fairness_class_samples preflight check for model svm[#####---------------] 2020-08-21 09:52:41.600087 - 4 of 16 checks (25.0% complete) - Finished all preflight checks for model svm[#####---------------] 2020-08-21 09:52:41.600161 - 4 of 16 checks (25.0% complete) - Running model_nondeterminism preflight check for model logit[######--------------] 2020-08-21 09:52:46.330665 - 5 of 16 checks (31.25% complete) - Running unknown_outcome_class preflight check for model logit[#######-------------] 2020-08-21 09:52:46.462958 - 6 of 16 checks (37.5% complete) - Running scan_time_estimate preflight check for model logit2020-08-21 09:52:46,466 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)[########------------] 2020-08-21 09:53:00.454901 - 7 of 16 checks (43.75% complete) - Running fairness_class_samples preflight check for model logit[##########----------] 2020-08-21 09:53:00.459405 - 8 of 16 checks (50.0% complete) - Finished all preflight checks for model logit[##########----------] 2020-08-21 09:53:00.459474 - 8 of 16 checks (50.0% complete) - Running model_nondeterminism preflight check for model dtree[###########---------] 2020-08-21 09:53:05.211731 - 9 of 16 checks (56.25% complete) - Running unknown_outcome_class preflight check for model dtree[############--------] 2020-08-21 09:53:05.340272 - 10 of 16 checks (62.5% complete) - Running scan_time_estimate preflight check for model dtree2020-08-21 09:53:05,342 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)[#############-------] 2020-08-21 09:53:16.939798 - 11 of 16 checks (68.75% complete) - Running fairness_class_samples preflight check for model dtree[###############-----] 2020-08-21 09:53:16.944578 - 12 of 16 checks (75.0% complete) - Finished all preflight checks for model dtree[###############-----] 2020-08-21 09:53:16.944660 - 12 of 16 checks (75.0% complete) - Running model_nondeterminism preflight check for model mlp[################----] 2020-08-21 09:53:21.915100 - 13 of 16 checks (81.25% complete) - Running unknown_outcome_class preflight check for model mlp[#################---] 2020-08-21 09:53:22.063400 - 14 of 16 checks (87.5% complete) - Running scan_time_estimate preflight check for model mlp2020-08-21 09:53:22,065 root WARNING Insufficient examples of some fairness classes to guarantee convergence (smallest class size is for 'male : married/widowed' with 1 samples)[##################--] 2020-08-21 09:53:35.914853 - 15 of 16 checks (93.75% complete) - Running fairness_class_samples preflight check for model mlp[####################] 2020-08-21 09:53:35.919279 - 16 of 16 checks (100.0% complete) - Finished all preflight checks for model mlpPreflight Scan Completed
When all preflight checks are complete, success/failure of preflight checks per model are displayed, as well as errors (not in the following example), warnings, and observations.
====== Preflight Summary ======Total number of results: 40Number of observations: 28Number of warnings: 12Number of errors: 0
Observations for svm model: Passed model non determinism check Passed unknown outcome classes check Expected time for robustness analysis is 12 seconds Expected time for fairness analysis is 12 seconds Expected time for explanation analysis is 28 seconds Expected time for explainability analysis is 12 seconds Expected time for performance analysis is 10 seconds
Warnings for svm model: No counterfactual explanation can be found approximately 92% of rows Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples) Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)
Observations for logit model: Passed model non determinism check Passed unknown outcome classes check Expected time for robustness analysis is 11 seconds Expected time for fairness analysis is 11 seconds Expected time for explanation analysis is 21 seconds Expected time for explainability analysis is 11 seconds Expected time for performance analysis is 10 seconds
Warnings for logit model: No counterfactual explanation can be found approximately 92% of rows Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples) Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)
Observations for dtree model: Passed model non determinism check Passed unknown outcome classes check Expected time for robustness analysis is 11 seconds Expected time for fairness analysis is 11 seconds Expected time for explanation analysis is 19 seconds Expected time for explainability analysis is 11 seconds Expected time for performance analysis is 10 seconds
Warnings for dtree model: No counterfactual explanation can be found approximately 92% of rows Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples) Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)
Observations for mlp model: Passed model non determinism check Passed unknown outcome classes check Expected time for robustness analysis is 11 seconds Expected time for fairness analysis is 11 seconds Expected time for explanation analysis is 21 seconds Expected time for explainability analysis is 11 seconds Expected time for performance analysis is 10 seconds
Warnings for mlp model: No counterfactual explanation can be found approximately 92% of rows Fairness grouping attribute 'age' has small sample size for some values (smallest '> 25 years' with 10 examples) Fairness grouping attribute 'status' has small sample size for some values (smallest 'male : single' with 1 examples)