Changelog
This changelog describes updates released in Certifai Toolkit, Pro, and Enterprise editions.
Release Date March 1, 2022 - Version 1.3.14
Infrastructure
- Flask updated to v1.1.4
Toolkit
- New evaluation type added: Data Statistics
- Scanning without model access
Scan Manager
- Scanning without model access
Policy
- Added the ability to specify the metric to use in score checks.
- Users can now specify a rationale and provide references when they configure Policy controls.
Early Access Features
- RAI Center: A user interface for AI project governance
- Monitoring Console: A dashboard for monitoring performance, fairness, data drift and data quality over time
Release Date October 27, 2021 - Version 1.3.13
Toolkit
- Counterfactual sampling works in local scans when values are missing in the datasets.
Release Date September 3, 2021 - Version 1.3.12
Certifai AI Risk Assessment Questionnaire and the Policy Select tool
- Apply a policy to a use case via the UI
- Edit an existing use case policy via the UI
- Download/upload questionnaire answers
Enterprise Platforms
- Support for deployment in EKS
Certifai Console
- A single Certifai landing page provides access to Scan Viewer, Scan Manager, and the Policy Select tool.
- Support for setting a scan baseline in the Scan Viewer
- Support for specifing which fairness metrics to use through the Scan Manager UI
Scan Manager
- Support for disparate impact as a fairness metric for classification tasks
- Other usability improvements
Release Date July 9, 2021 - Version 1.3.11
Enterprise
Announcing Scan Manager - a web-based user-interface for defining scans for Certifai. And the Scan Manager User's Guide.
The ability to create scan definitions from the Scan Manager UI is the major feature being shipped this release.
You can now define a new model use case by uploading a dataset (up to 1GB) and answering questions in the UI. Users no longer need to upload a scan definition.
Additional enterprise features:
- Updates to Certifai custom resource to support additional annotations in the ingress controller (added is
scan-manager.ingress-config
). - Added scaling guidance for model servers
Toolkit and General
- Added Shap and alternative fairness into the standard example reports in the toolkit
- Support added for saving explanations as csv as for other evaluation types
- CLI - Descriptions of sub-commands added as part of help message
Release Date April 1, 2021 - Version 1.3.10
General and Toolkit Improvements
- You can now download counterfactual explanations as a .csv file from the Console when you configure them to be saved in the
scan_definition.yml
file. - A link to the document "Understanding the Console" has been added to the help panel in Console. It provides guidance on interpreting Certifai scores displayed in the Console.
- Multiple fairness metrics can now be seen in the model view of Console results if it has been enabled for the scan.
- Rendering issue fixed in Console views.
Enterprise Improvements
The initial version of the Scan Manager UI enables users to upload models and datasets to run scans from the Console.
The CLI command
certifai remote list
returns a table that now displays "Creation Time", "Scan Command" and "Scan Variant" in addition to the fields it previously displayed. As an option the command returns a list in JSON format.BETA FEATURE: The CLI command
certifai remote explain
enables fast explanations. (Ask your CognitiveScale representative for more information)
Release Date March 2, 2021 - Version 1.3.9
General and Toolkit Improvements
- Added support for diverse counterfactuals for fast explanations
- Improved support for header-less datasets
- General upgrades to fix third party security vulnerabilities
- Added strict JSON encoding for model predictions
- Improvements to soft predict models usage
- Made inference of cat values in scan definition-create optional (non-default)
Pro and Enterprise Improvements
- Fixes and improvements to Certifai remote commands
- Certifai remote describe now provides similar output to kubectl describe
- Enterprise Documentation updates
Release Date January 13, 2021 - Version 1.3.8
General Improvements
- Fixed scanner crash when numeric categorical columns have missing values
- Added prediction service and regression examples:
Enterprise Features & Improvements
- Integration with Keycloak for authentication
- Integration with NGINX Plus ingress controller in addition to NGINX ingress controller
- Improved error handling for remote scans
- Fixed issue with parallel scanning
- Improved documentation for RedHat OpenShift Setup
Release Date December 4, 2020 - Version 1.3.7
Features
- CLI/API:
- Toleration of missing values in datasets
- Fixes for issues with Shap explanations
- Beta fast explanations feature (Ask your CognitiveScale representative for more information)
- Examples:
Release Date November 6, 2020 - Version 1.3.6
Features
- CLI:
- One-hot encodings in scan definition generator
- Explanation scan definition in scan definition generator
- API:
- explanation_utils functions to construct dataframes of explanations from results
- report_utils functions to construct dataframes of scores from results
- Examples:
- H2O MOJO model wrapper
- Containerization for a prediction service
- Notebook updates to use new API functions
Release Date October 8, 2020 - Version 1.3.5
Enterprise Features
- Parameters for scaling cpu/memory optimization at runtime
- Kubernetes secrets supported in model headers
- Risk Assessment and Policy integration (Ask your CognitiveScale representative for a demo of these features.):
- You can create a policy based on risk questionnaire output.
- New policies can be created from existing policies
- Custom questionnaires can be uploaded
Certifai Pro Features
- Forgot password can be disabled in Console
Documentation Features
- Guidance for interpreting scan results (by user type):
- Added docs sections specific to Certifai Enterprise and Certifai Pro.
Improvements
- Support for private storage endpoints in Azure
Release Date August 27, 2020 - Version 1.3.4
Features
- Toolkit:
- Scanner Preflight check option with scan time estimations (for scans running locally or in a Jupyter notebook)
- Target encoding options in Scan Definition file
- Notebooks: Target encoding
- BI and Policy integration: You can create a policy based on risk questionnaire output (Ask your CognitiveScale representative for a demo of this feature!)
- Enterprise: Open ID Connect and RBAC (Role-based access control) for authentication and authorization
Improvements
- Business context and violation severity supported in compliance reports
- Improved risk assessment and compliance report PDFs
- Support for high cardinality categorical features
Release Date July 30, 2020 - Version 1.3.3
Features
Please contact your CognitiveScale representative for assistance with these features:
- Risk assessment and policy checking
- Policy compliance reports included in BI reporting
- Certifai Pro on VMWare platform
Improvements
- GCP remote scanning improvements
- Feature reduction for wide datasets
- Algorithm fixes and optimizations
Release Date July 10, 2020 - Version 1.3.2
Fixes
- Issue for scans with one-hot encoded features that prevented the result from being viewed in the Console
NOTE
Because docs version 1.3.1 and 1.3.2 content was identical, version 1.3.1 has been removed.
Release Date July 9, 2020 - Version 1.3.1
NOTE
The following platforms have been verified and released at version 1.3.1: Toolkit, Certifai Pro on Azure, Certifai Pro on AWS, Certifai Enterprise on RHOS
ALERT
In this release CLI commands for running remote scans and remote scan management have been changed. These changes are NOT backward compatible. If you are running an earlier version of Certifai, use the correspondingly versioned docs for instructions on how to run and manage remote scans OR update your toolkit to the latest version.
Features
- Certifai Enterprise:
- Parallel scanning support (See the section entitled "Parallel Scan Execution")
- Business Intelligence (BI) connector
- Toolkit and Console:
- Multiple counterfactual explanations support in scan definition (hyperparameters) and Console.
- One-hot encoding support in the scan definition
- API:
- Model wrapper class (SimpleModelWrapper) for easing deployment of models in a flask app
- Notebooks and Examples:
Release Date May 28, 2020 - Version 1.2.14
Features
- Sagemaker connector module added
- Public repo hosting example notebooks and tutorials
- Notebooks: "SHAP as an alternate explanation type"
- Console: SHAP visualizations for explanations reports
- Certifai Pro Console: SSL cert configuration
- Docs: Scan definition reference doc
Improvements
- Docs: Azure storage instructions in Setup
- CLI: Scan progress reporting
Fixes
- Issue running in Anaconda-base environment in Windows
Release Date April 30, 2020 - Version 1.2.13
Features
- Multiclass classification definitions and visualizations for 3 types of use cases
- no favorable class defined
- all classes tagged as favorable or unfavorable
- favorability of classes ordered
- Certifai Pro edition on Google Cloud Portal
- Tooltips added to scan visualizations in Certifai Console
- Notebooks: Custom headers for Azure ML
Improvements
- Terms added to Glossary in Certifai docs
- Removed
ipynb
optional dependency for installing Notebooks
Release Date April 16, 2020 - Version 1.2.12 (add-on)
- Certifai Pro edition for AWS has been released on the AWS Marketplace.
- Certifai Pro edition for Azure has been released on the Azure Marketplace.
- New Docs Instructions: RHOS: Run remote scans
Release Date: April 9, 2020 - Version 1.2.12
Features
- Certifai Pro for AWS released in the Amazon Marketplace.
- Added CLI commands for:
- Support for new fairness grouping method, "buckets".
- New Toolkit-Example: Remote Scan Definition
- New Toolkit-Notebooks:
- (Tutorial): Building a scan programmatically with your own model
- Using different fairness metrics
- Console Additions:
Download Kubeconfig
link added for Certifai ProPopulate Sample Reports
link added for Certifai Pro
- New Docs Tutorials:
- New Docs Instructions:
Improvements
- Relaxed dependencies to support Azure-hosted notebooks
- Added support for pandas v1
- Additional Docs Instructions: How to Install Certifai Pro in Azure
- Console: Model Use Case Details page enhancements, including new visualizations.
Release Date: March 20, 2020 - Version 1.2.11
Features
- Support for Python versions 3.7 and 3.8
- Scan Details page in Console that displays models, evaluation types, dataset restrictions etc. for each scan
- CLI
--version
flag - Docs tutorials
Improvements
- Widened the version ranges of the 3rd-party dependencies used in the Certifai API to better support installing Certifai in existing notebooks
- Console links for Certifai Pro user to storage settings and change password pages
- Score visualizations in Notebooks
- Support for installing Certifai into existing notebooks
Fixes
- Added scanner support for 32-bit python on Windows
- Added missing py32win dependency for scanner on Windows
- Added API and prediction values for regression task type
Release Date: February 20, 2020
- Operator version: 1.2.10
- Toolkit version: 0.5.1
- API version: 0.5.1
Certifai Operator
- Changes to the operator configuration have been made to support Azure BLOB storage.
Certifai Toolkit
- The Console and Scanner now both default to reading or writing results at the location in the SCAN_RESULTS_DIRECTORY environment variable, or at "./reports" if the environment variable is not defined.
- Locations, may be overridden on the command line using the
-o
option withcertifai scan
, andcertifai console <location>
for the console.
- Locations, may be overridden on the command line using the
- You can now evaluate fairness using features that are only present in the dataset, and not passed to the model by marking these features as "hidden".
- The example notebooks have been extended to include an example of visualizing the frequency of changed features across a set of explanations.
Certifai Console
- The Console now opens to a model use case list view.
- A model use case folder is automatically created when the first scan for that use case is completed.
- When you click the kebab menu at the far right of a use case in the list, you may open a new model use case details page, or you may view a list of scan reports for the use case.
- NOTE: Reports created before this change that were not saved within a model use case folder appear as children of the 'default' model use case.
- From the scan results view, you can now display model detail visualizations using the toggle at the top right.
- The
Fairness Breakdown by Grouping Feature
visualization has been added to display details of fairness within each group (e.g. burden for each gender). You may access this chart by toggling to Model at the top right and clicking on the Fairness chart. - The scan reports list in the Console now includes a status field to indicate when reports have run successfully or not.
- The Console more accurately displays the results for regression use cases now.
Certifai CLI
- The Console can now be started via the CLI by running
certifai console <path-to-reports>
. - A scan run from the CLI can now be canceled using CTRL-C.
Certifai API
- The names of some classes in the API have been changed for consistency.