Version: 1.3.14

Changelog

This changelog describes updates released in Certifai Toolkit, Pro, and Enterprise editions.

Release Date March 1, 2022 - Version 1.3.14

Infrastructure

  • Flask updated to v1.1.4

Toolkit

  • New evaluation type added: Data Statistics
  • Scanning without model access

Scan Manager

  • Scanning without model access

Policy

  • Added the ability to specify the metric to use in score checks.
  • Users can now specify a rationale and provide references when they configure Policy controls.

Early Access Features

  • RAI Center: A user interface for AI project governance
  • Monitoring Console: A dashboard for monitoring performance, fairness, data drift and data quality over time

Release Date October 27, 2021 - Version 1.3.13

Toolkit

  • Counterfactual sampling works in local scans when values are missing in the datasets.

Release Date September 3, 2021 - Version 1.3.12

Certifai AI Risk Assessment Questionnaire and the Policy Select tool

  • Apply a policy to a use case via the UI
  • Edit an existing use case policy via the UI
  • Download/upload questionnaire answers

Enterprise Platforms

  • Support for deployment in EKS

Certifai Console

  • A single Certifai landing page provides access to Scan Viewer, Scan Manager, and the Policy Select tool.
  • Support for setting a scan baseline in the Scan Viewer
  • Support for specifing which fairness metrics to use through the Scan Manager UI

Scan Manager

  • Support for disparate impact as a fairness metric for classification tasks
  • Other usability improvements

Release Date July 9, 2021 - Version 1.3.11

Enterprise

Announcing Scan Manager - a web-based user-interface for defining scans for Certifai. And the Scan Manager User's Guide.

The ability to create scan definitions from the Scan Manager UI is the major feature being shipped this release.

You can now define a new model use case by uploading a dataset (up to 1GB) and answering questions in the UI. Users no longer need to upload a scan definition.

Additional enterprise features:

  • Updates to Certifai custom resource to support additional annotations in the ingress controller (added is scan-manager.ingress-config).
  • Added scaling guidance for model servers

Toolkit and General

  • Added Shap and alternative fairness into the standard example reports in the toolkit 
  • Support added for saving explanations as csv as for other evaluation types
  • CLI - Descriptions of sub-commands added as part of help message

Release Date April 1, 2021 - Version 1.3.10

General and Toolkit Improvements

  • You can now download counterfactual explanations as a .csv file from the Console when you configure them to be saved in the scan_definition.yml file.
  • A link to the document "Understanding the Console" has been added to the help panel in Console. It provides guidance on interpreting Certifai scores displayed in the Console.
  • Multiple fairness metrics can now be seen in the model view of Console results if it has been enabled for the scan.
  • Rendering issue fixed in Console views.

Enterprise Improvements

  • The initial version of the Scan Manager UI enables users to upload models and datasets to run scans from the Console.

  • The CLI command certifai remote list returns a table that now displays "Creation Time", "Scan Command" and "Scan Variant" in addition to the fields it previously displayed. As an option the command returns a list in JSON format.

  • BETA FEATURE: The CLI command certifai remote explain enables fast explanations. (Ask your CognitiveScale representative for more information)

Release Date March 2, 2021 - Version 1.3.9

General and Toolkit Improvements

  • Added support for diverse counterfactuals for fast explanations
  • Improved support for header-less datasets
  • General upgrades to fix third party security vulnerabilities
  • Added strict JSON encoding for model predictions
  • Improvements to soft predict models usage
  • Made inference of cat values in scan definition-create optional (non-default)

Pro and Enterprise Improvements

  • Fixes and improvements to Certifai remote commands
  • Certifai remote describe now provides similar output to kubectl describe
  • Enterprise Documentation updates

Release Date January 13, 2021 - Version 1.3.8

General Improvements

Enterprise Features & Improvements

Release Date December 4, 2020 - Version 1.3.7

Features

Release Date November 6, 2020 - Version 1.3.6

Features

Release Date October 8, 2020 - Version 1.3.5

Enterprise Features

Certifai Pro Features

  • Forgot password can be disabled in Console

Documentation Features

Improvements

  • Support for private storage endpoints in Azure

Release Date August 27, 2020 - Version 1.3.4

Features

  • Toolkit:
    • Scanner Preflight check option with scan time estimations (for scans running locally or in a Jupyter notebook)
    • Target encoding options in Scan Definition file
  • Notebooks: Target encoding
  • BI and Policy integration: You can create a policy based on risk questionnaire output (Ask your CognitiveScale representative for a demo of this feature!)
  • Enterprise: Open ID Connect and RBAC (Role-based access control) for authentication and authorization

Improvements

  • Business context and violation severity supported in compliance reports
  • Improved risk assessment and compliance report PDFs
  • Support for high cardinality categorical features

Release Date July 30, 2020 - Version 1.3.3

Features

Please contact your CognitiveScale representative for assistance with these features:

  • Risk assessment and policy checking
  • Policy compliance reports included in BI reporting
  • Certifai Pro on VMWare platform

Improvements

  • GCP remote scanning improvements
  • Feature reduction for wide datasets
  • Algorithm fixes and optimizations

Release Date July 10, 2020 - Version 1.3.2

Fixes

  • Issue for scans with one-hot encoded features that prevented the result from being viewed in the Console

Release Date July 9, 2020 - Version 1.3.1

Features

Release Date May 28, 2020 - Version 1.2.14

Features

Improvements

Fixes

  • Issue running in Anaconda-base environment in Windows

Release Date April 30, 2020 - Version 1.2.13

Features

  • Multiclass classification definitions and visualizations for 3 types of use cases
    • no favorable class defined
    • all classes tagged as favorable or unfavorable
    • favorability of classes ordered
  • Certifai Pro edition on Google Cloud Portal
  • Tooltips added to scan visualizations in Certifai Console
  • Notebooks: Custom headers for Azure ML

Improvements

Release Date April 16, 2020 - Version 1.2.12 (add-on)

Release Date: April 9, 2020 - Version 1.2.12

Features

Improvements

  • Relaxed dependencies to support Azure-hosted notebooks
  • Added support for pandas v1
  • Additional Docs Instructions: How to Install Certifai Pro in Azure
  • Console: Model Use Case Details page enhancements, including new visualizations.

Release Date: March 20, 2020 - Version 1.2.11

Features

Improvements

  • Widened the version ranges of the 3rd-party dependencies used in the Certifai API to better support installing Certifai in existing notebooks
  • Console links for Certifai Pro user to storage settings and change password pages
  • Score visualizations in Notebooks
  • Support for installing Certifai into existing notebooks

Fixes

  • Added scanner support for 32-bit python on Windows
  • Added missing py32win dependency for scanner on Windows
  • Added API and prediction values for regression task type

Release Date: February 20, 2020

  • Operator version: 1.2.10
  • Toolkit version: 0.5.1
  • API version: 0.5.1

Certifai Operator

  • Changes to the operator configuration have been made to support Azure BLOB storage.

Certifai Toolkit

  • The Console and Scanner now both default to reading or writing results at the location in the SCAN_RESULTS_DIRECTORY environment variable, or at "./reports" if the environment variable is not defined.
    • Locations, may be overridden on the command line using the -o option with certifai scan, and certifai console <location> for the console.
  • You can now evaluate fairness using features that are only present in the dataset, and not passed to the model by marking these features as "hidden".
  • The example notebooks have been extended to include an example of visualizing the frequency of changed features across a set of explanations.

Certifai Console

  • The Console now opens to a model use case list view.
    • A model use case folder is automatically created when the first scan for that use case is completed.
    • When you click the kebab menu at the far right of a use case in the list, you may open a new model use case details page, or you may view a list of scan reports for the use case.
    • NOTE: Reports created before this change that were not saved within a model use case folder appear as children of the 'default' model use case.
  • From the scan results view, you can now display model detail visualizations using the toggle at the top right.
  • The Fairness Breakdown by Grouping Feature visualization has been added to display details of fairness within each group (e.g. burden for each gender). You may access this chart by toggling to Model at the top right and clicking on the Fairness chart.
  • The scan reports list in the Console now includes a status field to indicate when reports have run successfully or not.
  • The Console more accurately displays the results for regression use cases now.

Certifai CLI

  • The Console can now be started via the CLI by running certifai console <path-to-reports>.
  • A scan run from the CLI can now be canceled using CTRL-C.

Certifai API

  • The names of some classes in the API have been changed for consistency.