certifai.scanner.report_utils module

class certifai.scanner.report_utils.ConfidenceBounds(low: float, high: float, confidence_percent: int)

Confidence bounds for a value

Parameters

  • low – Lower confidence bound

  • high – Upper confidence bound

  • confidence_percent – Percentage confidence of the bound

Create new instance of ConfidenceBounds(low, high, confidence_percent)

property low

Alias for field number 0

property high

Alias for field number 1

property confidence_percent

Alias for field number 2

class certifai.scanner.report_utils.Score(value: Optional[float], confidence_bounds: Optional[certifai.scanner.report_utils.ConfidenceBounds], context: Optional[str], name: str, type_name: Optional[str], children: Iterable[certifai.scanner.report_utils.Score])

Score element

Parameters

  • value – The score value

  • confidence_bounds – The optional confidence bounds

  • context – Optional titular context of the score

  • name – Name of the score

  • children – Nested child scores

Create new instance of Score(value, confidence_bounds, context, name, type_name, children)

property value

Alias for field number 0

property confidence_bounds

Alias for field number 1

property context

Alias for field number 2

property name

Alias for field number 3

property type_name

Alias for field number 4

property children

Alias for field number 5

certifai.scanner.report_utils.scores(evaluation_type: str, report: dict, max_depth: int = - 1) → Generator[certifai.scanner.report_utils.Score, None, None]

Generate scores for a given evaluation type from a report dictionary

Parameters

  • evaluation_type – one of ‘robustness’, ‘fairness’, ‘explainability’, ‘atx’

  • report – report dictionary

  • depth – nesting depth to include sub-scores to (default all)

Returns

generator of Score instances

certifai.scanner.report_utils.all_scores(report: dict, max_depth: int = - 1) → Generator[certifai.scanner.report_utils.Score, None, None]

Generate scores for all evaluation types from a report dictionary

Parameters

  • report – report dictionary

  • max_depth – nesting depth to include sub-scores to (default all)

Returns

generator of Score instances

certifai.scanner.report_utils.construct_scores_dataframe(scores: Iterable[certifai.scanner.report_utils.Score], index_by_root_context: bool = True, include_confidence: bool = True, include_score_type: bool = True, include_context: bool = True, type_match: Optional[str] = None)

Construct a Pandas DataFrame with scoring information from a list of Score

Parameters

  • scores – list of scores to extract information from

  • index_by_root_context – If True index the dataframe by the contexts of the top level scores (default True)

  • include_confidence – If True include confidence intervals for scores where available (default True)

  • type_match – Optional top-level Score type to filter by. Use to ensure a homogeneous type is selected from a list of scores of many types, such as different fairness measures

Returns

dataframe

certifai.scanner.report_utils.construct_group_scores_dataframe(scores: Iterable[certifai.scanner.report_utils.Score], include_confidence: bool = True, type_match: Optional[str] = None) → pandas.core.frame.DataFrame

Construct a dataframe with columns for each fairness group containing the detailed metric values for each fairness metric in the evaluation. There may be one or more rows for each model evaluation, depending on the fairness metric. For example, ‘equal odds’ has two metric values (‘true positive rate’ and ‘true negative rate’).

Parameters

  • scores – list of scores to extract information from

  • include_confidence – If True include confidence intervals for scores where available (default True)

  • type_match – Optional score type (e.g. burden) to filter by.

Returns

dataframe

certifai.scanner.report_utils.construct_feature_scores_dataframe(scores: Iterable[certifai.scanner.report_utils.Score], include_confidence: bool = True, type_match: Optional[str] = None) → pandas.core.frame.DataFrame

Construct a dataframe with columns for each fairness feature containing the score for each fairness metric in the evaluation.

Parameters

  • scores – list of scores to extract information from

  • include_confidence – If True include confidence intervals for scores where available (default True)

  • type_match – Optional score type (e.g. burden) to filter by.

Returns

pd.DataFrame

certifai.scanner.report_utils.construct_overall_scores_dataframe(scores: Iterable[certifai.scanner.report_utils.Score], include_confidence: bool = True, type_match: Optional[str] = None) → pandas.core.frame.DataFrame

Construct a dataframe with the overall score for each evaluation type.

Parameters

  • scores – list of scores to extract information from

  • include_confidence – If True include confidence intervals for scores where available (default True)

  • type_match – Optional score type (e.g. burden) to filter by.

Returns

pd.DataFrame