Comparator

The main goal of Astrapia is to allow simple comparison of explainers. The Comparator class allows for setting up comparisons, storing, and visualizing them.

class astrapia.comparator.ExplainerComparator

A comparator that allows the user to add explainers, let them explain instances and store metrics from the different explainers

add_explainer(explainer: Explainer, name: str)

Add an instantiated explainer to the comparator. Use the name attribute for uniquely identifying different explainers (E.g. between ‘Anchors acc>95%’ and ‘Anchors acc>85%’)

Parameters:

  • explainer – explainer object

  • name – unique name for identifying the explainer

explain_instances(instances: DataFrame, inferred_metrics=False)

Create explanations for all combinations of provided explainers and instances, then save metrics

Parameters:

  • inferred_metrics – Check whether you want to include inferred metrics in the report

  • instances – instances to be used to create explanations as pandas dataframe

explain_representative(data: Dataset, sampler: str = 'splime', count: int = 10, pred_fn=None, return_samples: bool = False, inferred_metrics=False, **kwargs)

Create a representative explanation for the given data

Parameters:

  • inferred_metrics – Check whether you want to include inferred metrics in the report

  • return_samples – Check whether sampled elements should be returned

  • pred_fn – Provide optional prediction function for sampling

  • data – pandas dataframe with the data to be explained

  • sampler – sampler to be used to create representative explanation

  • count – amount of representative samples to be created

Comparing explainers

To compare Explainers, they, as well as the Comparator itself first need to be initialized. For details on how to initialize specific explainers, visit the Explainers reference.

from astrapia.comparator import ExplainerComparator
from astrapia.dataset import load_dataset
from astrapia.explainers import LimeExplainer, AnchorsExplainer

data = load_dataset(...) # load a dataset for comparing on

lime = LimeExplainer(...) # initialize a lime explainer
anchors = AnchorsExplainer(...) # initialize an anchors explainer

comparator = ExplainerComparator() # initialize a comparator

Then, add each explainer using the add_explainer method.

comparator.add_explainer(lime, 'Lime')
comparator.add_explainer(anchors, 'Anchors')

Now, you can explain instances using the explain_instances method.

comparator.explain_instances(data.train.iloc[[0, 1, 2]])

Visualize the results using the visualization module.

Representative Sampling

As explaining can take a bit of time, it is often useful to compare explainers on a representative subset of samples. By using smart Samplers, you can limit the number of samples while keeping a meaninful comparison.

Instead of using the explain_instances method, you can run

comparator.explain_representative(data, sampler='splime', count=5, pred_fn=pred_fn)

Notice that for this method, you need to specify a prediction function as the SP-Lime Sampler needs to know how the model is predicting different samples.