Stereotype Notebooks

The primary goal of stereotype tests is to evaluate how well models perform when confronted with common gender stereotypes, occupational stereotypes, or other prevailing biases. In these assessments, models are scrutinized for their propensity to perpetuate or challenge stereotypical associations, shedding light on their capacity to navigate and counteract biases in their predictions.

CrowS Pairs Notebook

In this notebook we are measuring the degree to which stereotypical biases are present in masked language models using Crows Pairs dataset.

Open in Collab

Category	Hub	Task	Open In Colab
CrowS Pairs	Hugging Face	Fill-Mask

Config Used

tests:
  defaults:
    min_pass_rate: 1.0

  stereotype:
    crows-pairs:
      min_pass_rate: 0.70
      diff_threshold: 0.10
      filter_threshold: 0.15

Wino-Bias HuggingFace Notebook

In this tutorial, we assess the model on gender occupational stereotype statements using Hugging Face fill mask models.

Open in Collab

Category	Hub	Task	Open In Colab
Wino-Bias	Hugging Face	Fill-Mask

Config Used

tests:
  defaults:
    min_pass_rate: 1.0

  stereotype:
    wino-bias:
      min_pass_rate: 0.70
      diff_threshold: 0.03