In the Stereotype notebook, we’re evaluating gpt-3.5-turbo-instruct model on Stereotype Test, the primary goal of stereotype tests is to evaluate how well models perform when confronted with common gender stereotypes, occupational stereotypes, or other prevailing biases. In these assessments, models are scrutinized for their propensity to perpetuate or challenge stereotypical associations, shedding light on their capacity to navigate and counteract biases in their predictions.

Open in Collab

Category Hub Task Dataset Used Open In Colab
Stereotype OpenAI/AI21 Question-Answering Wino-test Open In Colab

Config Used

    min_pass_rate: 1.0
      min_pass_rate: 0.70

Supported Tests

  • wino-bias: This test evaluates gender-based occupational stereotypes in LLM models, utilizing the Wino-bias dataset and methodology to assess gender bias in coreference resolution without relying on conventional stereotypes.