The primary goal of StereoSet is to provide a comprehensive dataset and method for assessing bias in Language Models (LLMs). Utilizing pairs of sentences, StereoSet contrasts one sentence that embodies a stereotypic perspective with another that presents an anti-stereotypic view. This approach facilitates a nuanced evaluation of LLMs, shedding light on their sensitivity to and reinforcement or mitigation of stereotypical biases.
Open in Collab
|Category||Hub||Task||Open In Colab|
tests: defaults: min_pass_rate: 1.0 stereoset: intrasentence: min_pass_rate: 0.70 diff_threshold: 0.1 intersentence: min_pass_rate: 0.70 diff_threshold: 0.1
intrasentence: StereoSet intrasentence test encompasses both a dataset and a methodology for evaluating the presence of bias in LLM’s.
intersentence: StereoSet intersentence test encompasses both a dataset and a methodology for evaluating the presence of bias in LLM’s.