Available Tests

The tables presented below offer a comprehensive overview of diverse categories and tests, providing valuable insights into the varied testing procedures.

Accuracy Tests

Test Name	Supported Tasks
Min F1 Score	ner , text-classification
Min Macro-F1 Score	ner, text-classification
Min Micro-F1 Score	ner, text-classification
Min Precision Score	ner, text-classification
Min Recall Score	ner, text-classification
Min Weighted-F1 Score	ner, text-classification
Min Exact Match Score	question-answering , summarization
Min BLEU Score	question-answering , summarization
Min Rouge1 Score	question-answering , summarization
Min Rouge2 Score	question-answering , summarization
Min RougeL Score	question-answering , summarization
Min RougeLsum Score	question-answering , summarization
LLM Eval	question-answering

Bias Tests

Test Name	Supported Tasks
Replace To Asian First Names	ner, text-classification, question-answering , summarization
Replace To Asian Last Names	ner, text-classification, question-answering , summarization
Replace To Black First Names	ner, text-classification, question-answering , summarization
Replace To Black Last Names	ner, text-classification, question-answering , summarization
Replace To Buddhist Names	ner, text-classification, question-answering , summarization
Replace To Christian Names	ner, text-classification, question-answering , summarization
Replace To Female Pronouns	ner, text-classification, question-answering , summarization
Replace To High Income Country	ner, text-classification, question-answering , summarization
Replace To Hindu Names	ner, text-classification, question-answering , summarization
Replace To Hispanic First Names	ner, text-classification, question-answering , summarization
Replace To Hispanic Last Names	ner, text-classification, question-answering , summarization
Replace To Interracial Last Names	ner, text-classification, question-answering , summarization
Replace To Jain Names	ner, text-classification, question-answering , summarization
Replace To Lower Middle Income Country	ner, text-classification, question-answering , summarization
Replace To Low Income Country	ner, text-classification, question-answering , summarization
Replace To Male Pronouns	ner, text-classification, question-answering , summarization
Replace To Muslim Names	ner, text-classification, question-answering , summarization
Replace To Native American Last Names	ner, text-classification, question-answering , summarization
Replace To Neutral Pronouns	ner, text-classification, question-answering , summarization
Replace To Parsi Names	ner, text-classification, question-answering , summarization
Replace To Sikh Names	ner, text-classification, question-answering , summarization
Replace To Upper Middle Income Country	ner, text-classification, question-answering , summarization
Replace To White First Names	ner, text-classification, question-answering , summarization
Replace To White Last Names	ner, text-classification, question-answering , summarization

Fairness Tests

Test Name	Supported Tasks
Max Gender F1 Score	ner, text-classification
Min Gender F1 Score	ner, text-classification
Min Gender Rouge1 Score	question-answering , summarization
Min Gender Rouge2 Score	question-answering , summarization
Min Gender RougeL Score	question-answering , summarization
Min Gender RougeLSum Score	question-answering , summarization
Min Gender LLM Eval	question-answering
Max Gender Rouge1 Score	question-answering , summarization
Max Gender Rouge2 Score	question-answering , summarization
Max Gender RougeL Score	question-answering , summarization
Max Gender RougeLSum Score	question-answering , summarization
Max Gender LLM Eval	question-answering

Representation Tests

Test Name	Supported Tasks
Min Country Economic Representation Count	ner, text-classification, question-answering , summarization
Min Country Economic Representation Proportion	ner, text-classification, question-answering , summarization
Min Ethnicity Representation Count	ner, text-classification, question-answering , summarization
Min Ethnicity Representation Proportion	ner, text-classification, question-answering , summarization
Min Gender Representation Count	ner, text-classification, question-answering , summarization
Min Gender Representation Proportion	ner, text-classification, question-answering , summarization
Min Label Representation Count	ner, text-classification
Min Label Representation Proportion	ner, text-classification
Min Religion Name Representation Count	ner, text-classification, question-answering , summarization
Min Religion Name Representation Proportion	ner, text-classification, question-answering , summarization

Robustness Tests

Test Name	Supported Tasks
Add Context	ner, text-classification, question-answering , summarization , translation
Add Contraction	ner, text-classification, question-answering , summarization , translation
Add Punctuation	ner, text-classification, question-answering , summarization , translation
Add Typo	ner, text-classification, question-answering , summarization , translation
American to British	ner, text-classification, question-answering , summarization , translation
British to American	ner, text-classification, question-answering , summarization , translation
Lowercase	ner, text-classification, question-answering , summarization , translation
Strip Punctuation	ner, text-classification, question-answering , summarization , translation
Swap Entities	ner
Titlecase	ner, text-classification, question-answering , summarization , translation
Uppercase	ner, text-classification, question-answering , summarization , translation
Number to Word	ner, text-classification, question-answering , summarization , translation
Add OCR Typo	ner, text-classification, question-answering , summarization . translation
Dyslexia Word Swap	ner, text-classification, question-answering , summarization , translation
Add Slangs	ner, text-classification, question-answering , summarization , translation
Add Speech to Text Typo	ner, text-classification, question-answering , summarization , translation
Add Abbreviations	ner, text-classification, question-answering , summarization
Multiple Perturbations	text-classification, question-answering , summarization , translation
Adjective Synonym Swap	ner, text-classification, question-answering , summarization , translation
Adjective Antonym Swap	ner, text-classification, question-answering , summarization , translation
Strip All Punctution	ner, text-classification, question-answering , summarization , translation
Randomize Age	ner, text-classification, question-answering , summarization , translation

Toxicity Tests

Test Name	Supported Tasks
general_toxicity	text-generation
identity_attack	text-generation
insult	text-generation
homosexual_gay_or_lesbian	text-generation
threat	text-generation
obscene	text-generation

Sensitivity Tests

Test Name	Supported Tasks
Add Negation	question-answering
Add Toxic Words	question-answering

Sycophancy Tests

Test Name	Supported Tasks
Sycophancy Math	question-answering
Sycophancy NLP	question-answering

Stereotype Tests

Test Name	Supported Tasks
wino Bias	fill-mask , question-answering
CrowS Pairs	fill-mask

StereoSet Tests

Test Name	Supported Tasks
intersentence	question-answering
intrasentence	question-answering

Ideology Tests

Test Name	Supported Tasks
Political Compass	question-answering

Legal Tests

Test Name	Supported Tasks
legal-support	question-answering

Clinical Tests

Test Name	Supported Tasks
demographic-bias	text-generation

Security Tests

Test Name	Supported Tasks
prompt_injection_attack	text-generation

Disinformation Tests

Test Name	Supported Tasks
Narrative Wedging	text-generation

Factuality Tests

Test Name	Supported Tasks
Order Bias	question-answering

Grammar Tests

Test Name	Supported Tasks
Paraphrase	text-classification, question-answering