Data

The provided code initializes an instance of the Harness class. It accepts a data parameter, which can be specified as a dictionary with the following attributes.

{
   "data_source": "",
   "subset": "",
   "feature_column": "",
   "target_column": "",
   "split": "",
   "source": "huggingface"
}

Supported data_source formats are task-dependent. The following table provides an overview of the compatible data sources for each specific task.

NER

There are three options for datasets to test NER models: CoNLL, CSV and HuggingFace datasets. Here are some details of what these may look like:

CoNLL Format for NER

LEICESTERSHIRE NNP B-NP B-ORG
TAKE           NNP I-NP O
OVER           IN  B-PP O
AT             NNP B-NP O
TOP            NNP I-NP O
AFTER          NNP I-NP O
INNINGS        NNP I-NP O
VICTORY        NNP I-NP O

CSV Format for NER

Passing a NER Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task='ner',
                  model={'model': 'en_core_web_sm', 'hub':'spacy'},
                  data={"data_source":'test.conll'},
                  config='config.yml') #Either of the two formats can be specified.

Passing a Hugging Face Dataset for NER to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task="ner",
                  model={"model": "en_core_web_sm", "hub": "spacy"},
                  data={"data_source":'wikiann',
                  "subset":"en",
                  "feature_column":"tokens",
                  "target_column":'ner_tags',
                  "split":"test",
                  "source": "huggingface"
                  })

Text Classification

There are 2 options for datasets to test Text Classification models: CSV datasets or loading HuggingFace Datasets containing the name, subset, split, feature_column and target_column for loading the HF datasets. Here are some details of what these may look like:

CSV Format for Text Classification

Here’s a sample dataset:

For CSV files, we support different variations of the column names. They are shown below :

Passing a CSV Text Classification Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task='text-classification',
                  model={'model': 'mrm8488/distilroberta-finetuned-tweets-hate-speech', 'hub':'huggingface'},
                  data={"data_source":'sample.csv'},
                  config='config.yml')

Passing a Hugging Face Dataset for Text Classification to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task="text-classification",
                  model={'model': 'mrm8488/distilroberta-finetuned-tweets-hate-speech', 'hub':'huggingface'},
                  data={"data_source":'glue',
                  "subset":"sst2",
                  "feature_column":"sentence",
                  "target_column":'label',
                  "split":"train",
                  "source": "huggingface"
                  })

Question Answering

To test Question Answering models, the user is meant to select a benchmark dataset. You can see the benchmarks page for all available benchmarks: Benchmarks

You can access the tutorial notebooks to get a quick start on your preferred dataset here: Dataset Notebooks

Passing a Question Answering Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task="question-answering", 
                  model={"model": "text-davinci-003", "hub":"openai"}, 
                  data={"data_source" :"BBQ", "split":"test-tiny"}, config='config.yml')

Summarization

To test Summarization models, the user is meant to select a benchmark dataset from the available ones: Benchmarks

You can access the tutorial notebooks to get a quick start with your preferred dataset here: Dataset Notebooks

Passing a Summarization Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task="summarization", 
                  model={"model": "text-davinci-003","hub":"openai"}, 
                  data={"data_source" :"XSum", "split":"test-tiny"},
                  config='config.yml')
   

Passing a Hugging Face Dataset for Summarization to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task="summarization", 
                  model={'model': 'text-davinci-003', 'hub':'openai'}, 
                  data={"data_source":'samsum',
                  "feature_column":"dialogue",
                  "target_column":'summary',
                  "split":"test",
                  "source": "huggingface"
                  })

Toxicity

This test checks the toxicity of the completion., the user is meant to select a benchmark dataset from the following list:

Benchmark Datasets

Toxicity Benchmarks: Use Cases and Evaluations

Dataset	Use Case	Notebook
Real Toxicity Prompts	Evaluate your model’s accuracy in recognizing and handling toxic language with the Real Toxicity Prompts dataset. It contains real-world prompts from online platforms, ensuring robustness in NLP models to maintain safe environments.

Passing a Toxicity Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task={"task":"text-generation", "category":"toxicity"}, 
                  model={"model": "text-davinci-002","hub":"openai"}, 
                  data={"data_source" :'Toxicity', "split":"test"})

Disinformation Test

This test evaluates the model’s disinformation generation capability. Users should choose a benchmark dataset from the provided list.

Datasets

Disinformation Test Dataset: Use Cases and Evaluations

Dataset	Use Case	Notebook
Narrative-Wedging	Assess the model’s capability to generate disinformation targeting specific groups, often based on demographic characteristics such as race and religion.

Passing a Disinformation Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness  =  Harness(task={"task":"text-generation", "category":"disinformation-test"}, 
                    model={"model": "j2-jumbo-instruct", "hub":"ai21"},
                    data = {"data_source": "Narrative-Wedging"})

Ideology Test

This test evaluates the model’s political orientation. There is one default dataset used for this test.

Datasets

Passing a Disinformation Dataset to the Harness

In ideology test, the data is automatically loaded since there is only one dataset available for now:

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task={"task":"question-answering", "category":"ideology"}, 
            model={'model': "text-davinci-003", "hub": "openai"})

Factuality Test

The Factuality Test is designed to evaluate the ability of LLMs to determine the factuality of statements within summaries, particularly focusing on the accuracy of LLM-generated summaries and potential biases in their judgments. Users should choose a benchmark dataset from the provided list.

Datasets

Factuality Test Dataset: Use Cases and Evaluations

Dataset	Use Case	Notebook
Factual-Summary-Pairs	The Factuality Test is designed to evaluate the ability of LLMs to determine the factuality of statements within summaries, particularly focusing on the accuracy of LLM-generated summaries and potential biases in their judgments.

Passing a Factuality Test Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness  =  Harness(task={"task":"question-answering", "category":"factuality-test"}, 
                    model = {"model": "text-davinci-003", "hub":"openai"},
                    data = {"data_source": "Factual-Summary-Pairs"})

Sensitivity Test

The Sensitivity Test comprises two distinct evaluations: one focusing on assessing a model’s responsiveness to toxicity, particularly when toxic words are introduced into the input text, and the other aimed at gauging its sensitivity to negations, especially when negations are inserted after verbs like “is,” “was,” “are,” and “were”. Users should choose a benchmark dataset from the provided list.

Datasets

Test and Dataset Compatibility

Test Name	Supported Dataset	Notebook
toxicity	wikiDataset-test, wikiDataset-test-tiny
negation	NQ-open-test, NQ-open, NQ-open-test-tiny, OpenBookQA-test, OpenBookQA-test-tiny

Passing a Sensitivity Test Dataset to the Harness

In the Harness, we specify the data input in the following way:

# Import Harness from the LangTest library
from langtest import Harness

harness  =  Harness(task={"task":"question-answering", "category":"sensitivity-test"}, 
                    model = {"model": "text-davinci-003", "hub":"openai"},
                    data={"data_source" :"NQ-open","split":"test-tiny"})

Sycophancy Test

Sycophancy is an undesirable behavior where models tailor their responses to align with a human user’s view even when that view is not objectively correct. In this notebook, we propose a simple synthetic data intervention to reduce this behavior in language models.

Test and Dataset Compatibility

Test Name	Supported Dataset	Notebook
sycophancy_math	sycophancy-math-data
sycophancy_nlp	sycophancy-nlp-data

Passing a Sycophancy Math Dataset to the Harness

In the Harness, we specify the data input in the following way:

import os
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

# Import Harness from the LangTest library
from langtest import Harness

harness = Harness(task={"task":"question-answering", "category":"sycophancy-test"},
                  model={"model": "text-davinci-003","hub":"openai"}, 
                  data={"data_source": 'synthetic-math-data',})