Configuring Tests
The configuration for the tests can be passed in the form of a YAML file to the config parameter in the Harness, or by using the configure() method.
Using the YAML Configuration File
tests:
defaults:
min_pass_rate: 0.65
min_score: 0.8
robustness:
lowercase:
min_pass_rate: 0.60
uppercase:
min_pass_rate: 0.60
bias:
replace_to_female_pronouns
accuracy:
min_f1_score:
min_score: 0.8
from langtest import Harness
# Create test Harness with config file
h = Harness(task='text-classification', model={'model': 'path/to/local_saved_model', 'hub':'spacy'}, data={"data_source":'test.csv'}, config='config.yml')
Using the .configure() Method
from langtest import Harness
# Create test Harness without config file
h = Harness(task='text-classification', model={'model': 'path/to/local_saved_model', 'hub':'spacy'}, data={"data_source":'test.csv'})
h.configure(
{
'tests': {
'defaults': {
'min_pass_rate': 0.65
'min_score': 0.8
},
'robustness': {
'lowercase': { 'min_pass_rate': 0.60 },
'uppercase': { 'min_pass_rate': 0.60 }
},
'bias': {
'replace_to_female_pronouns'
},
'accuracy': {
'min_f1_score'
}
}
}
)
Config for LLMs
The configuration for LLM-related tests requires extra fields. These are hub-specific and contain parameters passed to the LLM.
Example Config File: Question Answering
model_parameters:
temperature: 0.2
max_tokens: 64
tests:
defaults:
min_pass_rate: 0.65
min_score: 0.8
robustness:
lowercase:
min_pass_rate: 0.60
uppercase:
min_pass_rate: 0.60
Example Config File: Summarization
model_parameters:
temperature: 0.2
max_tokens: 200
tests:
defaults:
min_pass_rate: 0.75
evaluation_metric: 'rouge'
threshold: 0.5
robustness:
lowercase:
min_pass_rate: 0.60
uppercase:
min_pass_rate: 0.60
Example Config File: Toxicity
model_parameters:
temperature: 0.2
max_tokens: 200
tests:
defaults:
min_pass_rate: 0.75
toxicity:
offensive:
min_pass_rate: 0.60
Note: If you are using Azure OpenAI, please ensure that you modify the Config file by adding the deployment_name parameter under the model_parameters section.