Load

Load Test Cases

The Harness is able to load the saved test cases, test configuration and test data.

# Load saved test cases, test configuration, test data  
harness = Harness.load(
    save_dir="path/to/saved_test_folder",
    model={"model": "gpt-3.5-turbo-instruct","hub":"openai"}, 
    task='question-answering', 
    load_testcases=True
)

Once the harness is loaded, the test cases can then be run with any new model by calling harness.run().

Load Model response

Load model responses along with the test harness configuration.

harness = Harness.load(
    save_dir="saved_model_reponse",
    model={"model": "gpt-3.5-turbo-instruct","hub":"openai"}, 
    task="question-answering",
    load_model_response=True
)

After loading the harness, you can re-evaluate the model responses using alternative metrics