nlptest.nlptest.Harness#

class Harness(model: str | Any, task: str, hub: str | None = None, data: str | None = None, config: str | dict | None = None)#

Bases: object

Harness is a testing class for NLP models.

Harness class evaluates the performance of a given NLP model. Given test data is used to test the model. A report is generated with test results.

__init__(model: str | Any, task: str, hub: str | None = None, data: str | None = None, config: str | dict | None = None)#

Initialize the Harness object.

Parameters:

  • task (str, optional) – Task for which the model is to be evaluated.

  • model (str | ModelFactory) – ModelFactory object or path to the model to be evaluated.

  • hub (str, optional) – model hub to load from the path. Required if path is passed as ‘model’.

  • data (str, optional) – Path to the data to be used for evaluation.

  • config (str | dict, optional) – Configuration for the tests to be performed.

Raises:

ValueError – Invalid arguments.

Methods

__init__(model, task[, hub, data, config])

Initialize the Harness object.

augment(input_path, output_path[, inplace])

Augments the data in the input file located at input_path and saves the result to output_path.

configure(config)

Configure the Harness with a given configuration.

generate()

Generates the testcases to be used when evaluating the model.

generated_results()

Generates an overall report with every textcase and labelwise metrics.

load(save_dir, model, task[, hub])

Loads a previously saved Harness from a given configuration and dataset

report()

Generate a report of the test results.

run()

Run the tests on the model using the generated testcases.

save(save_dir)

Save the configuration, generated testcases and the DataFactory to be reused later.

testcases()

Testcases after .generate() is called

Attributes

DEFAULTS_DATASET

SUPPORTED_HUBS

SUPPORTED_TASKS

augment(input_path: str, output_path: str, inplace: bool = False) → Harness#

Augments the data in the input file located at input_path and saves the result to output_path.

Parameters:

  • input_path (str) – Path to the input file.

  • output_path (str) – Path to save the augmented data.

  • inplace (bool, optional) – Whether to modify the input file directly. Defaults to False.

Returns:

The instance of the class calling this method.

Return type:

Harness

Raises:

ValueError – If the pass_rate or minimum_pass_rate columns have an unexpected data type.

Note

This method uses an instance of AugmentRobustness to perform the augmentation.

Example

>>> harness = Harness(...)
>>> harness.augment("train.conll", "augmented_train.conll")
configure(config: str | dict) → dict#

Configure the Harness with a given configuration.

Parameters:

config (str | dict) – Configuration file path or dictionary for the tests to be performed.

Returns:

Loaded configuration.

Return type:

dict

generate() → Harness#

Generates the testcases to be used when evaluating the model. The generated testcases are stored in _testcases attribute.

generated_results() → DataFrame | None#

Generates an overall report with every textcase and labelwise metrics.

Returns:

Generated dataframe.

Return type:

pd.DataFrame

classmethod load(save_dir: str, model: str | ModelFactory, task: str, hub: str | None = None) → Harness#

Loads a previously saved Harness from a given configuration and dataset

Parameters:

  • save_dir (str) – path to folder containing all the needed files to load an saved Harness

  • task (str) – task for which the model is to be evaluated.

  • model (str | ModelFactory) – ModelFactory object or path to the model to be evaluated.

  • hub (str, optional) – model hub to load from the path. Required if path is passed as ‘model’.

Returns:

Harness loaded from from a previous configuration along with the new model to evaluate

Return type:

Harness

report() → DataFrame#

Generate a report of the test results.

Returns:

DataFrame containing the results of the tests.

Return type:

pd.DataFrame

run() → Harness#

Run the tests on the model using the generated testcases.

Returns:

The evaluations are stored in generated_results attribute.

Return type:

None

save(save_dir: str) → None#

Save the configuration, generated testcases and the DataFactory to be reused later.

Parameters:

save_dir (str) – path to folder to save the different files

Returns:

testcases() → DataFrame#

Testcases after .generate() is called

Returns:

testcases formatted into a pd.DataFrame

Return type:

pd.DataFrame