langtest.transform.base.TestFactory#

class TestFactory#

Bases: object

A factory class for creating and running different types of tests on data.

__init__()#

Methods

__init__()
async_run(samples_list, model_handler, **kwargs)	Runs the specified tests on the given data and returns a list of results.
call_add_custom_bias(data, name, append)	Add custom bias to the given data.
run(samples_list, model_handler, **kwargs)	Runs the specified tests on the given data and returns a list of results.
test_categories()	Returns a dictionary mapping test category names to the corresponding test classes.
test_scenarios()	Returns a dictionary mapping test class names to the available test scenarios for each class.
transform(task, data, test_types, *args, ...)	Runs the specified tests on the given data and returns a list of results.

Attributes

is_augment
task

async classmethod async_run(samples_list: List[Sample], model_handler: ModelAPI, **kwargs)#

Runs the specified tests on the given data and returns a list of results.

Parameters:

• samples_list – List[Sample]

• model_handler – ModelAPI

static call_add_custom_bias(data: list | dict, name: str, append: bool) → None#

Add custom bias to the given data.

Parameters:

• data (Union[list, dict]) – The data to which the custom bias will be added.

• name (str) – The name of the custom bias.

• append (bool) – Indicates whether to append the custom bias or replace the existing bias.

Returns:

None

static run(samples_list: List[Sample], model_handler: ModelAPI, **kwargs)#

Runs the specified tests on the given data and returns a list of results.

Parameters:

• samples_list – List[Sample]

• model_handler – ModelAPI

classmethod test_categories() → Dict#

Returns a dictionary mapping test category names to the corresponding test classes.

Returns:

A dictionary mapping test category names to the corresponding test classes.

Return type:

Dict

classmethod test_scenarios()#

Returns a dictionary mapping test class names to the available test scenarios for each class.

Returns:

A dictionary mapping test class names to the available test scenarios for each class.

Return type:

Dict

static transform(task: str, data: List[Sample], test_types: dict, *args, **kwargs) → List[Result]#

Runs the specified tests on the given data and returns a list of results.

Parameters:

• data – List[Sample] The data to be tested.

• test_types – dict A dictionary mapping test category names to lists of test scenario names.

• model – ModelAPI Model to be tested.

Returns:

List[Results]

A list of results from running the specified tests on the data.