langtest.transform.factuality.FactualityTestFactory#

class FactualityTestFactory(data_handler: List[Sample], tests: Dict | None = None, **kwargs)#

Bases: ITests

Factory class for factuality test

__init__(data_handler: List[Sample], tests: Dict | None = None, **kwargs) None#

Initializes the FactualityTestFactory

Methods

__init__(data_handler[, tests])

Initializes the FactualityTestFactory

async_run(model, *args, **kwargs)

Runs factuality tests

available_tests()

Retrieves available factuality test types.

run(sample_list, model, **kwargs)

Runs factuality tests

transform()

Nothing to use transform for no longer to generating testcases.

Attributes

alias_name

supported_tasks

class TestConfig#

Bases: dict

clear() None.  Remove all items from D.#
copy() a shallow copy of D#
fromkeys(value=None, /)#

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)#

Return the value for key if key is in the dictionary, else default.

items() a set-like object providing a view on D's items#
keys() a set-like object providing a view on D's keys#
pop(k[, d]) v, remove specified key and return the corresponding value.#

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()#

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)#

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) None.  Update D from dict/iterable E and F.#

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values() an object providing a view on D's values#
async async_run(model: ModelAPI, *args, **kwargs)#

Runs factuality tests

Parameters:
  • sample_list (list[Sample]) – A list of Sample objects to be tested.

  • model (ModelAPI) – The model to be used for testing.

  • *args – Additional positional arguments.

  • **kwargs – Additional keyword arguments.

Returns:

A list of Sample objects with test results.

Return type:

list[Sample]

classmethod available_tests() Dict[str, str]#

Retrieves available factuality test types.

Returns:

A dictionary mapping test names to their corresponding classes.

Return type:

dict

async classmethod run(sample_list: List[Sample], model: ModelAPI, **kwargs) List[Sample]#

Runs factuality tests

Parameters:
  • sample_list (list[Sample]) – A list of Sample objects to be tested.

  • model (ModelAPI) – The model to be used for evaluation.

  • **kwargs – Additional keyword arguments.

Returns:

A list of Sample objects with test results.

Return type:

list[Sample]

transform() List[Sample]#

Nothing to use transform for no longer to generating testcases.

Returns:

Empty list