langtest.transform.sycophancy.SycophancyTestFactory#

class SycophancyTestFactory(data_handler: List[Sample], tests: Dict | None = None, **kwargs)#

Bases: ITests

A class for conducting Sycophancy tests on a given dataset.

This class provides comprehensive functionality for conducting Sycophancy tests on a provided dataset using various configurable test scenarios.

alias_name#

A string representing the alias name for this test factory.

Type:

str

__init__(data_handler: List[Sample], tests: Dict | None = None, **kwargs) None#

Initialize a new SycophancyTestFactory instance.

Parameters:
  • data_handler (List[Sample]) – A list of Sample objects representing the input dataset.

  • tests (Optional[Dict]) – A dictionary of test names and corresponding parameters (default is None).

  • **kwargs – Additional keyword arguments.

Raises:

ValueError – If the tests argument is not a dictionary.

Methods

__init__(data_handler[, tests])

Initialize a new SycophancyTestFactory instance.

available_tests()

Retrieve a dictionary of all available tests, with their names as keys and their corresponding classes as values.

run(sample_list, model, **kwargs)

Runs the specified tests on the given data and returns a list of results.

transform()

Execute the Sycophancy test and return resulting Sample objects.

Attributes

alias_name

supported_tasks

static available_tests() dict#

Retrieve a dictionary of all available tests, with their names as keys and their corresponding classes as values.

Returns:

A dictionary of test names and classes.

Return type:

dict

classmethod run(sample_list: Dict[str, List[Sample]], model: ModelAPI, **kwargs) List[Sample]#

Runs the specified tests on the given data and returns a list of results.

Parameters:
  • sample_list (Dict[str, List[Sample]]) – A dictionary mapping test scenario names to a list of Sample objects.

  • model (ModelAPI) – A ModelAPI object representing the model to be tested.

Returns:

A list of Sample objects with the test results.

Return type:

List[Sample]

transform() List[Sample]#

Execute the Sycophancy test and return resulting Sample objects.

Returns:

A list of Sample objects representing the resulting dataset after conducting the Sycophancy test.

Return type:

List[Sample]