langtest.transform.sycophancy.BaseSycophancy#

class BaseSycophancy#

Bases: ABC

Abstract base class for implementing sycophancy measures.

alias_name#

A name or list of names that identify the sycophancy measure.

Type:

str

transform(data

List[Sample]) -> Any: Transforms the input data into an output based on the implemented sycophancy measure.

__init__()#

Methods

__init__()

async_run(sample_list, model, **kwargs)

Creates a task to run the sycophancy measure.

run(sample_list, model, **kwargs)

Abstract method that implements the sycophancy measure.

transform(sample_list)

Abstract method that implements the sycophancy measure.

Attributes

alias_name

supported_tasks

test_types

async classmethod async_run(sample_list: List[Sample], model: ModelAPI, **kwargs)#

Creates a task to run the sycophancy measure.

Parameters:
  • sample_list (List[Sample]) – The input data to be transformed.

  • model (ModelAPI) – The model to be used for evaluation.

  • **kwargs – Additional arguments to be passed to the sycophancy measure.

Returns:

The task that runs the sycophancy measure.

Return type:

asyncio.Task

abstract async static run(sample_list: List[Sample], model: ModelAPI, **kwargs) List[Sample]#

Abstract method that implements the sycophancy measure.

Parameters:
  • sample_list (List[Sample]) – The input data to be transformed.

  • model (ModelAPI) – The model to be used for evaluation.

  • **kwargs – Additional arguments to be passed to the sycophancy measure.

Returns:

The transformed data based on the implemented sycophancy measure.

Return type:

List[Sample]

abstract static transform(sample_list: List[Sample]) List[Sample]#

Abstract method that implements the sycophancy measure.

Parameters:

sample_list (List[Sample]) – The input data to be transformed.

Returns:

The transformed data based on the implemented sycophancy measure.

Return type:

Any