langtest.transform.sycophancy.SycophancyMath#

class SycophancyMath#

Bases: BaseSycophancy

A class for performing sycophancy measures on mathematical claims.

This class implements a sycophancy measure specifically tailored for mathematical claims.

alias_name#

A string representing the alias name for this sycophancy measure.

Type:

str

transform(sample_list

List[Sample], prob: float = 1) -> List[Sample]: Transforms the input data into an output based on the mathematical sycophancy measure.

__init__()#

Methods

__init__()
async_run(sample_list, model, **kwargs)	Creates a task to run the sycophancy measure.
generate_name()	Generate a random name.
run(sample_list, model, **kwargs)	Abstract method that implements the sycophancy measure.
transform(sample_list, **kwargs)	Transforms the input data into an output based on the mathematical sycophancy measure.

Attributes

alias_name
supported_tasks
test_types

async classmethod async_run(sample_list: List[Sample], model: ModelAPI, **kwargs)#

Creates a task to run the sycophancy measure.

Parameters:

• sample_list (List[Sample]) – The input data to be transformed.

• model (ModelAPI) – The model to be used for evaluation.

• **kwargs – Additional arguments to be passed to the sycophancy measure.

Returns:

The task that runs the sycophancy measure.

Return type:

asyncio.Task

static generate_name() → str#

Generate a random name.

Returns:

A randomly generated name.

Return type:

str

abstract async static run(sample_list: List[Sample], model: ModelAPI, **kwargs) → List[Sample]#

Abstract method that implements the sycophancy measure.

Parameters:

• sample_list (List[Sample]) – The input data to be transformed.

• model (ModelAPI) – The model to be used for evaluation.

• **kwargs – Additional arguments to be passed to the sycophancy measure.

Returns:

The transformed data based on the implemented sycophancy measure.

Return type:

List[Sample]

static transform(sample_list: List[Sample], **kwargs) → List[Sample]#

Transforms the input data into an output based on the mathematical sycophancy measure.

Parameters:

• sample_list (List[Sample]) – The input data to be transformed.

• prob (float) – Probability of transformation (default is 1).

Returns:

The transformed data based on the mathematical sycophancy measure.

Return type:

List[Sample]