langtest.transform.sycophancy.SycophancyNlp#

class SycophancyNlp#

Bases: BaseSycophancy

A class for generating synthetic NLP data for the Sycophancy task.

__init__()#

Methods

__init__()
async_run(sample_list, model, **kwargs)	Creates a task to run the sycophancy measure.
generate_age()	Generate a random age.
generate_name()	Generate a random name.
generate_school()	Generate a random school name.
generate_text(text)	Generate synthetic text by removing specific content.
replace_values(text, value_dict)	Replace placeholders in the text with new values.
run(sample_list, model, **kwargs)	Abstract method that implements the sycophancy measure.
transform(sample_list, **kwargs)	Transform a list of samples into a modified list.

Attributes

alias_name
supported_tasks
test_types

async classmethod async_run(sample_list: List[Sample], model: ModelAPI, **kwargs)#

Creates a task to run the sycophancy measure.

Parameters:

• sample_list (List[Sample]) – The input data to be transformed.

• model (ModelAPI) – The model to be used for evaluation.

• **kwargs – Additional arguments to be passed to the sycophancy measure.

Returns:

The task that runs the sycophancy measure.

Return type:

asyncio.Task

static generate_age() → int#

Generate a random age.

Returns:

A randomly generated age between 30 and 90.

Return type:

int

static generate_name() → str#

Generate a random name.

Returns:

A randomly generated name.

Return type:

str

static generate_school() → str#

Generate a random school name.

Returns:

A randomly selected school name.

Return type:

str

static generate_text(text) → str#

Generate synthetic text by removing specific content.

Parameters:

text (str) – The input text.

Returns:

The modified text with specific content removed.

Return type:

str

static replace_values(text, value_dict)#

Replace placeholders in the text with new values.

Parameters:

• text (str) – The text containing placeholders to be replaced.

• value_dict (dict) – A dictionary mapping placeholders to new values.

Returns:

The text with placeholders replaced by their respective values.

Return type:

str

abstract async static run(sample_list: List[Sample], model: ModelAPI, **kwargs) → List[Sample]#

Abstract method that implements the sycophancy measure.

Parameters:

• sample_list (List[Sample]) – The input data to be transformed.

• model (ModelAPI) – The model to be used for evaluation.

• **kwargs – Additional arguments to be passed to the sycophancy measure.

Returns:

The transformed data based on the implemented sycophancy measure.

Return type:

List[Sample]

static transform(sample_list: List[Sample], **kwargs) → List[Sample]#

Transform a list of samples into a modified list.

Parameters:

sample_list (List[Sample]) – A list of Sample objects to transform.

Returns:

A list of modified Sample objects.

Return type:

List[Sample]