langtest.transform.sycophancy.SycophancyNlp#
- class SycophancyNlp#
Bases:
BaseSycophancy
A class for generating synthetic NLP data for the Sycophancy task.
- __init__()#
Methods
__init__
()async_run
(sample_list, model, **kwargs)Creates a task to run the sycophancy measure.
Generate a random age.
Generate a random name.
Generate a random school name.
generate_text
(text)Generate synthetic text by removing specific content.
replace_values
(text, value_dict)Replace placeholders in the text with new values.
run
(sample_list, model, **kwargs)Abstract method that implements the sycophancy measure.
transform
(sample_list, **kwargs)Transform a list of samples into a modified list.
Attributes
alias_name
supported_tasks
test_types
- class TestConfig#
Bases:
dict
- clear() None. Remove all items from D. #
- copy() a shallow copy of D #
- fromkeys(value=None, /)#
Create a new dictionary with keys from iterable and values set to value.
- get(key, default=None, /)#
Return the value for key if key is in the dictionary, else default.
- items() a set-like object providing a view on D's items #
- keys() a set-like object providing a view on D's keys #
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If the key is not found, return the default if given; otherwise, raise a KeyError.
- popitem()#
Remove and return a (key, value) pair as a 2-tuple.
Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.
- setdefault(key, default=None, /)#
Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
- update([E, ]**F) None. Update D from dict/iterable E and F. #
If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
- values() an object providing a view on D's values #
- async classmethod async_run(sample_list: List[Sample], model: ModelAPI, **kwargs)#
Creates a task to run the sycophancy measure.
- Parameters:
sample_list (List[Sample]) – The input data to be transformed.
model (ModelAPI) – The model to be used for evaluation.
**kwargs – Additional arguments to be passed to the sycophancy measure.
- Returns:
The task that runs the sycophancy measure.
- Return type:
asyncio.Task
- static generate_age() int #
Generate a random age.
- Returns:
A randomly generated age between 30 and 90.
- Return type:
int
- static generate_name() str #
Generate a random name.
- Returns:
A randomly generated name.
- Return type:
str
- static generate_school() str #
Generate a random school name.
- Returns:
A randomly selected school name.
- Return type:
str
- static generate_text(text) str #
Generate synthetic text by removing specific content.
- Parameters:
text (str) – The input text.
- Returns:
The modified text with specific content removed.
- Return type:
str
- static replace_values(text, value_dict)#
Replace placeholders in the text with new values.
- Parameters:
text (str) – The text containing placeholders to be replaced.
value_dict (dict) – A dictionary mapping placeholders to new values.
- Returns:
The text with placeholders replaced by their respective values.
- Return type:
str
- abstract async static run(sample_list: List[Sample], model: ModelAPI, **kwargs) List[Sample] #
Abstract method that implements the sycophancy measure.
- Parameters:
sample_list (List[Sample]) – The input data to be transformed.
model (ModelAPI) – The model to be used for evaluation.
**kwargs – Additional arguments to be passed to the sycophancy measure.
- Returns:
The transformed data based on the implemented sycophancy measure.
- Return type:
List[Sample]
- static transform(sample_list: List[Sample], **kwargs) List[Sample] #
Transform a list of samples into a modified list.
- Parameters:
sample_list (List[Sample]) – A list of Sample objects to transform.
- Returns:
A list of modified Sample objects.
- Return type:
List[Sample]