langtest.utils.custom_types.sample.BaseSample#

class BaseSample(*, original: str = None, test_type: str = None, test_case: str = None, expected_results: Result = None, actual_results: Result = None, transformations: List[Transformation] = None, category: str = None, state: str = None, threshold: float = None, dataset_name: str = None)#

Bases: BaseModel

Helper object storing the original text, the perturbed one and the corresponding predictions for each of them.

The specificity here is that it is task-agnostic, one only needs to call access the is_pass property to assess whether the expected_results and the actual_results are the same, regardless the downstream task.langtest/utils/custom_types.py

This way, to support a new task one only needs to create a XXXOutput model, overload the __eq__ operator and add the new model to the Result type variable.

__init__(**data)#: Constructor method

Methods

`__init__`(**data)	Constructor method
`construct`([_fields_set])	Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
`copy`(*[, include, exclude, update, deep])	Duplicate a model, optionally choose which fields to include, exclude and change.
`dict`(*[, include, exclude, by_alias, ...])	Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
`from_orm`(obj)
`is_pass`()	Checks if the sample passes based on the maximum score.
`json`(*[, include, exclude, by_alias, ...])	Generate a JSON representation of the model, include and exclude arguments as per dict().
`parse_file`(path, *[, content_type, ...])
`parse_obj`(obj)
`parse_raw`(b, *[, content_type, encoding, ...])
`schema`([by_alias, ref_template])
`schema_json`(*[, by_alias, ref_template])
`sort_transformations`(v)	Validator ensuring that transformations are in correct order
`to_dict`()	Returns the dict version of sample.
`update_forward_refs`(**localns)	Try to update ForwardRefs on fields based on this Model, globalns and localns.
`validate`(value)

Attributes

`irrelevant_transformations`	Retrieves the transformations that do not need to be taken into
`relevant_transformations`	Retrieves the transformations that need to be taken into account to realign original and test_case.
`original`
`test_type`
`test_case`
`expected_results`
`actual_results`
`transformations`
`category`
`state`
`threshold`
`dataset_name`

classmethod construct(_fields_set: SetStr | None = None, **values: Any) → Model#: Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed. Behaves as if Config.extra = ‘allow’ was set since it adds all passed values

Duplicate a model, optionally choose which fields to include, exclude and change.

Parameters:

include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating the new model: you should trust this data
deep – set to True to make a deep copy of the model

Returns:

new model instance

dict(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, by_alias: bool = False, skip_defaults: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny#: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

property irrelevant_transformations: List[Transformation] | None#

Retrieves the transformations that do not need to be taken into: account to realign original and test_case.

Returns:: list of transformations which should be ignored
Return type:: Optional[List[Transformation]]

is_pass() → bool#: Checks if the sample passes based on the maximum score.

json(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, by_alias: bool = False, skip_defaults: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Callable[[Any], Any] | None = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode#

Generate a JSON representation of the model, include and exclude arguments as per dict().

encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().

property relevant_transformations: List[Transformation] | None#

Retrieves the transformations that need to be taken into account to realign original and test_case.

Returns:: list of transformations which shouldn’t be ignored
Return type:: Optional[List[Transformation]]

classmethod sort_transformations(v)#: Validator ensuring that transformations are in correct order

to_dict() → Dict[str, Any]#: Returns the dict version of sample.

classmethod update_forward_refs(**localns: Any) → None#: Try to update ForwardRefs on fields based on this Model, globalns and localns.