langtest.utils.custom_types.sample.MaxScoreQASample#

class MaxScoreQASample(*, original_question: str, original_context: str, options: str = None, test_type: str = None, perturbed_question: str = None, perturbed_context: str = None, expected_results: Result = None, actual_results: Result = None, dataset_name: str = None, category: str = None, state: str = None, task: str = 'question-answering', test_case: str = None, config: str = None, distance_result: float = None, eval_model: str | tuple = None, ran_pass: bool = None, metric_name: str = None, gender: str = None, loaded_fields: Dict[str, Any] = None, feedback: str = None)#

Bases: QASample

A class representing a sample for question answering task with maximum score comparison.

__init__(**data)#

Constructor method

Methods

__init__(**data)

Constructor method

construct([_fields_set])

Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.

copy(*[, include, exclude, update, deep])

Duplicate a model, optionally choose which fields to include, exclude and change.

dict(*[, include, exclude, by_alias, ...])

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

from_orm(obj)

is_pass()

Checks if the sample has passed the evaluation.

json(*[, include, exclude, by_alias, ...])

Generate a JSON representation of the model, include and exclude arguments as per dict().

parse_file(path, *[, content_type, ...])

parse_obj(obj)

parse_raw(b, *[, content_type, encoding, ...])

run(model, **kwargs)

Runs the original and perturbed sentences through the model

schema([by_alias, ref_template])

schema_json(*[, by_alias, ref_template])

to_dict()

Returns the dictionary version of the sample.

transform(func, params, prob[, perturbations])

Transforms the original question and context using the specified function.

update_forward_refs(**localns)

Try to update ForwardRefs on fields based on this Model, globalns and localns.

validate(value)

Attributes

classmethod construct(_fields_set: SetStr | None = None, **values: Any) → Model#

Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed. Behaves as if Config.extra = ‘allow’ was set since it adds all passed values

copy(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, update: DictStrAny | None = None, deep: bool = False) → Model#

Duplicate a model, optionally choose which fields to include, exclude and change.

Parameters:

  • include – fields to include in new model

  • exclude – fields to exclude from new model, as with values this takes precedence over include

  • update – values to change/add in the new model. Note: the data is not validated before creating the new model: you should trust this data

  • deep – set to True to make a deep copy of the model

Returns:

new model instance

dict(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, by_alias: bool = False, skip_defaults: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny#

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

is_pass() → bool#

Checks if the sample has passed the evaluation.

json(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, by_alias: bool = False, skip_defaults: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Callable[[Any], Any] | None = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode#

Generate a JSON representation of the model, include and exclude arguments as per dict().

encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().

run(model, **kwargs)#

Runs the original and perturbed sentences through the model

to_dict() → Dict[str, Any]#

Returns the dictionary version of the sample.

Returns:

The dictionary representation of the sample.

Return type:

Dict[str, Any]

transform(func: Callable, params: Dict, prob: float, perturbations=None, **kwargs)#

Transforms the original question and context using the specified function.

Parameters:

  • func (function) – The transformation function to apply.

  • params (dict) – Additional parameters for the transformation function.

  • prob (float) – Probability of applying the transformation.

  • **kwargs – Additional keyword arguments for the transformation function.

Returns:

None

classmethod update_forward_refs(**localns: Any) → None#

Try to update ForwardRefs on fields based on this Model, globalns and localns.