langtest.utils.custom_types.helpers.transformer_prompt_eval#

transformer_prompt_eval(eval_model, inputs: List[dict], predictions: List[dict], template: str | None = None, grades: List[str] | None = None) → bool#

Evaluates model predictions using a transformer-based language model.

Parameters:

• eval_model – Transformer model for evaluation.

• inputs (List[dict]) – List of input dictionaries.

• predictions (List[dict]) – List of prediction dictionaries.

Returns:

True if the model’s prediction is correct, False otherwise.

Return type:

bool