langtest.utils.custom_types.helpers.transformer_prompt_eval#

transformer_prompt_eval(eval_model, inputs: List[dict], predictions: List[dict]) bool#

Evaluates model predictions using a transformer-based language model.

Parameters:
  • eval_model – Transformer model for evaluation.

  • inputs (List[dict]) – List of input dictionaries.

  • predictions (List[dict]) – List of prediction dictionaries.

Returns:

True if the model’s prediction is correct, False otherwise.

Return type:

bool