langtest.metrics.prometheus_eval.AbsoluteGrading#

class AbsoluteGrading(instruction: str, response: str, reference_answer: str, criteria_description: Dict[str, str])#

Bases: object

Class for absolute grading of the Prometheus model.

Absolute Grading (Direct Assessment) Prometheus requires 4 components in the input: An instruction, a response to evaluate, a score rubric, and a reference answer. You could refer to the prompt format below. You should fill in the instruction, response, reference answer, criteria description, and score description for score in range of n number or True or False.

__init__(instruction: str, response: str, reference_answer: str, criteria_description: Dict[str, str]) → None#

Methods

__init__(instruction, response, ...)
get_prompt()	Get the prompt for the model.
get_score_rubric()	Get the score rubric for the model.

Attributes

instruction
response
reference_answer
criteria_description

get_prompt() → str#

Get the prompt for the model.

Returns:

The prompt for the model.

get_score_rubric() → Dict[str, str]#

Get the score rubric for the model.

Returns:

The score rubric for the model.