langtest.utils.custom_types.helpers#

Functions

build_qa_input([context, question, options])

Builds the input data for a question-answering model.

build_qa_prompt(input_data[, dataset_name])

Builds the prompt data for a question-answering model.

create_dirs(default_location, *args, **kwargs)

Make directories.

create_folder(default_location, data_dict)

Create the folder based on the data_dict.

is_pass_embedding_distance(answer, ...[, ...])

Check if the sample passes based on embedding distance.

is_pass_llm_eval(eval_model, dataset_name, ...)

Determines whether the model's prediction passes the Language Model Metric (LLM) evaluation.

is_pass_prometheus_eval(task, ...[, ...])

Check if the sample passes based on prometheus evaluation.

is_pass_string_distance(answer, prediction, ...)

Check if the sample passes based on string distance.

llm_prompt_eval(eval_model, dataset_name, ...)

Evaluates model predictions using the Language Model Metric (LLM) with prompt-based evaluation.

prepare_llm_evaluation_data(...)

Prepares inputs and predictions in the required format for language model evaluation.

prepare_model_response(data)

transformer_prompt_eval(eval_model, inputs, ...)

Evaluates model predictions using a transformer-based language model.

Classes

HashableDict

A hashable dictionary with support for nested dictionaries and lists.

SimplePromptTemplate(input_variables, template)

Simple prompt template for formatting messages with variables.

Span(*, start, end, word)

Representation of a text's slice

TestResultManager()

Transformation(*, original_span, new_span[, ...])

Helper object keeping track of an alteration performed on a piece of text.