Index

_ | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W

_

__generate_templates (TemplaticAugment attribute) __getattribute__() (ErrorsWithCodes method) __init__() (AbbreviationInsertion method) (AbsoluteGrading method) (AccuracyTestFactory method) (AddContext method) (AddContraction method) (AddNegation method) (AddOcrTypo method) (AddPunctuation method) (AddSlangifyTypo method) (AddSpeechToTextTypo method) (AddToxicWords method) (AddTypo method) (AdjectiveAntonymSwap method) (AdjectiveSynonymSwap method) (AugmentRobustness method), [1] (BaseAccuracy method) (BaseAugmentaion method) (BaseBias method) (BaseClincial method) (BaseDataset method) (BaseFairness method) (BaseFormatter method) (BaseGrammar method) (BaseIdeology method) (BasePerformance method) (BasePipeline method) (BaseQASample method) (BaseRepresentation method) (BaseRobustness method) (BaseSafetyTest method) (BaseSample method) (BaseSecurity method) (BaseSensitivity method) (BaseSycophancy method) (BaseTask method) (BaseToxicity method) (BenchmarkDatasets method) (BiasTestFactory method) (Brand2Generic method) (CheckpointManager method) (Clinical method) (ClinicalSample method) (ClinicalTestFactory method) (ConllDataset method) (Conversion method) (ConvertAccent method) (ConvertNumberToWord method) (CountryEconomicBias method) (CountryEconomicRepresentation method) (CrowsPairs method) (CrowsPairsSample method) (CSVDataset method) (DataAugmenter method) (DataFactory method) (DemographicBias method) (Disinformation method) (DisinformationSample method) (DisinformationTestFactory method) (DyslexiaWordSwap method) (EmbeddingDistance method) (EmbeddingPipeline method) (Errors method) (ErrorsWithCodes method) (EthnicityNameBias method) (EthnicityRepresentation method) (Factuality method) (FactualitySample method) (FactualityTestFactory method) (FairnessTestFactory method) (FillMask method) (FillMaskSample method) (Formatter method) (G2p method) (GenderClassifier method) (GenderPronounBias method) (GenderRepresentation method) (GeneralToxicity method) (Generic2Brand method) (GrammarTestFactory method) (Harness method) (HashableDict method) (HuggingFaceDataset method) (HuggingfaceEmbeddings method) (HuggingFacePipeline method) (Ideology method) (IdeologyTestFactory method) (ITests method) (JSONDataset method) (JSONLDataset method) (LabelRepresentation method) (LangTestCallback method) (Leaderboard method) (Legal method) (LegalSample method) (LegalTestFactory method) (LLMAnswerSample method) (LLMEval method) (LlmEval method) (Logger method) (LowerCase method) (MaxGenderF1Score method) (MaxGenderLLMEval method) (MaxGenderRougeScore method) (MaxScoreOutput method) (MaxScoreQASample method) (MaxScoreSample method) (MessageType method) (MinBLEUcore method) (MinEMcore method) (MinF1Score method) (MinGenderF1Score method) (MinGenderLLMEval method) (MinGenderRougeScore method) (MinMacroF1Score method) (MinMicroF1Score method) (MinPrecisionScore method) (MinRecallScore method) (MinROUGEcore method) (MinScoreOutput method) (MinScoreQASample method) (MinScoreSample method) (MinWeightedF1Score method) (Misuse method) (ModelAPI method) (MultiplePerturbations method) (NER method) (NERDataset method) (NEREnd2EndPipeline method) (NEROutput method) (NEROutputFormatter method) (NERPrediction method) (NERSample method) (NumberToWord method) (OpenaiEmbeddings method) (PandasDataset method) (Paraphrase method) (PerformanceTestFactory method) (PhoneFunctions method) (PoliticalCompass method) (Posology method) (PretrainedCustomModel method) (PretrainedJSLModel method) (PretrainedModel method) (PretrainedModelForClinical method), [1], [2] (PretrainedModelForCrowsPairs method), [1] (PretrainedModelForDisinformation method), [1], [2] (PretrainedModelForDisinformationTest method) (PretrainedModelForFactuality method), [1], [2] (PretrainedModelForFactualityTest method) (PretrainedModelForIdeology method), [1], [2] (PretrainedModelForLegal method), [1], [2] (PretrainedModelForNER method), [1], [2], [3], [4] (PretrainedModelForPolitical method) (PretrainedModelForQA method), [1], [2], [3] (PretrainedModelForSecurity method), [1], [2], [3] (PretrainedModelForSensitivity method), [1], [2] (PretrainedModelForSensitivityTest method) (PretrainedModelForStereoSet method) (PretrainedModelForSummarization method), [1], [2], [3] (PretrainedModelForSycophancy method), [1], [2] (PretrainedModelForSycophancyTest method) (PretrainedModelForTextClassification method), [1], [2], [3] (PretrainedModelForToxicity method), [1], [2], [3] (PretrainedModelForTranslation method), [1], [2], [3] (PretrainedModelForWinoBias method), [1], [2], [3] (PrometheusEval method) (PromptConfig method) (PromptInjection method) (PromptManager method) (PronunciationFunctions method) (QAFormatter method) (QASample method) (QuestionAnswering method) (RandomAge method) (RelativeGrading method) (ReligionBias method) (ReligionRepresentation method) (RepresentationOperation method) (RepresentationTestFactory method) (RobustnessTestFactory method) (SafetyTestFactory method) (Search method) (Security method) (SecuritySample method) (SecurityTestFactory method) (Sensitivity method) (SensitivitySample method) (SensitivityTestFactory method) (SequenceClassificationOutput method) (SequenceClassificationOutputFormatter method) (SequenceClassificationSample method) (SequenceLabel method) (SimplePromptTemplate method) (Span method) (Speed method) (SpeedTestSample method) (Stereoset method) (StereoSetSample method) (StereoSetTestFactory method) (StereoTypeTestFactory method) (StringDistance method) (StripAllPunctuation method) (StripPunctuation method) (Summarization method) (SummarizationSample method) (Summary method) (SwapEntities method) (Sycophancy method) (SycophancyMath method) (SycophancyNlp method) (SycophancySample method) (SycophancyTestFactory method) (SynteticDataset method) (TaskManager method) (TemplaticAugment method) (TestFactory method) (TestResultManager method) (TextClassification method) (TextGeneration method) (TextGenerationSample method) (TitleCase method) (Toxicity method) (ToxicitySample method) (ToxicityTestFactory method) (ToxicityTypes method) (Transformation method) (Translation method) (TranslationOutput method) (TranslationSample method) (UpperCase method) (Warnings method) (WinoBias method) (WinoBiasSample method) (WordFunctions method)

__show_templates (TemplaticAugment attribute) __task (TemplaticAugment attribute) __templates (TemplaticAugment attribute) _file_path (CSVDataset attribute) _is_eval() (FactualitySample method)

A

AbbreviationInsertion (class in langtest.transform.robustness) AbsoluteGrading (class in langtest.metrics.prometheus_eval) AccuracyTestFactory (class in langtest.transform.accuracy) actual_result (SensitivitySample attribute) actual_results (MaxScoreSample attribute) (MinScoreSample attribute) (SecuritySample attribute) (SequenceClassificationSample attribute) (SummarizationSample attribute) (SycophancySample attribute) add_console_handler() (Logger method) add_custom_data() (in module langtest.transform.custom_data) add_custom_representation() (RepresentationOperation static method) add_prompt() (PromptManager method) add_report() (Summary method) add_spaces_around_punctuation() (TemplaticAugment static method) AddContext (class in langtest.transform.robustness) AddContraction (class in langtest.transform.robustness) AddNegation (class in langtest.transform.sensitivity) AddOcrTypo (class in langtest.transform.robustness) AddPunctuation (class in langtest.transform.robustness) AddSlangifyTypo (class in langtest.transform.robustness) AddSpeechToTextTypo (class in langtest.transform.robustness) AddToxicWords (class in langtest.transform.sensitivity) AddTypo (class in langtest.transform.robustness) AdjectiveAntonymSwap (class in langtest.transform.robustness) AdjectiveSynonymSwap (class in langtest.transform.robustness) alias_name (AddNegation attribute) (AddToxicWords attribute) (BaseAccuracy attribute) (BaseBias attribute) (BaseFairness attribute) (BaseIdeology attribute) (BaseRepresentation attribute) (BaseRobustness attribute) (BaseSensitivity attribute) (BaseSycophancy attribute) (CountryEconomicRepresentation attribute) (EthnicityRepresentation attribute) (GenderRepresentation attribute) (LabelRepresentation attribute) (LLMEval attribute) (MaxGenderF1Score attribute) (MaxGenderLLMEval attribute) (MaxGenderRougeScore attribute) (MinBLEUcore attribute) (MinEMcore attribute) (MinF1Score attribute) (MinGenderF1Score attribute) (MinGenderLLMEval attribute) (MinGenderRougeScore attribute) (MinMacroF1Score attribute) (MinMicroF1Score attribute) (MinPrecisionScore attribute) (MinRecallScore attribute) (MinROUGEcore attribute) (MinWeightedF1Score attribute) (ReligionRepresentation attribute) (SensitivityTestFactory attribute) (SycophancyMath attribute) (SycophancyTestFactory attribute) answer (LLMAnswerSample attribute) article_sent (FactualitySample attribute) async_run() (AbbreviationInsertion class method) (AddContext class method) (AddContraction class method) (AddNegation class method) (AddOcrTypo class method) (AddPunctuation class method) (AddSlangifyTypo class method) (AddSpeechToTextTypo class method) (AddToxicWords class method) (AddTypo class method) (AdjectiveAntonymSwap class method) (AdjectiveSynonymSwap class method) (BaseAccuracy class method) (BaseBias class method) (BaseClincial class method) (BaseFairness class method) (BaseIdeology class method) (BasePerformance class method) (BaseRepresentation class method) (BaseRobustness class method) (BaseSafetyTest class method) (BaseSecurity class method) (BaseSensitivity class method) (BaseSycophancy class method) (BaseToxicity class method) (Brand2Generic class method) (ConvertAccent class method) (CountryEconomicBias class method) (CountryEconomicRepresentation class method) (DemographicBias class method) (DyslexiaWordSwap class method) (EthnicityNameBias class method) (EthnicityRepresentation class method) (FactualityTestFactory method) (GenderPronounBias class method) (GenderRepresentation class method) (GeneralToxicity class method) (Generic2Brand class method) (LabelRepresentation class method) (LegalTestFactory method) (LLMEval class method) (LowerCase class method) (MaxGenderF1Score class method) (MaxGenderLLMEval class method) (MaxGenderRougeScore class method) (MinBLEUcore class method) (MinEMcore class method) (MinF1Score class method) (MinGenderF1Score class method) (MinGenderLLMEval class method) (MinGenderRougeScore class method) (MinMacroF1Score class method) (MinMicroF1Score class method) (MinPrecisionScore class method) (MinRecallScore class method) (MinROUGEcore class method) (MinWeightedF1Score class method) (Misuse class method) (MultiplePerturbations class method) (NumberToWord class method) (PoliticalCompass class method) (PromptInjection class method) (RandomAge class method) (ReligionBias class method) (ReligionRepresentation class method) (Speed class method) (StereoSetTestFactory static method) (StereoTypeTestFactory static method) (StripAllPunctuation class method) (StripPunctuation class method) (SwapEntities class method) (SycophancyMath class method) (SycophancyNlp class method) (TestFactory class method) (TitleCase class method) (ToxicityTypes class method) (UpperCase class method)

augment() (DataAugmenter method) (Harness method) (NEREnd2EndPipeline method) AugmentRobustness (class in langtest.augmentation.base) available_embedding_distance() (EmbeddingDistance class method) available_string_distance() (StringDistance class method) available_tests() (AccuracyTestFactory static method) (BaseClincial class method) (BiasTestFactory static method) (Brand2Generic class method) (ClinicalTestFactory class method) (DemographicBias class method) (DisinformationTestFactory class method) (FactualityTestFactory class method) (FairnessTestFactory static method) (Generic2Brand class method) (GrammarTestFactory static method) (Harness static method) (IdeologyTestFactory static method) (ITests static method) (LegalTestFactory class method) (PerformanceTestFactory class method) (RepresentationTestFactory static method) (RobustnessTestFactory static method) (SafetyTestFactory class method) (SecurityTestFactory class method) (SensitivityTestFactory static method) (StereoSetTestFactory class method) (StereoTypeTestFactory class method) (SycophancyTestFactory static method) (ToxicityTestFactory static method)

B

BadChunkingOptionError BaseAccuracy (class in langtest.transform.accuracy) BaseAugmentaion (class in langtest.augmentation.base) BaseBias (class in langtest.transform.bias) BaseClincial (class in langtest.transform.clinical) BaseDataset (class in langtest.datahandler.datasource) BaseFairness (class in langtest.transform.fairness) BaseFormatter (class in langtest.datahandler.format) BaseGrammar (class in langtest.transform.grammar) BaseIdeology (class in langtest.transform.ideology) BasePerformance (class in langtest.transform.performance) BasePipeline (class in langtest.pipelines.embedding) BaseQASample (class in langtest.utils.custom_types.sample) BaseRepresentation (class in langtest.transform.representation) BaseRobustness (class in langtest.transform.robustness)

BaseSafetyTest (class in langtest.transform.safety) BaseSample (class in langtest.utils.custom_types.sample) BaseSecurity (class in langtest.transform.security) BaseSensitivity (class in langtest.transform.sensitivity) BaseSycophancy (class in langtest.transform.sycophancy) BaseTask (class in langtest.tasks.task) BaseToxicity (class in langtest.transform.toxicity) BenchmarkDatasets (class in langtest.utils.config_utils) BiasTestFactory (class in langtest.transform.bias) Brand2Generic (class in langtest.transform.clinical) build_dataset() (in module langtest.utils.hf_utils) build_input() (SensitivitySample method) build_prompt() (SensitivitySample method) build_qa_input() (in module langtest.utils.custom_types.helpers) build_qa_prompt() (in module langtest.utils.custom_types.helpers)

C

calculate_f1_score() (in module langtest.utils.util_metrics) call_add_custom_bias() (TestFactory static method) case (LegalSample attribute) category (ClinicalSample attribute) (CrowsPairsSample attribute) (FactualitySample attribute) (LegalSample attribute) (LLMAnswerSample attribute) (SecuritySample attribute) (SensitivitySample attribute) (StereoSetSample attribute) (SummarizationSample attribute) (SycophancySample attribute) (TaskManager property) (ToxicitySample attribute) (WinoBiasSample attribute) category_wise() (Leaderboard method) chat_completion_api() (in module langtest.modelhandler.lmstudio_modelhandler) check_memory() (in module langtest.metrics.prometheus_eval) check_name() (in module langtest.transform.utils) CheckpointManager (class in langtest.utils.checkpoints) classification_report() (in module langtest.utils.util_metrics) clean_input() (in module langtest.utils.hf_utils) clear() (HashableDict method) Clinical (class in langtest.tasks.task) ClinicalSample (class in langtest.utils.custom_types.sample) ClinicalTestFactory (class in langtest.transform.clinical) cmd() (NEREnd2EndPipeline method) color_cells() (in module langtest.utils.report_utils) column_mapping() (BaseTask method) (Clinical method) (CrowsPairs method) (Disinformation method) (Factuality method) (FillMask method) (Ideology method) (Legal method) (NER method) (QuestionAnswering method) (Security method) (Sensitivity method) (Stereoset method) (Summarization method) (Sycophancy method) (TextClassification method) (TextGeneration method) (Toxicity method) (Translation method) (WinoBias method) COLUMN_NAMES (CSVDataset attribute) ColumnNameError compare() (NEREnd2EndPipeline method) compare_generations_overlap() (in module langtest.transform.utils) completion (ToxicitySample attribute) completion_toxicity (ToxicitySample attribute) compute_ner_metrics() (in module langtest.pipelines.utils.metrics) config (AugmentRobustness attribute) (SensitivitySample attribute) ConfigError configure() (Harness method) ConllDataset (class in langtest.datahandler.datasource) construct() (BaseQASample class method) (BaseSample class method) (ClinicalSample class method) (Conversion class method) (CrowsPairsSample class method) (DisinformationSample class method) (FactualitySample class method) (FillMaskSample class method) (LegalSample class method) (LLMAnswerSample class method) (MaxScoreOutput class method) (MaxScoreQASample class method) (MaxScoreSample class method) (MessageType class method) (MinScoreOutput class method) (MinScoreQASample class method) (MinScoreSample class method) (NEROutput class method) (NERPrediction class method) (NERSample class method) (PromptConfig class method) (QASample class method) (SecuritySample class method) (SensitivitySample class method) (SequenceClassificationOutput class method) (SequenceClassificationSample class method) (SequenceLabel class method) (Span class method) (SpeedTestSample class method) (StereoSetSample class method) (SummarizationSample class method) (SycophancySample class method) (TextGenerationSample class method) (ToxicitySample class method) (Transformation class method) (TranslationOutput class method) (TranslationSample class method) (WinoBiasSample class method)

Conversion (class in langtest.prompts) convert_ns_to_unit() (SpeedTestSample method) ConvertAccent (class in langtest.transform.robustness) ConvertNumberToWord (class in langtest.utils.number_to_word) copy() (BaseQASample method) (BaseSample method) (ClinicalSample method) (Conversion method) (CrowsPairsSample method) (DisinformationSample method) (FactualitySample method) (FillMaskSample method) (HashableDict method) (LegalSample method) (LLMAnswerSample method) (MaxScoreOutput method) (MaxScoreQASample method) (MaxScoreSample method) (MessageType method) (MinScoreOutput method) (MinScoreQASample method) (MinScoreSample method) (NEROutput method) (NERPrediction method) (NERSample method) (PromptConfig method) (QASample method) (SecuritySample method) (SensitivitySample method) (SequenceClassificationOutput method) (SequenceClassificationSample method) (SequenceLabel method) (Span method) (SpeedTestSample method) (StereoSetSample method) (SummarizationSample method) (SycophancySample method) (TextGenerationSample method) (ToxicitySample method) (Transformation method) (TranslationOutput method) (TranslationSample method) (WinoBiasSample method) correct_conlusion (LegalSample attribute) correct_sent (FactualitySample attribute) CountryEconomicBias (class in langtest.transform.bias) CountryEconomicRepresentation (class in langtest.transform.representation) create_dirs() (in module langtest.utils.custom_types.helpers) create_folder() (in module langtest.leaderboard) (in module langtest.utils.custom_types.helpers) create_leaderboard() (in module langtest.leaderboard) create_sample() (BaseTask class method) (Clinical method) (CrowsPairs method) (Disinformation method) (Factuality method) (FillMask class method) (Ideology method) (Legal method) (NER method) (QuestionAnswering method) (Security method) (Sensitivity method) (Stereoset method) (Summarization method) (Sycophancy method) (TaskManager method) (TextClassification method) (TextGeneration method) (Toxicity method) (Translation method) (WinoBias method) create_terminology() (in module langtest.transform.utils) CrowsPairs (class in langtest.tasks.task) CrowsPairsSample (class in langtest.utils.custom_types.sample) CSVDataset (class in langtest.datahandler.datasource) custom_wise() (Leaderboard method)

D

DataAugmenter (class in langtest.augmentation.augmenter) DataFactory (class in langtest.datahandler.datasource) dataset_name (ClinicalSample attribute) (FactualitySample attribute) (LegalSample attribute) (SecuritySample attribute) (SensitivitySample attribute) (SummarizationSample attribute) (SycophancySample attribute) (ToxicitySample attribute) default() (Leaderboard method) delimiter (CSVDataset attribute) DemographicBias (class in langtest.transform.clinical) device (HuggingfaceEmbeddings attribute) diagnosis (ClinicalSample attribute) dict() (BaseQASample method) (BaseSample method) (ClinicalSample method) (Conversion method) (CrowsPairsSample method) (DisinformationSample method) (FactualitySample method) (FillMaskSample method) (LegalSample method) (LLMAnswerSample method) (MaxScoreOutput method) (MaxScoreQASample method) (MaxScoreSample method) (MessageType method) (MinScoreOutput method) (MinScoreQASample method) (MinScoreSample method) (NEROutput method) (NERPrediction method) (NERSample method) (PromptConfig method) (QASample method) (SecuritySample method) (SensitivitySample method) (SequenceClassificationOutput method) (SequenceClassificationSample method) (SequenceLabel method) (Span method) (SpeedTestSample method) (StereoSetSample method) (SummarizationSample method) (SycophancySample method) (TextGenerationSample method) (ToxicitySample method) (Transformation method) (TranslationOutput method) (TranslationSample method) (WinoBiasSample method)

Disinformation (class in langtest.tasks.task) DisinformationSample (class in langtest.utils.custom_types.sample) DisinformationTestFactory (class in langtest.transform.disinformation) divide_into_batches() (in module langtest.utils.checkpoints) DyslexiaWordSwap (class in langtest.transform.robustness)

E

edit_testcases() (Harness method) EmbeddingDistance (class in langtest.metrics.embedding_distance) EmbeddingPipeline (class in langtest.pipelines.embedding) end() (NEREnd2EndPipeline method) enword() (ConvertNumberToWord method) Errors (class in langtest.errors) ErrorsWithCodes (class in langtest.errors) EthnicityNameBias (class in langtest.transform.bias) EthnicityRepresentation (class in langtest.transform.representation) evaluate() (LlmEval method) (NEREnd2EndPipeline method) (PrometheusEval method) evaluate_batch() (LlmEval method) (PrometheusEval method) evaluate_example() (LlmEval method) evaluate_response() (PrometheusEval method) expected_result (SensitivitySample attribute)

expected_results (MaxScoreSample attribute) (MinScoreSample attribute) (SecuritySample attribute) (SequenceClassificationSample attribute) (SummarizationSample attribute) (SycophancySample attribute) export() (DataFactory method) export_data() (BaseDataset method) (ConllDataset method) (CSVDataset method) (HuggingFaceDataset method) (JSONDataset method) (JSONLDataset method) (PandasDataset method) (SynteticDataset method) extend() (DataAugmenter method) extract_data_with_equal_proportion() (SynteticDataset static method) extract_variable_names() (TemplaticAugment static method)

F

Factuality (class in langtest.tasks.task) FactualitySample (class in langtest.utils.custom_types.sample) FactualityTestFactory (class in langtest.transform.factuality) FairnessTestFactory (class in langtest.transform.fairness) FillMask (class in langtest.tasks.task) FillMaskSample (class in langtest.utils.custom_types.sample) filter_curated_bias() (DataFactory class method) filter_unique_samples() (in module langtest.transform.utils)

fix() (AugmentRobustness method), [1] (BaseAugmentaion method), [1] (TemplaticAugment method) foreach_stack() (NEREnd2EndPipeline method) format() (SimplePromptTemplate method) Formatter (class in langtest.datahandler.format) from_prompt_configs() (PromptManager class method) fromkeys() (HashableDict method)

G

G2p (class in langtest.utils.SoundsLikeFunctions) GatedRepoAccessError gender (NERSample attribute) GenderClassifier (class in langtest.utils.gender_classifier) GenderPronounBias (class in langtest.transform.bias) GenderRepresentation (class in langtest.transform.representation) GeneralToxicity (class in langtest.transform.toxicity) generate() (Harness method) generate_age() (SycophancyNlp static method) generate_folder_key() (in module langtest.leaderboard) generate_name() (SycophancyMath static method) (SycophancyNlp static method) generate_pronunciation() (PronunciationFunctions static method) generate_school() (SycophancyNlp static method) generate_store_testcases() (in module langtest.leaderboard) generate_text() (SycophancyNlp static method) generated_results() (Harness method) Generic2Brand (class in langtest.transform.clinical) get() (HashableDict method) get_aligned_span_pairs() (NERSample method) get_country_economic_representation_dict() (RepresentationOperation static method) get_dataset_dict() (BenchmarkDatasets class method) get_datasets() (BenchmarkDatasets class method) get_embedding() (HuggingfaceEmbeddings method) (OpenaiEmbeddings method) get_entity_representation_proportions() (RepresentationOperation static method) get_ethnicity_representation_dict() (RepresentationOperation static method) get_example (MessageType property) get_examples (Conversion property) (PromptConfig property) get_gendered_data() (FairnessTestFactory static method) get_input_variables (PromptConfig property) get_label_representation_dict() (RepresentationOperation static method) get_leaderboard() (Harness method) get_lm_studio_model_name() (in module langtest.leaderboard) get_logger() (Logger method) get_model_n_tokenizer() (in module langtest.utils.hf_utils) get_parameters() (in module langtest.leaderboard)

get_prompt() (AbsoluteGrading method) (PromptManager method) (RelativeGrading method) get_religion_name_representation_dict() (RepresentationOperation static method) get_sample_class (TaskManager property) get_sample_cls (BaseTask property) (Clinical property) (CrowsPairs property) (Disinformation property) (Factuality property) (FillMask property) (Ideology property) (Legal property) (NER property) (QuestionAnswering property) (Security property) (Sensitivity property) (Stereoset property) (Summarization property) (Sycophancy property) (TextClassification property) (TextGeneration property) (Toxicity property) (Translation property) (WinoBias property) get_score_rubric() (AbsoluteGrading method) (RelativeGrading method) get_store_path() (in module langtest.leaderboard) get_substitution_names() (in module langtest.transform.utils) get_template (MessageType property) (PromptConfig property) GrammarTestFactory (class in langtest.transform.grammar) ground_truth (SycophancySample attribute) group1bsub() (ConvertNumberToWord method) group1sub() (ConvertNumberToWord method) group2sub() (ConvertNumberToWord method) group3sub() (ConvertNumberToWord method) group_entities() (PretrainedModelForNER method), [1] gt (SycophancySample attribute)

H

h_report (AugmentRobustness attribute) Harness (class in langtest.langtest) HashableDict (class in langtest.utils.custom_types.helpers) hub (PretrainedModelForQA attribute) (SensitivitySample attribute)

HuggingFaceDataset (class in langtest.datahandler.datasource) HuggingfaceEmbeddings (class in langtest.embeddings.huggingface) HuggingFacePipeline (class in langtest.utils.hf_utils) hundfn() (ConvertNumberToWord method) hundsub() (ConvertNumberToWord method)

I

Ideology (class in langtest.tasks.task) IdeologyTestFactory (class in langtest.transform.ideology) ignored_predictions (NERSample property) import_edited_testcases() (Harness method) incorrect_sent (FactualitySample attribute) index (NEREnd2EndPipeline property) inplace() (DataAugmenter method) input (NEREnd2EndPipeline property) input_process() (PretrainedModelForNER method) irrelevant_transformations (BaseSample property) (MaxScoreSample property) (MinScoreSample property) (NERSample property) (SequenceClassificationSample property) is_classifier() (PretrainedModelForTextClassification static method) is_ner_annotator() (PretrainedModelForNER static method) is_pass() (BaseSample method) (FactualitySample method), [1] (MaxScoreQASample method) (MaxScoreSample method), [1] (MinScoreQASample method) (MinScoreSample method), [1] (NERSample method) (QASample method) (SensitivitySample method), [1] (SequenceClassificationSample method), [1] (SpeedTestSample method) (SummarizationSample method) (SycophancySample method), [1] (ToxicitySample method) (TranslationSample method)

is_pass_embedding_distance() (in module langtest.utils.custom_types.helpers) is_pass_llm_eval() (in module langtest.utils.custom_types.helpers) is_pass_prometheus_eval() (in module langtest.utils.custom_types.helpers) is_pass_string_distance() (in module langtest.utils.custom_types.helpers) is_pass_with_ground_truth() (SycophancySample method) is_pass_without_ground_truth() (SycophancySample method) is_translator() (PretrainedModelForTranslation static method) items() (HashableDict method) ITests (class in langtest.transform.base)

J

json() (BaseQASample method) (BaseSample method) (ClinicalSample method) (Conversion method) (CrowsPairsSample method) (DisinformationSample method) (FactualitySample method) (FillMaskSample method) (LegalSample method) (LLMAnswerSample method) (MaxScoreOutput method) (MaxScoreQASample method) (MaxScoreSample method) (MessageType method) (MinScoreOutput method) (MinScoreQASample method) (MinScoreSample method) (NEROutput method) (NERPrediction method) (NERSample method) (PromptConfig method) (QASample method) (SecuritySample method) (SensitivitySample method) (SequenceClassificationOutput method) (SequenceClassificationSample method) (SequenceLabel method) (Span method) (SpeedTestSample method) (StereoSetSample method) (SummarizationSample method) (SycophancySample method) (TextGenerationSample method) (ToxicitySample method) (Transformation method) (TranslationOutput method) (TranslationSample method) (WinoBiasSample method)

JSONDataset (class in langtest.datahandler.datasource) JSONLDataset (class in langtest.datahandler.datasource)

K

keys() (HashableDict method)

kwargs (PretrainedModelForQA attribute)

L

LabelRepresentation (class in langtest.transform.representation) labels (PretrainedModelForTextClassification property), [1] langtest module langtest.augmentation module langtest.augmentation.augmenter module langtest.augmentation.base module langtest.callback module langtest.config module langtest.datahandler module langtest.datahandler.dataset_info module langtest.datahandler.datasource module langtest.datahandler.format module langtest.embeddings module langtest.embeddings.huggingface module langtest.embeddings.openai module langtest.errors module langtest.langtest module langtest.leaderboard module langtest.logger module langtest.metrics module langtest.metrics.embedding_distance module langtest.metrics.llm_eval module langtest.metrics.prometheus_eval module langtest.metrics.string_distance module langtest.modelhandler module langtest.modelhandler.custom_modelhandler module langtest.modelhandler.jsl_modelhandler module langtest.modelhandler.llm_modelhandler module langtest.modelhandler.lmstudio_modelhandler module langtest.modelhandler.modelhandler module langtest.modelhandler.spacy_modelhandler module langtest.modelhandler.transformers_modelhandler module langtest.pipelines module langtest.pipelines.embedding module langtest.pipelines.transformers module langtest.pipelines.transformers.ner_pipeline module langtest.pipelines.utils module langtest.pipelines.utils.data_helpers module langtest.pipelines.utils.data_helpers.ner_dataset module langtest.pipelines.utils.metrics module langtest.prompts module langtest.tasks module langtest.tasks.task module langtest.transform module langtest.transform.accuracy module langtest.transform.base module langtest.transform.bias module langtest.transform.clinical module langtest.transform.constants module langtest.transform.custom_data module langtest.transform.disinformation module langtest.transform.factuality module langtest.transform.fairness module langtest.transform.grammar module langtest.transform.ideology module langtest.transform.legal module langtest.transform.performance module langtest.transform.representation module langtest.transform.robustness module langtest.transform.safety module langtest.transform.security module langtest.transform.sensitivity module langtest.transform.stereoset module langtest.transform.stereotype module langtest.transform.sycophancy module

langtest.transform.toxicity module langtest.transform.utils module langtest.utils module langtest.utils.benchmark_utils module langtest.utils.checkpoints module langtest.utils.config_utils module langtest.utils.custom_types module langtest.utils.custom_types.helpers module langtest.utils.custom_types.output module langtest.utils.custom_types.predictions module langtest.utils.custom_types.sample module langtest.utils.gender_classifier module langtest.utils.hf_utils module langtest.utils.lib_manager module langtest.utils.number_to_word module langtest.utils.report_utils module langtest.utils.SoundsLikeFunctions module langtest.utils.util_metrics module LangTestCallback (class in langtest.callback) Leaderboard (class in langtest.utils.benchmark_utils) Legal (class in langtest.tasks.task) legal_claim (LegalSample attribute) legal_conclusion_A (LegalSample attribute) legal_conclusion_B (LegalSample attribute) LegalSample (class in langtest.utils.custom_types.sample) LegalTestFactory (class in langtest.transform.legal) llm_prompt_eval() (in module langtest.utils.custom_types.helpers) LLMAnswerSample (class in langtest.utils.custom_types.sample) LlmEval (class in langtest.metrics.llm_eval) LLMEval (class in langtest.transform.accuracy) load() (DataFactory method) (Harness class method) load_batches() (CheckpointManager method) load_checkpoint() (CheckpointManager method) load_checkpoints() (Harness class method) load_config() (DataAugmenter method) load_curated_bias() (DataFactory class method) load_data() (BaseDataset method) (ConllDataset method) (CSVDataset method) (HuggingFaceDataset method) (JSONDataset method) (JSONLDataset method) (PandasDataset method) (SynteticDataset method) load_data_from_file() (Summary method) load_model() (BaseTask class method) (Clinical class method) (CrowsPairs class method) (Disinformation class method) (Factuality class method) (FillMask class method) (Ideology class method) (Legal class method) (ModelAPI method) (NER class method) (PretrainedJSLModel class method) (PretrainedModel class method) (PretrainedModelForClinical class method), [1], [2] (PretrainedModelForCrowsPairs class method), [1] (PretrainedModelForDisinformation class method), [1], [2] (PretrainedModelForDisinformationTest class method) (PretrainedModelForFactuality class method), [1], [2] (PretrainedModelForFactualityTest class method) (PretrainedModelForIdeology class method), [1], [2] (PretrainedModelForLegal class method), [1], [2] (PretrainedModelForNER class method), [1], [2], [3], [4] (PretrainedModelForPolitical class method) (PretrainedModelForQA class method), [1], [2], [3] (PretrainedModelForSecurity class method), [1], [2], [3] (PretrainedModelForSensitivity class method), [1], [2] (PretrainedModelForSensitivityTest class method) (PretrainedModelForStereoSet class method) (PretrainedModelForSummarization class method), [1], [2], [3] (PretrainedModelForSycophancy class method), [1], [2] (PretrainedModelForSycophancyTest class method) (PretrainedModelForTextClassification class method), [1], [2], [3] (PretrainedModelForToxicity class method), [1], [2], [3] (PretrainedModelForTranslation class method), [1], [2], [3] (PretrainedModelForWinoBias class method), [1], [2], [3] (QuestionAnswering class method) (Security class method) (Sensitivity class method) (Stereoset class method) (Summarization class method) (Sycophancy class method) (TextClassification class method) (TextGeneration class method) (Toxicity class method) (Translation class method) (WinoBias class method) load_old_testcases() (in module langtest.leaderboard) load_raw() (DataFactory method) load_raw_data() (BaseDataset method) (ConllDataset method) (CSVDataset method) (HuggingFaceDataset method) (JSONDataset method) (JSONLDataset method) (PandasDataset method) (SynteticDataset method) load_remaining_batch() (CheckpointManager method) load_synthetic_math_data() (SynteticDataset method) load_synthetic_nlp_data() (SynteticDataset method) log_verbosity_handler() (in module langtest.utils.lib_manager) Logger (class in langtest.logger) login_with_token() (in module langtest.utils.hf_utils) loss_diff (SensitivitySample attribute) LowerCase (class in langtest.transform.robustness)

M

mask1 (CrowsPairsSample attribute) (StereoSetSample attribute) mask2 (CrowsPairsSample attribute) (StereoSetSample attribute) masked_text (WinoBiasSample attribute) max_prop (AugmentRobustness attribute) MaxGenderF1Score (class in langtest.transform.fairness) MaxGenderLLMEval (class in langtest.transform.fairness) MaxGenderRougeScore (class in langtest.transform.fairness) MaxScoreOutput (class in langtest.utils.custom_types.output) MaxScoreQASample (class in langtest.utils.custom_types.sample) MaxScoreSample (class in langtest.utils.custom_types.sample) mean_pooling() (HuggingfaceEmbeddings method) merge_artifacts() (NEREnd2EndPipeline method) message (ConfigError attribute) MessageType (class in langtest.prompts) millfn() (ConvertNumberToWord method) MinBLEUcore (class in langtest.transform.accuracy) MinEMcore (class in langtest.transform.accuracy) MinF1Score (class in langtest.transform.accuracy) MinGenderF1Score (class in langtest.transform.fairness) MinGenderLLMEval (class in langtest.transform.fairness) MinGenderRougeScore (class in langtest.transform.fairness) MinMacroF1Score (class in langtest.transform.accuracy) MinMicroF1Score (class in langtest.transform.accuracy) MinPrecisionScore (class in langtest.transform.accuracy) MinRecallScore (class in langtest.transform.accuracy) MinROUGEcore (class in langtest.transform.accuracy) MinScoreOutput (class in langtest.utils.custom_types.output) MinScoreQASample (class in langtest.utils.custom_types.sample) MinScoreSample (class in langtest.utils.custom_types.sample) MinWeightedF1Score (class in langtest.transform.accuracy) Misuse (class in langtest.transform.safety) mlflow_report() (in module langtest.utils.report_utils) model (HuggingfaceEmbeddings attribute) (PretrainedCustomModel attribute) (PretrainedJSLModel attribute) (PretrainedModel attribute) (PretrainedModelForQA attribute) (PretrainedModelForSensitivity attribute) (PretrainedModelForTextClassification attribute) model() (TaskManager method) model_conclusion (LegalSample attribute) model_report() (in module langtest.utils.report_utils) model_response() (Harness method) ModelAPI (class in langtest.modelhandler.modelhandler) module langtest langtest.augmentation langtest.augmentation.augmenter langtest.augmentation.base langtest.callback langtest.config langtest.datahandler langtest.datahandler.dataset_info langtest.datahandler.datasource langtest.datahandler.format langtest.embeddings langtest.embeddings.huggingface langtest.embeddings.openai langtest.errors langtest.langtest langtest.leaderboard langtest.logger langtest.metrics langtest.metrics.embedding_distance langtest.metrics.llm_eval langtest.metrics.prometheus_eval langtest.metrics.string_distance langtest.modelhandler langtest.modelhandler.custom_modelhandler langtest.modelhandler.jsl_modelhandler langtest.modelhandler.llm_modelhandler langtest.modelhandler.lmstudio_modelhandler langtest.modelhandler.modelhandler langtest.modelhandler.spacy_modelhandler langtest.modelhandler.transformers_modelhandler langtest.pipelines langtest.pipelines.embedding langtest.pipelines.transformers langtest.pipelines.transformers.ner_pipeline langtest.pipelines.utils langtest.pipelines.utils.data_helpers langtest.pipelines.utils.data_helpers.ner_dataset langtest.pipelines.utils.metrics langtest.prompts langtest.tasks langtest.tasks.task langtest.transform langtest.transform.accuracy langtest.transform.base langtest.transform.bias langtest.transform.clinical langtest.transform.constants langtest.transform.custom_data langtest.transform.disinformation langtest.transform.factuality langtest.transform.fairness langtest.transform.grammar langtest.transform.ideology langtest.transform.legal langtest.transform.performance langtest.transform.representation langtest.transform.robustness langtest.transform.safety langtest.transform.security langtest.transform.sensitivity langtest.transform.stereoset langtest.transform.stereotype langtest.transform.sycophancy langtest.transform.toxicity langtest.transform.utils langtest.utils langtest.utils.benchmark_utils langtest.utils.checkpoints langtest.utils.config_utils langtest.utils.custom_types langtest.utils.custom_types.helpers langtest.utils.custom_types.output langtest.utils.custom_types.predictions langtest.utils.custom_types.sample langtest.utils.gender_classifier langtest.utils.hf_utils langtest.utils.lib_manager langtest.utils.number_to_word langtest.utils.report_utils langtest.utils.SoundsLikeFunctions langtest.utils.util_metrics

mro() (ErrorsWithCodes method) multi_dataset_multi_model_report() (in module langtest.utils.report_utils) multi_dataset_report() (in module langtest.utils.report_utils) multi_model_report() (in module langtest.utils.report_utils) MultiplePerturbations (class in langtest.transform.robustness)

N

NER (class in langtest.tasks.task) NERDataset (class in langtest.pipelines.utils.data_helpers.ner_dataset) NEREnd2EndPipeline (class in langtest.pipelines.transformers.ner_pipeline) NEROutput (class in langtest.utils.custom_types.output) NEROutputFormatter (class in langtest.datahandler.format) NERPrediction (class in langtest.utils.custom_types.predictions) NERSample (class in langtest.utils.custom_types.sample)

new_data() (DataAugmenter method) new_sample() (TemplaticAugment method) next() (NEREnd2EndPipeline method) None (BasePerformance attribute) (BaseSecurity attribute) number_to_words() (ConvertNumberToWord method) NumberToWord (class in langtest.transform.robustness) NumOutOfRangeError

O

on_epoch_begin() (LangTestCallback method) on_epoch_end() (LangTestCallback method) on_evaluate() (LangTestCallback method) on_init_end() (LangTestCallback method) on_log() (LangTestCallback method) on_predict() (LangTestCallback method) on_prediction_step() (LangTestCallback method) on_save() (LangTestCallback method) on_step_begin() (LangTestCallback method) on_step_end() (LangTestCallback method)

on_substep_end() (LangTestCallback method) on_train_begin() (LangTestCallback method) on_train_end() (LangTestCallback method) op1 (SensitivitySample attribute) op2 (SensitivitySample attribute) OpenaiEmbeddings (class in langtest.embeddings.openai) options (SensitivitySample attribute) original (SensitivitySample attribute) (SummarizationSample attribute) original_question (SycophancySample attribute) output() (SycophancySample method)

P

PandasDataset (class in langtest.datahandler.datasource) Paraphrase (class in langtest.transform.grammar) partial() (SimplePromptTemplate method) pass_custom_data() (Harness method) patient_info_A (ClinicalSample attribute) patient_info_B (ClinicalSample attribute) perfectHomophones() (Search static method) PerformanceTestFactory (class in langtest.transform.performance) perturbed_question (SycophancySample attribute) PhoneFunctions (class in langtest.utils.SoundsLikeFunctions) political_report() (in module langtest.utils.report_utils) PoliticalCompass (class in langtest.transform.ideology) pop() (HashableDict method) popitem() (HashableDict method) Posology (class in langtest.transform.clinical) predict() (ModelAPI method) (PretrainedJSLModel method) (PretrainedModel method) (PretrainedModelForClinical method), [1], [2] (PretrainedModelForCrowsPairs method), [1] (PretrainedModelForDisinformation method), [1], [2] (PretrainedModelForDisinformationTest method) (PretrainedModelForFactuality method), [1], [2] (PretrainedModelForFactualityTest method) (PretrainedModelForIdeology method), [1], [2] (PretrainedModelForLegal method), [1], [2] (PretrainedModelForNER method), [1], [2], [3], [4], [5] (PretrainedModelForPolitical method) (PretrainedModelForQA method), [1], [2], [3], [4], [5] (PretrainedModelForSecurity method), [1], [2], [3] (PretrainedModelForSensitivity method), [1], [2] (PretrainedModelForSensitivityTest method) (PretrainedModelForStereoSet method) (PretrainedModelForSummarization method), [1], [2], [3] (PretrainedModelForSycophancy method), [1], [2] (PretrainedModelForSycophancyTest method) (PretrainedModelForTextClassification method), [1], [2], [3], [4] (PretrainedModelForToxicity method), [1], [2], [3] (PretrainedModelForTranslation method), [1], [2], [3] (PretrainedModelForWinoBias method), [1], [2], [3] predict_raw() (PretrainedModel method) (PretrainedModelForClinical method), [1] (PretrainedModelForDisinformation method), [1] (PretrainedModelForFactuality method), [1] (PretrainedModelForIdeology method), [1] (PretrainedModelForLegal method), [1] (PretrainedModelForNER method), [1], [2], [3], [4] (PretrainedModelForQA method), [1] (PretrainedModelForSecurity method), [1] (PretrainedModelForSensitivity method), [1] (PretrainedModelForSummarization method), [1] (PretrainedModelForSycophancy method), [1] (PretrainedModelForTextClassification method), [1], [2] (PretrainedModelForToxicity method), [1] (PretrainedModelForTranslation method), [1] (PretrainedModelForWinoBias method), [1] prepare_accuracy_summary() (in module langtest.leaderboard) prepare_llm_evaluation_data() (in module langtest.utils.custom_types.helpers) prepare_model_response() (in module langtest.utils.custom_types.helpers) (TestResultManager method) prepare_robustness_summary() (in module langtest.leaderboard) PretrainedCustomModel (class in langtest.modelhandler.custom_modelhandler) PretrainedJSLModel (class in langtest.modelhandler.jsl_modelhandler) PretrainedModel (class in langtest.modelhandler.lmstudio_modelhandler) PretrainedModelForClinical (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForCrowsPairs (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.transformers_modelhandler)

PretrainedModelForDisinformation (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForDisinformationTest (class in langtest.modelhandler.custom_modelhandler) PretrainedModelForFactuality (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForFactualityTest (class in langtest.modelhandler.custom_modelhandler) PretrainedModelForIdeology (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForLegal (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForNER (class in langtest.modelhandler.jsl_modelhandler) (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.spacy_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForPolitical (class in langtest.modelhandler.custom_modelhandler) PretrainedModelForQA (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForSecurity (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForSensitivity (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForSensitivityTest (class in langtest.modelhandler.custom_modelhandler) PretrainedModelForStereoSet (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForSummarization (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForSycophancy (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForSycophancyTest (class in langtest.modelhandler.custom_modelhandler) PretrainedModelForTextClassification (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.jsl_modelhandler) (class in langtest.modelhandler.spacy_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForToxicity (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForTranslation (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.jsl_modelhandler) (class in langtest.modelhandler.spacy_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) PretrainedModelForWinoBias (class in langtest.modelhandler.custom_modelhandler) (class in langtest.modelhandler.llm_modelhandler) (class in langtest.modelhandler.lmstudio_modelhandler) (class in langtest.modelhandler.transformers_modelhandler) print3() (in module langtest.utils.number_to_word) process() (Formatter static method) PrometheusEval (class in langtest.metrics.prometheus_eval) prompt (SecuritySample attribute) (ToxicitySample attribute) prompt_style() (PromptConfig method) prompt_toxicity (ToxicitySample attribute) PromptConfig (class in langtest.prompts) PromptInjection (class in langtest.transform.security) PromptManager (class in langtest.prompts) pronunciation() (WordFunctions static method) PronunciationFunctions (class in langtest.utils.SoundsLikeFunctions)

Q

QAFormatter (class in langtest.datahandler.format) QASample (class in langtest.utils.custom_types.sample)

question (LLMAnswerSample attribute) QuestionAnswering (class in langtest.tasks.task)

R

ran_pass (SensitivitySample attribute) rand_range() (SynteticDataset static method) RandomAge (class in langtest.transform.robustness) read_config() (in module langtest.config) realigned_spans (NERSample property) reevaluate() (NEREnd2EndPipeline method) RelativeGrading (class in langtest.metrics.prometheus_eval) relevant_transformations (BaseSample property) (MaxScoreSample property) (MinScoreSample property) (NERSample property) (SequenceClassificationSample property) ReligionBias (class in langtest.transform.bias) ReligionRepresentation (class in langtest.transform.representation) remove_console_handler() (Logger method) remove_file_handler() (Logger method) remove_punctuation() (FactualitySample method), [1] renamed_extensions() (PandasDataset class method) reorder_columns() (in module langtest.leaderboard) replace_values() (SycophancyNlp static method) (SynteticDataset static method) report() (Harness method) RepresentationOperation (class in langtest.transform.utils) RepresentationTestFactory (class in langtest.transform.representation) reset() (PromptManager method) result (FactualitySample attribute) retrain() (NEREnd2EndPipeline method) RobustnessTestFactory (class in langtest.transform.robustness) run() (AbbreviationInsertion static method) (AccuracyTestFactory class method) (AddContext static method) (AddContraction static method) (AddNegation static method) (AddOcrTypo static method) (AddPunctuation static method) (AddSlangifyTypo static method) (AddSpeechToTextTypo static method) (AddToxicWords static method) (AddTypo static method) (AdjectiveAntonymSwap static method) (AdjectiveSynonymSwap static method) (BaseAccuracy static method) (BaseBias static method) (BaseClincial static method) (BaseFairness static method) (BaseGrammar static method) (BaseIdeology static method) (BasePerformance static method) (BaseQASample method) (BaseRepresentation class method) (BaseRobustness static method) (BaseSafetyTest class method) (BaseSecurity static method) (BaseSensitivity static method) (BaseSycophancy static method) (BaseToxicity static method) (BiasTestFactory class method) (Brand2Generic static method) (ClinicalTestFactory class method) (ConvertAccent static method) (CountryEconomicBias static method) (CountryEconomicRepresentation static method) (DemographicBias static method) (DisinformationTestFactory class method) (DyslexiaWordSwap static method) (EthnicityNameBias static method) (EthnicityRepresentation static method) (FactualitySample method), [1] (FactualityTestFactory class method) (FairnessTestFactory class method) (GenderPronounBias static method) (GenderRepresentation static method) (GeneralToxicity static method) (Generic2Brand static method) (GrammarTestFactory class method) (Harness method) (IdeologyTestFactory class method) (ITests class method) (LabelRepresentation static method) (LegalTestFactory class method) (LLMEval static method) (LowerCase static method) (MaxGenderF1Score static method) (MaxGenderLLMEval static method) (MaxGenderRougeScore static method) (MaxScoreQASample method) (MinBLEUcore static method) (MinEMcore static method) (MinF1Score method) (MinGenderF1Score static method) (MinGenderLLMEval static method) (MinGenderRougeScore static method) (MinMacroF1Score method) (MinMicroF1Score method) (MinPrecisionScore method) (MinRecallScore method) (MinROUGEcore static method) (MinScoreQASample method) (MinWeightedF1Score method) (Misuse class method) (MultiplePerturbations static method) (NumberToWord static method) (Paraphrase static method) (PerformanceTestFactory class method) (PoliticalCompass static method) (PromptInjection static method) (QASample method) (RandomAge static method) (ReligionBias static method) (ReligionRepresentation static method) (RepresentationTestFactory class method) (RobustnessTestFactory class method) (SafetyTestFactory class method) (SecurityTestFactory class method) (SensitivitySample method), [1] (SensitivityTestFactory class method) (Speed static method) (StereoSetTestFactory class method) (StereoTypeTestFactory class method) (StripAllPunctuation static method) (StripPunctuation static method) (SummarizationSample method) (SwapEntities static method) (SycophancyMath static method) (SycophancyNlp static method) (SycophancySample method), [1] (SycophancyTestFactory class method) (TestFactory static method) (TitleCase static method) (ToxicitySample method) (ToxicityTestFactory class method) (ToxicityTypes static method) (TranslationSample method) (UpperCase static method)

run_store_checkpoints() (in module langtest.leaderboard) run_time (SpeedTestSample attribute)

S

SafetyTestFactory (class in langtest.transform.safety) sample_class (Clinical attribute) (CrowsPairs attribute) (Disinformation attribute) (Factuality attribute) (Ideology attribute) (Legal attribute) (NER attribute) (QuestionAnswering attribute) (Security attribute) (Sensitivity attribute) (Stereoset attribute) (Summarization attribute) (Sycophancy attribute) (TextClassification attribute) (Toxicity attribute) (Translation attribute) (WinoBias attribute) save() (DataAugmenter method) (Harness method) save_all_batches() (CheckpointManager method) save_checkpoint() (CheckpointManager method) save_file() (in module langtest.leaderboard) save_format() (in module langtest.utils.report_utils) save_summary() (Summary method) script_name (NEREnd2EndPipeline property) Search (class in langtest.utils.SoundsLikeFunctions) search_sample_results() (TemplaticAugment static method) Security (class in langtest.tasks.task) SecuritySample (class in langtest.utils.custom_types.sample) SecurityTestFactory (class in langtest.transform.security) Sensitivity (class in langtest.tasks.task) SensitivitySample (class in langtest.utils.custom_types.sample) SensitivityTestFactory (class in langtest.transform.sensitivity) sentence (CrowsPairsSample attribute) (StereoSetSample attribute) SequenceClassificationOutput (class in langtest.utils.custom_types.output) SequenceClassificationOutputFormatter (class in langtest.datahandler.format) SequenceClassificationSample (class in langtest.utils.custom_types.sample) SequenceLabel (class in langtest.utils.custom_types.predictions) set_level() (Logger method) setdefault() (HashableDict method) setup() (NEREnd2EndPipeline method) SimplePromptTemplate (class in langtest.utils.custom_types.helpers) sort_by_appearance() (NEROutput class method) sort_transformations() (BaseSample class method) (MaxScoreSample class method) (MinScoreSample class method) (NERSample class method) (SequenceClassificationSample class method)

Span (class in langtest.utils.custom_types.helpers) Speed (class in langtest.transform.performance) SpeedTestSample (class in langtest.utils.custom_types.sample) split_wise() (Leaderboard method) start() (NEREnd2EndPipeline method) state (ClinicalSample attribute) (FactualitySample attribute) (LegalSample attribute) (SecuritySample attribute) (SensitivitySample attribute) (SummarizationSample attribute) (SycophancySample attribute) (ToxicitySample attribute) Stereoset (class in langtest.tasks.task) StereoSetSample (class in langtest.utils.custom_types.sample) StereoSetTestFactory (class in langtest.transform.stereoset) StereoTypeTestFactory (class in langtest.transform.stereotype) str_to_sample() (TemplaticAugment method) StringDistance (class in langtest.metrics.string_distance) StripAllPunctuation (class in langtest.transform.robustness) StripPunctuation (class in langtest.transform.robustness) suggestions() (AugmentRobustness method), [1] Summarization (class in langtest.tasks.task) SummarizationSample (class in langtest.utils.custom_types.sample) Summary (class in langtest.utils.benchmark_utils) supported_tasks (BaseRepresentation attribute) (CountryEconomicRepresentation attribute) (EthnicityRepresentation attribute) (GenderRepresentation attribute) (LabelRepresentation attribute) (LLMEval attribute) (MaxGenderLLMEval attribute) (MaxGenderRougeScore attribute) (MinGenderLLMEval attribute) (MinGenderRougeScore attribute) (Misuse attribute) (ReligionRepresentation attribute) SwapEntities (class in langtest.transform.robustness) swapped_result (FactualitySample attribute) Sycophancy (class in langtest.tasks.task) SycophancyMath (class in langtest.transform.sycophancy) SycophancyNlp (class in langtest.transform.sycophancy) SycophancySample (class in langtest.utils.custom_types.sample) SycophancyTestFactory (class in langtest.transform.sycophancy) SynteticDataset (class in langtest.datahandler.datasource)

T

task (AugmentRobustness attribute) (ClinicalSample attribute) (CSVDataset attribute) (FactualitySample attribute) (LegalSample attribute) (SecuritySample attribute) (SensitivitySample attribute) (SequenceClassificationSample attribute) (SummarizationSample attribute) (SycophancySample attribute) (TemplaticAugment property) (ToxicitySample attribute) task_name (TaskManager property) TaskManager (class in langtest.tasks.task) templates (TemplaticAugment property) TemplaticAugment (class in langtest.augmentation.base) tenfn() (ConvertNumberToWord method) tensub() (ConvertNumberToWord method) test() (NEREnd2EndPipeline method) test_case (LLMAnswerSample attribute) (SensitivitySample attribute) (SummarizationSample attribute) (SycophancySample attribute) test_categories() (TestFactory class method) test_scenarios() (TestFactory class method) test_type (ClinicalSample attribute) (CrowsPairsSample attribute) (FactualitySample attribute) (LegalSample attribute) (LLMAnswerSample attribute) (SecuritySample attribute) (SensitivitySample attribute) (StereoSetSample attribute) (SummarizationSample attribute) (SycophancySample attribute) (ToxicitySample attribute) (WinoBiasSample attribute) test_wise() (Leaderboard method) testcases() (Harness method) TestFactory (class in langtest.transform.base) TestResultManager (class in langtest.utils.custom_types.helpers) TextClassification (class in langtest.tasks.task) TextGeneration (class in langtest.tasks.task) TextGenerationSample (class in langtest.utils.custom_types.sample) TitleCase (class in langtest.transform.robustness) to_conll() (BaseFormatter static method) (NEROutputFormatter static method) (QAFormatter static method) (SequenceClassificationOutputFormatter static method) to_csv() (BaseFormatter static method) (NEROutputFormatter static method) (QAFormatter static method) (SequenceClassificationOutputFormatter static method) to_dict() (BaseSample method) (ClinicalSample method) (CrowsPairsSample method) (FactualitySample method), [1] (FillMaskSample method) (LegalSample method) (LLMAnswerSample method) (MaxScoreQASample method) (MaxScoreSample method) (MinScoreQASample method) (MinScoreSample method) (NERSample method) (QASample method) (SecuritySample method) (SensitivitySample method), [1] (SequenceClassificationSample method) (SpeedTestSample method) (StereoSetSample method) (SummarizationSample method) (SycophancySample method), [1] (TextGenerationSample method) (ToxicitySample method) (TranslationSample method) (WinoBiasSample method) to_jsonl() (QAFormatter method) to_str_list() (MaxScoreOutput method) (MinScoreOutput method) (NEROutput method) (SequenceClassificationOutput method) (TranslationOutput method) tokenizer (HuggingfaceEmbeddings attribute) (PretrainedModelForSensitivity attribute) total (SpeedTestSample attribute) total_time() (SpeedTestSample method) Toxicity (class in langtest.tasks.task) ToxicitySample (class in langtest.utils.custom_types.sample) ToxicityTestFactory (class in langtest.transform.toxicity) ToxicityTypes (class in langtest.transform.toxicity) train() (NEREnd2EndPipeline method) transform() (AbbreviationInsertion static method) (AccuracyTestFactory method) (AddContext static method) (AddContraction static method) (AddNegation static method) (AddOcrTypo static method) (AddPunctuation static method) (AddSlangifyTypo static method) (AddSpeechToTextTypo static method) (AddToxicWords static method) (AddTypo static method) (AdjectiveAntonymSwap static method) (AdjectiveSynonymSwap static method) (BaseAccuracy class method) (BaseBias method) (BaseClincial static method) (BaseFairness static method) (BaseIdeology method) (BasePerformance static method) (BaseQASample method) (BaseRepresentation class method) (BaseRobustness static method) (BaseSafetyTest method) (BaseSecurity static method) (BaseSensitivity static method) (BaseSycophancy static method) (BaseToxicity static method) (BiasTestFactory method) (Brand2Generic static method) (ClinicalTestFactory method) (ConvertAccent static method) (CountryEconomicBias static method) (CountryEconomicRepresentation class method) (DemographicBias static method) (DisinformationTestFactory method) (DyslexiaWordSwap static method) (EthnicityNameBias static method) (EthnicityRepresentation class method) (FactualityTestFactory method) (FairnessTestFactory method) (GenderPronounBias static method) (GenderRepresentation class method) (GeneralToxicity static method) (Generic2Brand static method) (GrammarTestFactory method) (IdeologyTestFactory method) (ITests method) (LabelRepresentation class method) (LegalTestFactory method) (LLMEval class method) (LowerCase static method) (MaxGenderF1Score class method) (MaxGenderLLMEval class method) (MaxGenderRougeScore class method) (MaxScoreQASample method) (MinBLEUcore class method) (MinBLEUcore method) (MinEMcore class method) (MinEMcore method) (MinF1Score class method) (MinGenderF1Score class method) (MinGenderLLMEval class method) (MinGenderRougeScore class method) (MinMacroF1Score class method) (MinMacroF1Score method) (MinMicroF1Score class method) (MinPrecisionScore class method) (MinPrecisionScore method) (MinRecallScore class method) (MinRecallScore method) (MinROUGEcore class method) (MinROUGEcore method) (MinScoreQASample method) (MinWeightedF1Score class method) (MinWeightedF1Score method) (Misuse method) (MultiplePerturbations static method) (NumberToWord static method) (PerformanceTestFactory method) (PoliticalCompass static method) (QASample method) (RandomAge static method) (ReligionBias static method) (ReligionRepresentation class method) (RepresentationTestFactory method) (RobustnessTestFactory method) (SafetyTestFactory method) (SecurityTestFactory method) (SensitivitySample method) (SensitivityTestFactory method) (Speed static method) (StereoSetTestFactory method) (StereoTypeTestFactory method) (StripAllPunctuation static method) (StripPunctuation static method) (SummarizationSample method) (SwapEntities static method) (SycophancyMath static method) (SycophancyNlp static method) (SycophancySample method) (SycophancyTestFactory method) (TestFactory static method) (TitleCase static method) (ToxicityTestFactory method) (ToxicityTypes static method) (UpperCase static method)

transform_time (SpeedTestSample attribute) Transformation (class in langtest.utils.custom_types.helpers) transformer_prompt_eval() (in module langtest.utils.custom_types.helpers) Translation (class in langtest.tasks.task) TranslationOutput (class in langtest.utils.custom_types.output) TranslationSample (class in langtest.utils.custom_types.sample) treatment_plan_A (ClinicalSample attribute) treatment_plan_B (ClinicalSample attribute) try_import_lib() (in module langtest.utils.lib_manager)

U

unitfn() (ConvertNumberToWord method) unitsub() (ConvertNumberToWord method) unstressed_phone() (PhoneFunctions static method) update() (HashableDict method) update_config() (in module langtest.config) update_forward_refs() (BaseQASample class method) (BaseSample class method) (ClinicalSample class method) (Conversion class method) (CrowsPairsSample class method) (DisinformationSample class method) (FactualitySample class method) (FillMaskSample class method) (LegalSample class method) (LLMAnswerSample class method) (MaxScoreOutput class method) (MaxScoreQASample class method) (MaxScoreSample class method) (MessageType class method) (MinScoreOutput class method) (MinScoreQASample class method) (MinScoreSample class method) (NEROutput class method) (NERPrediction class method) (NERSample class method) (PromptConfig class method) (QASample class method) (SecuritySample class method) (SensitivitySample class method) (SequenceClassificationOutput class method) (SequenceClassificationSample class method) (SequenceLabel class method) (Span class method) (SpeedTestSample class method) (StereoSetSample class method) (SummarizationSample class method) (SycophancySample class method) (TextGenerationSample class method) (ToxicitySample class method) (Transformation class method) (TranslationOutput class method) (TranslationSample class method) (WinoBiasSample class method)

update_leaderboard() (in module langtest.leaderboard) update_status() (CheckpointManager method) update_summary() (in module langtest.leaderboard) upload_file_to_hub() (Harness method) upload_folder_to_hub() (Harness method) UpperCase (class in langtest.transform.robustness)

V

validate_input() (EmbeddingDistance method) (StringDistance method)

values() (HashableDict method)

W

Warnings (class in langtest.errors) WinoBias (class in langtest.tasks.task)

WinoBiasSample (class in langtest.utils.custom_types.sample) WordFunctions (class in langtest.utils.SoundsLikeFunctions) write_config() (in module langtest.config)