Robustness

ADD Abbreviations

This test replaces familiar words or expressions in texts with their abbreviations. These abbreviations are commonly used on social media platforms and some are generic. It evaluates the NLP model’s ability to handle text with such abbreviations.

alias_name: add_abbreviation

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_abbreviation:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_abbreviation test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Add Context

This test checks if the NLP model can handle input text with added context, such as a greeting or closing.

alias_name: add_context

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_context:
    min_pass_rate: 0.65
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
    parameters:
      ending_context: ['Bye', 'Reported']
      starting_context: ['Hi', 'Good morning', 'Hello']
      count: 1 # Defaults to 1

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_context test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• starting_context (<List[str]>): Phrases to be added at the start of inputs.
• ending_context (<List[str]>): Phrases to be added at the end of inputs.
• prob (float): Controls the proportion of words to be changed.
• count (int): Number of variations of sentence to be constructed.

Examples

Add Contraction

This test checks if the NLP model can handle input text if the data uses contractions instead of expanded forms.

alias_name: add_contraction

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_contraction:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_contraction test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Add OCR Typo

This test checks if the NLP model can handle input text with common ocr typos. A ocr typo dictionary is used to apply most common ocr typos to the input data.

alias_name: add_ocr_typo

Config

add_ocr_typo:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
    parameters:
        count: 1 # Defaults to 1

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_ocr_typo test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.
• count (int): Number of variations of sentence to be constructed.

Examples

Add Punctuation

This test checks if the NLP model can handle input text with sentences with a punctuation at the end. The added punctuation is randomly chosen from the list ['!', '?', ',', '.', '-', ':', ';']. If there already is a punctuation it is not changed.

alias_name: add_punctuation

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_punctuation:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_punctuation test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Add Slangs

This test involves substituting certain words (specifically nouns, adjectives, and adverbs) in the original text with their corresponding slang terms. The purpose is to assess the NLP model’s ability to handle input text that includes slang language.

alias_name: add_slangs

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_slangs:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_slangs test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Add Speech to Text Typo

This test evaluates the NLP model’s proficiency in handling input text that contains common typos resulting from Speech to Text conversion. A Speech to Text typo dictionary is utilized to apply the most frequent typos found in speech recognition output to the input data.

alias_name: add_speech_to_text_typo

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_speech_to_text_typo:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
    parameters:
      count: 1 # Defaults to 1

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_speech_to_text_typo test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.
• count (int): Number of variations of sentence to be constructed.

Examples

Add Typo

This test checks if the NLP model can handle input text with typos. A typo frequency dictionary is used to apply most common typos to the input data.

alias_name: add_typo

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

add_typo:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
    parameters:
      count: 1 # Defaults to 1

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during add_typo test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.
• count (int): Number of variations of sentence to be constructed.

Examples

Adjective Antonym Swap

This test provides a convenient way to convert adjectives into their equivalent antonyms.

alias_name: adjective_antonym_swap

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

adjective_antonym_swap:
    min_pass_rate: 0.8
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during adjective_antonym_swap test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Adjective Synonym Swap

This test provides a convenient way to convert adjectives into their equivalent synonyms.

alias_name: adjective_synonym_swap

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

adjective_synonym_swap:
    min_pass_rate: 0.8
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during adjective_synonym_swap test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

American to British

This test checks if the NLP model can handle input text with british accent. An accent dictionary is used to convert sentences into british accent.

alias_name: american_to_british

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

american_to_british:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during american_to_british test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

British to American

This test checks if the NLP model can handle input text with American accent. An accent dictionary is used to convert sentences into American accent.

alias_name: british_to_american

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

british_to_american:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during british_to_american test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Dyslexia Word Swap

This test assesses the NLP model’s capability to handle input text with common word swap errors associated with dyslexia. A Dyslexia Word Swap dictionary is employed to apply the most common word swap errors found in dyslexic writing to the input data.

alias_name: dyslexia_word_swap

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

dyslexia_word_swap:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during dyslexia_word_swap test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Lowercase

This test checks if the NLP model can handle input text that is in all lowercase. Like the uppercase test, this is important to ensure that your NLP model can handle input text in any case.

alias_name: lowercase

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

lowercase:
    min_pass_rate: 0.8
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during lowercase test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Multiple Perturbations

The multiple_perturbations test combines multiple tests into a single test by applying a sequence of perturbations to transform the given sentences. These perturbations are applied in a specific sequence.

Please note that this test is only supported for the text-classification, question-answering, and summarization tasks.

alias_name: multiple_perturbations

Config YAML format :

multiple_perturbations:
    min_pass_rate: 0.60
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
    perturbations1:
        lowercase
        add_ocr_typo
        titlecase

The perturbation set perturbations1 follows the transformation order: lowercase ➜ add_ocr_typo ➜ titlecase

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during multiple_perturbations test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Number To Word

This test provides a convenient way to convert numerical values in text into their equivalent words. It is particularly useful for applications that require handling or processing textual data containing numbers.

alias_name: number_to_word

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

number_to_word:
    min_pass_rate: 0.8
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during number_to_word test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Randomize Age

This test checks if the NLP model can handle age differences. The test replaces age statements like “x years old” with x ± random_amount. The value is set to 1 if its smaller than 0.

alias_name: randomize_age

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

randomize_age:
    min_pass_rate: 0.65
    prob: 1.0 # Defaults to 1.0, which means all statements will be transformed.
    parameters:
      random_amount: 5 # 
      count: 1 # Defaults to 1

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of statements to be changed during randomize_age test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• random_amount (int): Range of random value to be added/substracted from existing age value.
• prob (float): Controls the proportion of statements to be changed.
• count (int): Number of variations of sentence to be constructed.

Examples

Strip All Punctuation

This test checks if the NLP model can handle sentences with no punctuations.

alias_name: strip_all_punctuation

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

strip_all_punctuation:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during strip_all_punctuation test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Strip Punctuation

This test checks if the NLP model can handle sentences without punctuations at the end.

alias_name: strip_punctuation

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

strip_punctuation:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during strip_punctuation test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Swap Entities

This test shuffles the labeled entities in the input to test the models robustness.

alias_name: swap_entities

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

swap_entities:
    min_pass_rate: 0.8
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
    parameters:
      count: 1 # Defaults to 1

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during swap_entities test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.
• count (int): Number of variations of sentence to be constructed.

Examples

Titlecase

This test checks if the NLP model can handle input text that is in titlecase format, where the first letter of each word is capitalized.

alias_name: titlecase

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

titlecase:
    min_pass_rate: 0.7
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during titlecase test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples

Uppercase

This test checks if the NLP model can handle input text that is in all uppercase. Accidentally entering text in all caps is common, and you want to ensure that your NLP model can still process it correctly.

alias_name: uppercase

To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.

Config

uppercase:
    min_pass_rate: 0.8
    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.

You can adjust the level of transformation in the sentence by using the “prob” parameter, which controls the proportion of words to be changed during uppercase test.

• min_pass_rate (float): Minimum pass rate to pass the test.
• prob (float): Controls the proportion of words to be changed.

Examples