The following table gives an overview of the different tutorial notebooks. In this section we have miscellaneous notebooks to enhance user experience with different features.
Miscellaneous Notebooks
| Tutorial Description | Hub | Task | Open In Colab |
|---|---|---|---|
| Add Custom Data: In this tutorial, we explore the process of incorporating custom data for bias and representation testing from JSON files. | Spacy/Hugging Face | NER/Text-Classification |  |
| Augmentation Control: In this tutorial, we explore the process of how to contol the augmentations. | John Snow Labs | NER | |
| Multiple Model Comparison: In this tutorial, we compared different language models on various taks. | Hugging Face/John Snow Labs/Spacy | NER/Text-Classification | |
| Custom Hub: In this tutorial, we compared different language models on various taks. | Custom | Text-Classification | |
| Report Exportation: In this tutorial, we explored different ways in which user can export their report. | Spacy | NER | |
| Editing Testcases: In this section, we discussed how to edit test cases in the Harness class. | Hugging Face | NER | |
| Evaluation Metrics: In this section, we discussed different evatuation metrics for evauate Quetion-Answering models. | OpenAI | Question-Answering | |
| HuggingFace Datasets: In this section, we dive into testing of HuggingFace Models for different HuggingFace Datasets. | Hugging Face/Spacy/OpenAI | NER/Text-Classification/Question-Answering/Summarization | |
| Custom Column Loading: In this section, we discussed how to load a csv data for different task such as QA, Text-Classification, NER, Summarization. | Hugging Face/OpenAI | NER/Text-Classification/Question-Answering/Summarization | |
Multiple Variations: In this section, we discussed Multiple variations for a perturbation. Some of the robustness tests take a parameter count which specifies how many instances/variations of a sentence to produce. |
John Snow Labs | NER | |
| Templatic Augmentation: In this section, we discussed about Templatic Augmentation which is a technique that allows you to generate new training data by applying a set of predefined templates to the original training data. | John Snow Labs | NER | |
| LangTestCallback: In this section, we discussed how to utilize the LangTestCallback funtion while training an NER transformers model. | Hugging Face | NER | |
| LangTestCallback: In this section, we discussed how to utilize the LangTestCallback funtion while training an Text Classification transformers model. | Hugging Face | Text-Classification | |
| Multiple_dataset: In this section, we discussed how to evaluate multiple datasets for a particular model. | OpenAI | Question-Answering | |
| Generic API-Based Model: In this section, we discussed how to test API-based models hosted using Ollama, vLLM, and other tools. | Web | Question-Answering | |
| Data Augmenter: In this Notebook, we can allows for streamlined and harness-free data augmentation, making it simpler to enhance your datasets and improve model robustness. | - | NER | |
| Multi-Dataset Prompt Configs: In this Notebook, we discussed about optimized prompt handling for multiple datasets, allowing users to add custom prompts for each dataset, enabling seamless integration and efficient testing. | OpenAI | Question-Answering | |
| Multi-Model, Multi-Dataset: In this Notebook, we discussed about testing on multiple models with multiple datasets, allowing users to allows for comprehensive comparisons and performance assessments in a streamlined manner. | OpenAI | Question-Answering | |
| Evaluation_with_Prometheus_Eval: In this Notebook, we disscussed about integrating the Prometheus model to langtest brings enhanced evaluation capabilities, providing more detailed and insightful metrics for model performance assessment. | OpenAI | Question-Answering | |
| Misuse_Test_with_Prometheus_evaluation: In this Notebook, we discussed about new safety testing features to identify and mitigate potential misuse and safety issues in your models | OpenAI | Question-Answering | |
| Visual_QA: In this Notebook, we discussed about the visual question answering tests to evaluate how models handle both visual and textual inputs, offering a deeper understanding of their versatility. | OpenAI | Visual-Question-Answering (visualqa) | |
| Add_New_Lines_and_Tabs_Tests: In this Notebook, we discussed about new tests like inserting new lines and tab characters into text inputs, challenging your models to handle structural changes without compromising accuracy. | Hugging Face/John Snow Labs/Spacy | Text-Classification/Question-Answering/Summarization | |
| Safety_Tests_With_PromptGuard: In this Notebook, we discussed about evaluating prompts before they are sent to large language models (LLMs), ensuring harmful or unethical outputs are avoided with PromptGuard. | Hugging Face/John Snow Labs/Spacy | Text-Classification/Question-Answering/Summarization | |