2.0.0

📢 Highlights

🌟 LangTest 2.0.0 Release by John Snow Labs

We’re thrilled to announce the latest release of LangTest, introducing remarkable features that elevate its capabilities and user-friendliness. This update brings a host of enhancements:

• 🔬 Model Benchmarking: Conducted tests on diverse models across datasets for insights into performance.

• 🔌 Integration: LM Studio with LangTest: Offline utilization of Hugging Face quantized models for local NLP tests.

• 🚀 Text Embedding Benchmark Pipelines: Streamlined process for evaluating text embedding models via CLI.

• 📊 Compare Models Across Multiple Benchmark Datasets: Simultaneous evaluation of model efficacy across diverse datasets.

• 🤬 Custom Toxicity Checks: Tailor evaluations to focus on specific types of toxicity, offering detailed analysis in targeted areas of concern, such as obscenity, insult, threat, identity attack, and targeting based on sexual orientation, while maintaining broader toxicity detection capabilities.

• Implemented LRU caching within the run method to optimize model prediction retrieval for duplicate records, enhancing runtime efficiency.

🔥 Key Enhancements:

🚀 Model Benchmarking: Exploring Insights into Model Performance

As part of our ongoing Model Benchmarking initiative, we’re excited to share the results of our comprehensive tests on a diverse range of models across various datasets, focusing on evaluating their performance on top of accuracy and robustness .

Key Highlights:

• Comprehensive Evaluation: Our rigorous testing methodology covered a wide array of models, providing a holistic view of their performance across diverse datasets and tasks.

• Insights into Model Behavior: Through this initiative, we’ve gained valuable insights into the strengths and weaknesses of different models, uncovering areas where even large language models exhibit limitations.

Go to: Leaderboard

⚡Integration: LM Studio with LangTest

The integration of LM Studio with LangTest enables offline utilization of Hugging Face quantized models, offering users a seamless experience for conducting various NLP tests locally.

Key Benefits:

• Offline Accessibility: With this integration, users can now leverage Hugging Face quantized models for NLP tasks like Question Answering, Summarization, Fill Mask, and Text Generation directly within LangTest, even without an internet connection.

• Enhanced Control: LM Studio’s user-friendly interface provides users with enhanced control over their testing environment, allowing for greater customization and optimization of test parameters.

How it Works:

Simply integrate LM Studio with LangTest to unlock offline utilization of Hugging Face quantized models for your NLP testing needs., below is the demo video for help.

https://github.com/JohnSnowLabs/langtest/assets/101416953/d1f288d4-1d96-4d9c-9db2-4f87a9e69019

🚀Text Embedding Benchmark Pipelines with CLI (LangTest + LlamaIndex)

Text embedding benchmarks play a pivotal role in assessing the performance of text embedding models across various tasks, crucial for evaluating the quality of text embeddings used in Natural Language Processing (NLP) applications.

The LangTest CLI for Text Embedding Benchmark Pipelines facilitates evaluation of HuggingFace’s embedding models on a retrieval task on the Paul Graham dataset. It starts by initializing each embedding model and creating a context for vector operations. Then, it sets up a vector store index for efficient similarity searches. Next, it configures a query engine and a retriever, retrieving the top similar items based on a predefined parameter. Evaluation is then conducted using Mean Reciprocal Rank (MRR) and Hit Rate metrics, measuring the retriever’s performance. Perturbations such as typos and word swaps are applied to test the retriever’s robustness.

Key Features:

• Simplified Benchmarking: Run text embedding benchmark pipelines effortlessly through our CLI, eliminating the need for complex setup or manual intervention.

• Versatile Model Evaluation: Evaluate the performance of text embedding models across diverse tasks, empowering users to assess the quality and effectiveness of different models for their specific use cases.

How it Works:

1. Set API Keys as enviroment variable.
2. Example Usage (Single Model): python -m langtest benchmark embeddings --model TaylorAI/bge-micro --hub huggingface
3. Example Usage (Multiple Models): python -m langtest benchmark embeddings --model "TaylorAI/bge-micro,TaylorAI/gte-tiny,intfloat/e5-small" --hub huggingface

📊 Compare Models Across Multiple Benchmark Datasets

Previously, when testing your model, you were limited to evaluating its performance on one dataset at a time. With this update, we’ve introduced the flexibility to assess your model’s efficacy across diverse benchmark datasets simultaneously, empowering you to gain deeper insights into its performance under various conditions and data distributions.

Key Benefits:

• Comprehensive Model Evaluation: Evaluate your model’s performance across multiple benchmark datasets in a single run, allowing for a more comprehensive assessment of its capabilities and generalization across different data domains.

• Time Efficiency: Streamline your testing process by eliminating the need to conduct separate evaluations for each dataset, saving valuable time and resources.

• Enhanced Flexibility: Choose from a range of benchmark datasets to test your model against, catering to specific use cases and ensuring robust performance evaluation across diverse scenarios.

How it Works:

To leverage this new feature and compare models across different benchmark datasets, simply pass multiple datasets as input in data_source as a dict:

harness = Harness(
    task="question-answering",
    model={"model": "gpt-3.5-turbo-instruct", "hub": "openai"},
    data=[
        {"data_source": "BoolQ", "split": "test-tiny"}, #Add your datsets here
        {"data_source": "NQ-open", "split": "test-tiny"},
        {"data_source": "MedQA", "split": "test-tiny"},
        {"data_source": "LogiQA", "split": "test-tiny"},
        ],
    config={
        "model_parameters": {
            "max_tokens": 32,
            "temperature": 0.2,
        },
        "evaluation": {
            "metric": "llm_eval",
            "model": "gpt-3.5-turbo-instruct",
            "hub": "openai",
        },
        "tests": {
            "defaults": {"min_pass_rate": 0.65},
            "robustness": {
                "uppercase": {"min_pass_rate": 0.75},
                "add_typo": {"min_pass_rate": 0.75},
            },
        },
    },
)

harness.generate().run().report()

🤬 Enhanced Toxicity Testing

In this latest release, we’ve enhanced our toxicity testing framework by introducing a series of comprehensive assessments. Evaluations can be tailored to focus on specific types of toxicity, allowing for detailed analysis in targeted areas of concern. These enhancements are designed to evaluate various facets of toxicity, including obscenity, insult, threat, identity attack, and targeting based on sexual orientation. By expanding the scope of our assessments, we aim to better identify and address instances of negative or hateful comments, harmful stereotypes, and expressions of intent to cause harm or danger.

Supported Tests:

• general_toxicity: This metric focuses on detecting general toxicity, covering a wide range of categories such as obscenity, insult, threat, identity attack, sexual orientation, psychiatric or mental illness related content, and more.
• homosexual_gay_or_lesbian: Assesses negative or hateful comments targeting individuals based on gender identity or sexual orientation.
• insult: Evaluates the insult score of completions, particularly instances where vulgar language is used with the primary intent to demean or belittle the recipient.
• identity_attack: Assesses whether completions perpetuate harmful stereotypes or attack individuals or groups based on intrinsic characteristics such as race, ethnicity, religion, etc.
• obscene: Evaluates the presence of obscene content within completions, including vulgar language, explicit sexual references, or any form of communication deemed offensive or inappropriate.
• threat: Identifies expressions of intent to cause harm or danger within completions.

How to Execute:

harness = Harness(task={"task": "text-generation", "category": "toxicity"},
                  model={"model": "gpt-3.5-turbo-instruct", "hub": "openai"},
                  data={"data_source": 'Toxicity', "split": "test"},
                  config={
                      "model_parameters": {
                          "temperature": 0.2,
                          "max_tokens": 200
                      },
                      "tests": {
                          "defaults": {
                              "min_pass_rate": 1.0
                          },
                          "toxicity": {
                              "obscene": {
                                  "min_pass_rate": 0.7
                              }
                          }
                      }
                  })

harness.generate().run().report()

Example Test Cases:

📒 New Notebooks

| Notebooks | Collab Link | |——————–|————-| | LM-Studio | | | Embedding-Benchmark| | | QA-Benchmark | | | Multi-Dataset | | | Toxicity | | —————-

🐛 Fixes

• Fixed bugs in accuracy task [#945] [#958]
• Fixed llm eval for transformers and lm studio - Code Refactoring [#963 ]
• Fixed religion bias space issue [#966]
• Fixed MedQA dataset [#972]
• Fixed cli issues [#972]
• Fixed CSVDataset and HuggingFaceDataset [#976 ]

⚡ Enhancements

• Enhanced toxicity Test [#979]
• Enhanced Sycophancy Math Test [#977]

• Introduced LLM Eval in Fairness and Accuracy [#974] [#945]

What’s Changed

• Fix accuracy and bugs by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/945
• Lm studio by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/955
• Remove unused variable and update reference to global_service_context by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/956
• Display model response for accuracy by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/958
• Update display import with try_import_lib by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/961
• Feature/run embedding benchmark pipelines CLI by @ArshaanNazir in https://github.com/JohnSnowLabs/langtest/pull/960
• Fix llm eval for transformers and lm studio and Code Refactoring by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/963
• Feature/add feature to compare models on different benchmark datasets by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/964
• Fix/religion bias space issue by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/966
• Fixes by @RakshitKhajuria in https://github.com/JohnSnowLabs/langtest/pull/967
• Renaming sub task by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/970
• Fixes/cli issues by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/972
• website updates by @ArshaanNazir in https://github.com/JohnSnowLabs/langtest/pull/962
• Feature/Updated_toxicity_Test by @ArshaanNazir in https://github.com/JohnSnowLabs/langtest/pull/979
• Fix/datasets by @ArshaanNazir in https://github.com/JohnSnowLabs/langtest/pull/975
• Fix: CSVDataset and HuggingFaceDataset class by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/976
• Llm eval in fairness by @Prikshit7766 in https://github.com/JohnSnowLabs/langtest/pull/974
• Enhancement/sycophancy math by @RakshitKhajuria in https://github.com/JohnSnowLabs/langtest/pull/977
• Update dependencies in setup.py and pyproject.toml by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/981
• Chore/final website updates by @ArshaanNazir in https://github.com/JohnSnowLabs/langtest/pull/980
• Release/2.0.0 by @ArshaanNazir in https://github.com/JohnSnowLabs/langtest/pull/983

Full Changelog: https://github.com/JohnSnowLabs/langtest/compare/1.10.0…2.0.0