Overview
The Evaluation Metrics notebook demonstrates the configurability of the LangTest library, especially metrics and models used for embeddings and distances. This configuration defines a system for comparing embeddings generated by different models using specified distance metrics. Two default models are provided, one from the OpenAI Hub (“text-embedding-ada-002”) and another from the HuggingFace Hub (“sentence-transformers/all-mpnet-base-v2”). The distance metrics available for comparing embeddings include cosine similarity, Euclidean distance, Manhattan distance, Chebyshev distance, and Hamming distance, each with default thresholds. Users can customize the configuration by adjusting thresholds based on their specific needs. The configuration file also includes parameters for the embedding models, evaluation metrics, and test scenarios, allowing users to define various test cases and set minimum pass rates for each scenario, ensuring the robustness and effectiveness of the embedding models in different situations.
Open in Collab
| Category | Hub | Task | Open In Colab |
|---|---|---|---|
| Evaluation Metrics : In this section, we discussed different evatuation metrics for evauate Quetion-Answering models. | OpenAI | Question-Answering |  |
Config Used
tests:
defaults:
min_pass_rate: 0.65
robustness:
lowercase:
min_pass_rate: 0.66
uppercase:
min_pass_rate: 0.66