Overview
In the QA/Summarization notebook section, our primary focus is on the evaluation of OpenAI, Ai21, Cohere, Asure-OpenAi models tailored for Question Answering (QA) and Summarization tasks. This involves a meticulous testing process to gauge the models’ efficiency in responding to questions and producing concise and informative summaries. In a noteworthy addition, we’re introducing perturbations to the text during these tasks. By incorporating variations in the input, such as adding perturbations, we aim to observe how the model responses adapt to changes in the text.
Config Used
tests:
defaults:
min_pass_rate: 1.0
robustness:
add_typo:
min_pass_rate: 0.70
lowercase:
min_pass_rate: 0.70
OpenAi Question-Answering & Summarization
In this notebook we are testing OpenAI gpt-3.5-turbo-instruct Model For Question Answering and Summarization task.
Open in Collab
| Category | Hub | Task | Datset Used | Open In Colab |
|---|---|---|---|---|
| OpenAI QA/Summarization | OpenAI | Question-Answering/Summarization | BoolQ, NQ-Open, Xsum |
 |
AI21 Question-Answering & Summarization
In this notebook we are testing AI21 j2-jumbo-instruct Model For Question Answering and Summarization task.
Open in Collab
| Category | Hub | Task | Datset Used | Open In Colab |
|---|---|---|---|---|
| Question-Answering & Summarization | AI21 | Question-Answering/Summarization | BoolQ, NQ-Open, Xsum |
|
Cohere Question-Answering & Summarization
In this notebook we are testing Cohere command-xlarge-nightly Model For Question Answering and Summarization task.
Open in Collab
| Category | Hub | Task | Datset Used | Open In Colab |
|---|---|---|---|---|
| Question-Answering & Summarization | Cohere | Question-Answering/Summarization | BoolQ, NQ-Open, Xsum |
|
Hugging Face Inference API Question-Answering & Summarization
In this notebook we are testing Hugging Face Inference API google/flan-t5-small Model For Question Answering and google/pegasus-newsroom for Summarization task.
Open in Collab
| Category | Hub | Task | Datset Used | Open In Colab |
|---|---|---|---|---|
| Question-Answering & Summarization | Hugging Face Inference API | Question-Answering/Summarization | BoolQ, NQ-Open, Xsum |
|
Hugging Face Hub Question-Answering & Summarization
In this notebook we are testing Hugging Face Hub facebook/opt-1.3b Model For Question Answering and Summarization task.
Open in Collab
| Category | Hub | Task | Datset Used | Open In Colab |
|---|---|---|---|---|
| Question-Answering & Summarization | Hugging Face Hub | Question-Answering/Summarization | BoolQ, NQ-Open, Xsum |
|
Azure-OpenAI Question-Answering & Summarization
In this notebook we are testing Azure-OpenAI gpt-3.5-turbo-instruct Model For Question Answering and Summarization task.
Open in Collab
| Category | Hub | Task | Datset Used | Open In Colab |
|---|---|---|---|---|
| Question-Answering & Summarization | Azure-OpenAI | Question-Answering/Summarization | BoolQ, NQ-Open, Xsum |
|