langtest.datahandler.datasource.JSONLDataset#

class JSONLDataset(file_path: str, task: TaskManager)#

Bases: BaseDataset

Class to handle JSONL datasets. Subclass of BaseDataset.

__init__(file_path: str, task: TaskManager) → None#

Initializes JSONLDataset object.

Parameters:

• file_path (str) – Path to the data file.

• task (str) – name of the task to perform

Methods

__init__(file_path, task)	Initializes JSONLDataset object.
export_data(data, output_path)	Exports the data to the corresponding format and saves it to 'output_path'.
load_data(*args, **kwargs)	Loads data from a JSONL file and format it into a list of Sample.
load_raw_data()	Loads data from a JSON file into a list

Attributes

COLUMN_NAMES
data_sources
dataset_size
supported_tasks

export_data(data: List[Sample], output_path: str)#

Exports the data to the corresponding format and saves it to ‘output_path’.

Parameters:

• data (List[Sample]) – data to export

• output_path (str) – path to save the data to

load_data(*args, **kwargs) → List[Sample]#

Loads data from a JSONL file and format it into a list of Sample.

Returns:

Loaded text data.

Return type:

list[Sample]

load_raw_data() → List[Dict]#

Loads data from a JSON file into a list