Unverified Commit f31af392 authored by Mathias Nielsen's avatar Mathias Nielsen Committed by GitHub
Browse files

[ examples] fix loading jsonl with load dataset in run translation example (#26924)



* Renamed variable extension to builder_name

* If builder name is jsonl change to json to align with load_datasets

* Apply suggestions from code review
Co-authored-by: default avatarQuentin Lhoest <42851186+lhoestq@users.noreply.github.com>

---------
Co-authored-by: default avatarQuentin Lhoest <42851186+lhoestq@users.noreply.github.com>
parent e4280d65
......@@ -374,8 +374,12 @@ def main():
if data_args.test_file is not None:
data_files["test"] = data_args.test_file
extension = data_args.test_file.split(".")[-1]
if extension == "jsonl":
builder_name = "json" # the "json" builder reads both .json and .jsonl files
else:
builder_name = extension # e.g. "parquet"
raw_datasets = load_dataset(
extension,
builder_name,
data_files=data_files,
cache_dir=model_args.cache_dir,
token=model_args.token,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment