Skip to content

GitLab

Unverified Commit bca0dd5e authored by Suraj Patil's avatar Suraj Patil Committed by GitHub
Browse files

[run_clm.py] fix getting extention

parent 5442a11f
Hide whitespace changes
Inline Side-by-side
Showing with 5 additions and 1 deletion
examples/language-modeling/run_clm.py
View file @ bca0dd5e
...@@ -227,7 +227,11 @@ def main(): ...@@ -227,7 +227,11 @@ def main():
data_files["train"] = data_args.train_file data_files["train"] = data_args.train_file
if data_args.validation_file is not None: if data_args.validation_file is not None:
data_files["validation"] = data_args.validation_file data_files["validation"] = data_args.validation_file
extension = data_args.train_file.split(".")[-1] extension = (
data_args.train_file.split(".")[-1]
if data_args.train_file is not None
else data_args.validation_file.split(".")[-1]
)
if extension == "txt": if extension == "txt":
extension = "text" extension = "text"
datasets = load_dataset(extension, data_files=data_files) datasets = load_dataset(extension, data_files=data_files)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment