Skip to content

GitLab

"git@developer.sourcefind.cn:OpenDAS/megatron-lm.git" did not exist on "1a2cb60c5ca7456f4971772c66a66e2a80f99232"
Commit 5aee67b4 authored by Reed's avatar Reed Committed by Taylor Robie
Browse files

Speed up cache construction. (#5131) 

This is done by using a higher Pickle protocol version, which the Python docs describe as being "slightly more efficient". This reduces the file write time at the beginning from 2 1/2 minutes to 5 seconds.
parent 0864b2a4
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
official/recommendation/data_preprocessing.py
View file @ 5aee67b4
...@@ -314,7 +314,7 @@ def generate_train_eval_data(df, approx_num_shards, num_items, cache_paths): ...@@ -314,7 +314,7 @@ def generate_train_eval_data(df, approx_num_shards, num_items, cache_paths):
tf.logging.info("Writing test data to file.") tf.logging.info("Writing test data to file.")
tf.gfile.MakeDirs(cache_paths.eval_data_subdir) tf.gfile.MakeDirs(cache_paths.eval_data_subdir)
with tf.gfile.Open(cache_paths.eval_raw_file, "wb") as f: with tf.gfile.Open(cache_paths.eval_raw_file, "wb") as f:
pickle.dump(eval_data, f) pickle.dump(eval_data, f, protocol=pickle.HIGHEST_PROTOCOL)
def construct_cache(dataset, data_dir, num_data_readers): def construct_cache(dataset, data_dir, num_data_readers):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment