Commit 5aee67b4 authored by Reed's avatar Reed Committed by Taylor Robie
Browse files

Speed up cache construction. (#5131)

This is done by using a higher Pickle protocol version, which the Python docs describe as being "slightly more efficient". This reduces the file write time at the beginning from 2 1/2 minutes to 5 seconds.
parent 0864b2a4
......@@ -314,7 +314,7 @@ def generate_train_eval_data(df, approx_num_shards, num_items, cache_paths):
tf.logging.info("Writing test data to file.")
tf.gfile.MakeDirs(cache_paths.eval_data_subdir)
with tf.gfile.Open(cache_paths.eval_raw_file, "wb") as f:
pickle.dump(eval_data, f)
pickle.dump(eval_data, f, protocol=pickle.HIGHEST_PROTOCOL)
def construct_cache(dataset, data_dir, num_data_readers):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment