Unverified Commit ea6b879e authored by Harsh Rangwani's avatar Harsh Rangwani Committed by GitHub
Browse files

Only pull keys from db in lsun for faster cache. (#2544)

* Only pull keys from db in lsun for faster cache.

This pull request inhances the speed of the cache creation for lsun dataset. For the "kitchen_train" the speed was getting slow with cache creation taking more then two hours. This speeds up to cache creation in within minutes. The issue was pulling the large image values each time and dropping them.

For more details on this please refer this issue https://github.com/jnwatson/py-lmdb/issues/195.

* Fixed bug in lsun.py when loading multiple categories

* Make linter happy
parent 8dc08196
......@@ -28,7 +28,7 @@ class LSUNClass(VisionDataset):
self.keys = pickle.load(open(cache_file, "rb"))
else:
with self.env.begin(write=False) as txn:
self.keys = [key for key, _ in txn.cursor()]
self.keys = [key for key in txn.cursor().iternext(keys=True, values=False)]
pickle.dump(self.keys, open(cache_file, "wb"))
def __getitem__(self, index: int) -> Tuple[Any, Any]:
......@@ -114,10 +114,10 @@ class LSUN(VisionDataset):
raise ValueError(msg.format(type(classes)))
classes = list(classes)
msg_fmtstr = ("Expected type str for elements in argument classes, "
"but got type {}.")
msg_fmtstr_type = ("Expected type str for elements in argument classes, "
"but got type {}.")
for c in classes:
verify_str_arg(c, custom_msg=msg_fmtstr.format(type(c)))
verify_str_arg(c, custom_msg=msg_fmtstr_type.format(type(c)))
c_short = c.split('_')
category, dset_opt = '_'.join(c_short[:-1]), c_short[-1]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment