Merge pull request #1393 from alexgorban/master

A script to generate a text file with FSNS URLs.

street/README.md

@@ -79,6 +79,7 @@ Note that these datasets are very large. The approximate sizes are:
 *   Validation: 64 files of 40MB each.
 *   Test: 64 files of 50MB each.
 *   Testdata: some smaller data files of a few MB for testing.
+*   Total: ~158 Gb.
 
 Here is a list of the download paths:
 
@@ -99,9 +100,14 @@ https://download.tensorflow.org/data/fsns-20160927/validation/validation-00000-o
 https://download.tensorflow.org/data/fsns-20160927/validation/validation-00063-of-00064
 ```
 
-The above files need to be downloaded individually, as they are large and
-downloads are more likely to succeed with the individual files than with a
-single archive containing them all.
+All URLs are stored in the text file `python/fsns_urls.txt`, to download them in
+parallel:
+
+```
+aria2c -c -j 20 -i fsns_urls.txt
+```
+If you ctrl+c and re-execute the command it will continue the aborted download.
+
 
 ## Confidence Tests
 
@@ -256,4 +262,3 @@ defines a Tensor Flow graph that can be used to process images of variable sizes
 to output a 1-dimensional sequence, like a transcription/OCR problem, or a
 0-dimensional label, as for image identification problems. For more information
 see [vgslspecs](g3doc/vgslspecs.md)
-

street/python/fsns_urls.py

+# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Creates a text file with URLs to download FSNS dataset using aria2c.
+
+The FSNS dataset has 640 files and takes 158Gb of the disk space. So it is
+highly recommended to use some kind of a download manager to download it.
+
+Aria2c is a powerful download manager which can download multiple files in
+parallel, re-try if encounter an error and continue previously unfinished
+downloads.
+"""
+
+import os
+
+_FSNS_BASE_URL  = 'http://download.tensorflow.org/data/fsns-20160927/'
+_SHARDS = {'test': 64, 'train': 512, 'validation':64}
+_OUTPUT_FILE = "fsns_urls.txt"
+_OUTPUT_DIR = "data/fsns"
+
+def fsns_paths():
+  paths = ['charset_size=134.txt']
+  for name, shards in _SHARDS.items():
+    for i in range(shards):
+      paths.append('%s/%s-%05d-of-%05d' % (name, name, i, shards))
+  return paths
+
+
+if __name__ == "__main__":
+  with open(_OUTPUT_FILE, "w") as f:
+    for path in fsns_paths():
+      url = _FSNS_BASE_URL + path
+      dst_path = os.path.join(_OUTPUT_DIR, path)
+      f.write("%s\n  out=%s\n" % (url, dst_path))
+  print("To download FSNS dataset execute:")
+  print("aria2c -c -j 20 -i %s" % _OUTPUT_FILE)
+  print("The downloaded FSNS dataset will be stored under %s" % _OUTPUT_DIR)