GitLab

Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* Migrate mnist dataset from np.frombuffer

* Add a copy with bytearray for non-writable buffers

* Add byte reversal for mnist
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

We no longer cache the data in a custom binary, but simply read from the raw data directly.
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* add ufmt as code formatter

* cleanup

* quote ufmt requirement

* split imports into more groups

* regenerate circleci config

* fix CI

* clarify local testing utils section

* use ufmt pre-commit hook

* split relative imports into local category

* Revert "split relative imports into local category"

This reverts commit f2e224cde2008c56c9347c1f69746d39065cdd51.

* pin black and usort dependencies

* fix local test utils detection

* fix ufmt rev

* add reference utils to local category

* fix usort config

* remove custom categories sorting

* Run pre-commit without fixing flake8

* got a double import in merge
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

Summary: Enable analytics logging for MNIST and CIFAR

Reviewed By: colin2328, fmassa

Differential Revision: D29183132

fbshipit-source-id: 67d9f6881354f87659fdec4183b22a1be3ec0eb6
Co-authored-by: Dmitriy Genzel <dgenzel@fb.com>

* remove caching from (Fashion|K)?MNIST

* remove unnecessary lazy import

* remove false check of binaries against the md5 of archives

* remove caching from EMNIST

* remove caching from QMNIST

* lint

* fix EMNIST

* streamline QMNIST download

* add mirrors to (Fashion|K)MNIST

* fix download tests for MNIST

We are experiencing 403s when trying to download from the main mnist
site so lets fallback to our own mirror on failure.
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Simplify code and remove used vars.

* Simplify expressions and remove used parenthesis.

* Jit fixes.

* Making check more readable.

* fixing styles

* Add float support to ColorJitter

* Fix byclass EMNIST

* Fix bymerge, balance, letters EMNIST

* Fix whitespace indent

* Revert unrelated file changes

* Revert unrelated file changes

* Removing unnecessary type conversions.

* Removing the transform and adding dummy class instead.
Co-authored-by: Vasileios Vryniotis <vvryniotis@fb.com>

* enable mypy for torchvision.datasets and ignore existing errors

* imagenet

* utils

* lint

* remove sys.version_info == 2

* remove sys.version_info < 3

* remove from __future__ imports

* Fixes #1716

Fixes EMNIST classes attribute is wrong #1716

* Fixed the Classes for Letters Split

* Update mnist.py

* Move classes attribute inside init definition

* Fix Linting errors

* Removed unnecessary class variables.

* The integrity of dataset files is now being checked right after the download finished. Thus making sure that a corrupt file is not being extracted. In case of corruption we throw a RuntimeError.

* Added missing md5 hashes to MNIST, FashionMNIST, KMNIST, EMNIST and QMNIST datasets.

* Removed printing of error message when integrity check failed.
Reformulated error message.

* Reformatted code to be lint conform.

* Fixed formatting in utils.py

* fix url

* update comment

* introduced function to verify str arguments

* flake8

* added FIXME to VOC

* Fixed error message

* added test for verify_str_arg

* cleanup todos

* added option for custom error message

* fix VOC

* fixed Caltech

* fixed call to the VisionDataset constructor

* change call from keyword arguments to positional

* changed order of arguments

* removed transforms argument once again

* Fixed call to constructor of parent class

* fixed LSUN

* fixed Caltech256

* Added general reader for sn3 tensors in "pascalvincent" format

* Added class QMNIST into mnist.py

* QMNIST dataset: make some pt files smaller

* Change request from fmassa.

* read_sn3_pascalvincent_tensor: cse

* read_sn3_pascalvincent_tensor: check file size (when strict!=False)

* Fix lint

* More lint

* Add documentation and expose QMNIST to dataset namespace

* added fake data

* fixed fake data

* renamed extract and download methods and added functionality

* added raw fake data

* refactored imagenet and added test

* flake8

* added fake devkit and mocked download_url

* reversed uncommenting

* added mock to CI

* fixed tests for imagefolder

* flake8

* WIP

* WIP: minor improvements

* Add tests

* Fix typo

* Use download_and_extract on caltech, cifar and omniglot

* Add a print message during extraction

* Remove EMNIST from test

* introduced new super class for all vision datasets

* Removed root from repr if dataset has no root

* reverted some overly-ambitious autoformatting

* reverted some overly-ambitious autoformatting

* added split attribute to repr of STL10 dataset

* made Python2 friendly and more robust

* Fixed call of the superclass constructor

* moved transform and target_transform back to the base classes

* added check if transforms are present before printing to avoid setting them within the constructor

* added missing transforms and target_transforms to base classes

* fixed linter error

* Update URL for EMNIST

Update the dataset and download URL for EMNIST to the url from the updated arxiv version.

* Update mnist.py

* Add KMNIST

* Add documentation

* Add KMNIST to module's __all__ list

* Revert "move area calculation out of loop (#641)"

This reverts commit 62cbf0bf.

* Revert "Fix FashionMNIST loading MNIST (#640)"

This reverts commit 36215690.

Before this would lead FashionMNIST to contain mnist data:

```
MNIST(root, download=True)
FashionMNIST(root, download=True)
```

As MNIST and FashionMNIST are the same classes, the processed
outputs actual ended up to be the same files. This commit now
stores them at different files and also stores the class name when
saving them. I also added md5 sums.

* cast mnist targer to int

* fix unused variables

* fix syntax in tests

* remove # noqa; rename l variable to line_split; use except OSError

* add W504

* add W504

* add W503,W504 tox.ini

* MNIST and FashionMNIST now have their own 'raw' and 'processed' folders

* mkdir exist_ok