GitLab

Summary: Enable analytics logging for MNIST and CIFAR

Reviewed By: colin2328, fmassa

Differential Revision: D29183132

fbshipit-source-id: 67d9f6881354f87659fdec4183b22a1be3ec0eb6
Co-authored-by: Dmitriy Genzel <dgenzel@fb.com>

* Initial commit

* pmeiers comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers changes
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* replace pandas with system library to avoid crashes

* Lint

* Lint

* fixing unittest

* Minor comments removal

* pmeier comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* remove asserts

* address pmeier formatting changes

* address pmeier changes
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeier changes
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* rename n_classes to num_classes

* formatting changes

* doc change to add ".mp4" to backported class

* formatting to correct line length

* adding **kwargs to Kinetics400 class

* remove urlib request and download the file directly

* annotations and files can be already downloaded

* test fix

* add download tests for Kinetics

* users now dont need to provide full path within the root for new Kinetics dataset

* linter

* Update test/test_datasets_download.py

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* revert whitespace (3680#discussion_r626382842)

* addressing annotation_path parameter which is unnecessary

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* kwargs update
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* expose num_download_workers as public

* swap os.isfile with check_integrity

* nit on private things

* special case if there are no default arguments

* revert changes to kinetics400 test case for BC

* add split_folder changes and support for legacy format

* pmeiers suggestions
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestions - root comment

* pmeiers comments - annotation attribute remmoved

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* minor debugging

* nit picks

* only include public kwargs into defaults

* add _use_legacy_structure in favour of **kwargs

* add type hints for Kinetics400

* flake8

* flake8

* flake8

* rename to make thigs clearer

* permuting the output
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* fix

* add docs for VisionDataset

* add getitem

* add deprecated note to transform

* Update torchvision/datasets/vision.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* fix docs

* Update torchvision/datasets/vision.py

* flake8
Co-authored-by: Nicolas Hug <nicolashug@fb.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

Reviewed By: fmassa

Differential Revision: D28537606

fbshipit-source-id: beaf15f4e38a7d986ef26587383aa2aee3e48952

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/3823

Uploaded FashionMNIST dataset to [manifold](https://www.internalfb.com/intern/network/manifold/?bucket=torchvision&path=tree%2Fdatasets) bucket `torchvision`. Any new dataset that needs to be added could be uploaded under `tree/datasets/<dataset_name>`.

Reviewed By: datumbox

Differential Revision: D28358470

fbshipit-source-id: 6f2282d3f1ce4b1416e962de8fb132896d4b2d76

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* fix check for exceeded quota on Google Drive

* safeguard quota exceed check

* Remove pandas dependecy for CelebA dataset

* address PR comments

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Added KITTI dataset

* Addressed review comments

* Changed type of target to List[Dict] and corrected the data types of the returned values.

* Updated unit test to rely on ImageDatasetTestCase

* Added kitti to dataset documentation

* Cleaned up test and some minor changes

* Made data_url a string instead of a list

* Removed unnecessary try and print
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* Added defusedxml to parse untrusted XML data

* Added typecheck disable for defusedxml
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* redirect the urls

* adds /edit

* Apply suggestions from code review

* fix lint

* undo urls
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* add tests for (Dataset|Image)Folder

* lint

* remove old tests

* cleanup

* more cleanup

* adapt tests

* fix make_dataset

* remove powerset

* readd import

* factor out find_classes

* use find_classes in video datasets

* adapt old tests

* added .item() for target inside getitem

* changed fakedataset's expected return type
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* remove caching from (Fashion|K)?MNIST

* remove unnecessary lazy import

* remove false check of binaries against the md5 of archives

* remove caching from EMNIST

* remove caching from QMNIST

* lint

* fix EMNIST

* streamline QMNIST download

* generalize extract_archive

* [test] re-enable extraction tests on windows

* add tests for detect_file_type

* add error messages to detect_file_type

* Revert "[test] re-enable extraction tests on windows"

This reverts commit 7fafebb0f6b4c49bd72c4b5e0a0b4b8c432bce57.

* add utility functions for better mock call checking

* add tests for decompress

* simplify logic by using pathlib

* lint

* Apply suggestions from code review
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* make decompress private

* remove unnecessary checks

* add error message

* fix mocking

* add remaining tests

* lint
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* use head request for redirects

* remove requests dependency

* add missing docs

* tries fixing docs

* fixes docs

* fixes ..code
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* add mirrors to (Fashion|K)MNIST

* fix download tests for MNIST

We are experiencing 403s when trying to download from the main mnist
site so lets fallback to our own mirror on failure.
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* add custom user agent for download_url

* fix progress bar

* lint

* [test] use repo instead of nightly for download tests

* add tests for PhotoTour dataset

* fix grayscale image generation

* fix test_feature_types for a examples of a single feature

* make image size variable instead of hard coding it

* make dataset length variable instead of hard coding it

* replace numpy with torch

* fix typo

* include intended structure of root directory in docstring of Kinetics400

* fix syntax
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: vfdev <vfdev.5@gmail.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>