GitLab

Co-authored-by: Nicolas Hug <nicolashug@fb.com>

Summary: We would like to track the internal usage of TorchVision components. We're tracking [datasets usage](https://fburl.com/daiquery/dqpmemn3

) now. This diff expand the tracking to all models.

Reviewed By: fmassa

Differential Revision: D31441632

fbshipit-source-id: e26072e582ac9f832c2056307ebf0eccf2ed6c9c
Co-authored-by: Kai Zhang <kaizh@fb.com>

* Migrate mnist dataset from np.frombuffer

* Add a copy with bytearray for non-writable buffers

* Add byte reversal for mnist
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* make mypy more strict for prototype datasets

* fix code format

* apply strictness only to datasets

* fix more mypy issues

* cleanup

* fix mnist annotations

* refactor celeba

* warn on redundant casts

* remove redundant cast

* simplify annotation

* fix import

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

We no longer cache the data in a custom binary, but simply read from the raw data directly.
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

* add prototype for VOC dataset

* fix code format

* fix mypy

* resolve todo

* Allow redefinition for mypy

* appease mypy
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* add ufmt as code formatter

* cleanup

* quote ufmt requirement

* split imports into more groups

* regenerate circleci config

* fix CI

* clarify local testing utils section

* use ufmt pre-commit hook

* split relative imports into local category

* Revert "split relative imports into local category"

This reverts commit f2e224cde2008c56c9347c1f69746d39065cdd51.

* pin black and usort dependencies

* fix local test utils detection

* fix ufmt rev

* add reference utils to local category

* fix usort config

* remove custom categories sorting

* Run pre-commit without fixing flake8

* got a double import in merge
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* Added LFW Dataset

* Added dataset to list in __init__.py

* Updated lfw.py
* Created a common superclass for people and pairs type datatsets
* corrected the .download() method

* Added docstrings and updated datasets.rst

* Wrote tests for LFWPeople and LFWPairs

* Resolved mypy error: Need type annotation for "data"

* Updated inject_fake_data method for LFWPeople

* Updated tests for LFW

* Updated LFW tests and minor changes in lfw.py

* Updated LFW
* Added functionality for 10-fold validation view
* Optimized the code so to replace repeated lines by method in super
  class
* Updated LFWPeople to get classes from all lfw-names.txt rather than
  just the classes fron trainset

* Updated lfw.py and tests
* Updated inject_fake_data method to create 10fold fake data
* Minor changes in docstring and extra_repr

* resolved py lint errors

* Added checksums for annotation files

* Minor changes in test

* Updated docstrings, defaults and minor changes in test

* Removed 'os.path.exists' check
Co-authored-by: ABD-01 <myac931@gmai.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Vassilis Nicodemou <nikodim@ics.forth.gr>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update UCF101 docstring

* addressing @NicolasHugs comments

* Fixed flake8 error
Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* style: Added missing typing annotations

* style: Fixed last missing typing annotation

* chore: Fixed missing import

* style: Fixed typing

* refactor: Switched file selection from list to set
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Add iNaturalist dataset

* Add download support

* address comments
Co-authored-by: dgenzel <dgenzel@fb.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Summary: Enable analytics logging for MNIST and CIFAR

Reviewed By: colin2328, fmassa

Differential Revision: D29183132

fbshipit-source-id: 67d9f6881354f87659fdec4183b22a1be3ec0eb6
Co-authored-by: Dmitriy Genzel <dgenzel@fb.com>

* Initial commit

* pmeiers comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers changes
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* replace pandas with system library to avoid crashes

* Lint

* Lint

* fixing unittest

* Minor comments removal

* pmeier comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* remove asserts

* address pmeier formatting changes

* address pmeier changes
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeier changes
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* rename n_classes to num_classes

* formatting changes

* doc change to add ".mp4" to backported class

* formatting to correct line length

* adding **kwargs to Kinetics400 class

* remove urlib request and download the file directly

* annotations and files can be already downloaded

* test fix

* add download tests for Kinetics

* users now dont need to provide full path within the root for new Kinetics dataset

* linter

* Update test/test_datasets_download.py

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* revert whitespace (3680#discussion_r626382842)

* addressing annotation_path parameter which is unnecessary

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* kwargs update
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* expose num_download_workers as public

* swap os.isfile with check_integrity

* nit on private things

* special case if there are no default arguments

* revert changes to kinetics400 test case for BC

* add split_folder changes and support for legacy format

* pmeiers suggestions
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestions - root comment

* pmeiers comments - annotation attribute remmoved

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* pmeiers suggestion
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/datasets/kinetics.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* minor debugging

* nit picks

* only include public kwargs into defaults

* add _use_legacy_structure in favour of **kwargs

* add type hints for Kinetics400

* flake8

* flake8

* flake8

* rename to make thigs clearer

* permuting the output
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* fix

* add docs for VisionDataset

* add getitem

* add deprecated note to transform

* Update torchvision/datasets/vision.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* fix docs

* Update torchvision/datasets/vision.py

* flake8
Co-authored-by: Nicolas Hug <nicolashug@fb.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

Reviewed By: fmassa

Differential Revision: D28537606

fbshipit-source-id: beaf15f4e38a7d986ef26587383aa2aee3e48952

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/3823

Uploaded FashionMNIST dataset to [manifold](https://www.internalfb.com/intern/network/manifold/?bucket=torchvision&path=tree%2Fdatasets) bucket `torchvision`. Any new dataset that needs to be added could be uploaded under `tree/datasets/<dataset_name>`.

Reviewed By: datumbox

Differential Revision: D28358470

fbshipit-source-id: 6f2282d3f1ce4b1416e962de8fb132896d4b2d76