GitLab

* fix MNIST byte flipping

* add test

* move to utils

* remove lazy import

* allow single extension as str in make_dataset

* remove test class

* remove regex

* revert collection to tuple

* cleanup

Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* add ufmt as code formatter

* cleanup

* quote ufmt requirement

* split imports into more groups

* regenerate circleci config

* fix CI

* clarify local testing utils section

* use ufmt pre-commit hook

* split relative imports into local category

* Revert "split relative imports into local category"

This reverts commit f2e224cde2008c56c9347c1f69746d39065cdd51.

* pin black and usort dependencies

* fix local test utils detection

* fix ufmt rev

* add reference utils to local category

* fix usort config

* remove custom categories sorting

* Run pre-commit without fixing flake8

* got a double import in merge
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* make tests that involve GDrive more robust

* fix expected IDs
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Replace in test_datasets*

* Replace in test_image.py

* Replace in test_transforms_tensor.py

* Replace in test_internet.py and test_io.py

* get_list_of_videos is util function still use get_tmp_dir

* Fix get_list_of_videos siginiture

* Add get_tmp_dir import

* Modify test_datasets_video_utils.py for test to pass

* Fix indentation

* Replace get_tmp_dir in util functions in test_dataset_sampler.py

* Replace get_tmp_dir in util functions in test_dataset_video_utils.py

* Move get_tmp_dir() to datasets_utils.py and refactor

* Fix pylint, indentation and imports

* import shutil to common_util.py

* Fix function signiture

* Remove get_list_of_videos under context manager

* Move get_list_of_videos to common_utils.py

* Move get_tmp_dir() back to common_utils.py

* Fix pylint and imports
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* mock out redirect behavior for testing

* revert unrelated change

* simplify

* move tests out of internet tests

* replace urls with dummys

* improve checking of mock calls

* Put internet tests in single file

* adding the file

* edit contirbuting guide

* restore videoapi test

* generalize extract_archive

* [test] re-enable extraction tests on windows

* add tests for detect_file_type

* add error messages to detect_file_type

* Revert "[test] re-enable extraction tests on windows"

This reverts commit 7fafebb0f6b4c49bd72c4b5e0a0b4b8c432bce57.

* add utility functions for better mock call checking

* add tests for decompress

* simplify logic by using pathlib

* lint

* Apply suggestions from code review
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* make decompress private

* remove unnecessary checks

* add error message

* fix mocking

* add remaining tests

* lint
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* fix test_extract_(zip|tar|tar_xz|gzip) on windows

* lint
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* Check if the file is located on Google Drive

download_file() detects if the provided URL contains a link to Google
Drive and passes the request through the download_file_from_google_drive()
if it is the case.

* remove lazy re import

* add guard clause

* use urlparse

* add tests

* Fixing lint.

* make id matching more safe

* Fixing type check.

* Remove AnyStr.
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <vvryniotis@fb.com>

* Make download_url() follow redirects

Fix bug related to the incorrect processing of redirects.
Follow the redirect chain until the destination is reached or
the number of redirects exceeds the max allowed value (by default 10).

* Parametrize value of max allowed redirect number

Make max number of hops a function argument and assign its default value to 10

* Propagate the max number of hops to download_url()

Add the maximum number of redirect hops parameter to download_url()

* check file existence before redirect

* remove print

* remove recursion

* add tests

* Reducing max_redirect_hops
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <vvryniotis@fb.com>

* Adding output channels implementation for pngs.

* Adding tests for png.

* Adding channels in the API and documentation.

* Fixing formatting.

* Refactoring test_image.py to remove huge grace_hopper_517x606.pth file from assets and reduce duplicate code. Moving jpeg assets used by encode and write unit-tests on their separate folders.

* Adding output channels implementation for jpegs. Fix asset locations.

* Add tests for JPEG, adding the channels in the API and documentation and adding checks for inputs.

* Changing folder for unit-test.

* Fixing windows flakiness, removing duplicate test.

* Replacing components to channels.

* Adding reference for supporting CMYK.

* Minor changes: num_components to output_components, adding comments, fixing variable name etc.

* Reverting output_components to num_components.

* Replacing decoding with generic method on tests.

* Palette converted to Gray.

* remove six from python code

* remove six from setup.py

* remove six from tests

* remove six from references

* remove six from packaging

* revert str to torch._six._string_classes

* revert str to torch._six._string_classes

* tgz updates

* tgz updates

* tgz updates

* add tar.xz archive handler

* update unittest for tar.xz archive

* remove .tar.xz unittest

* add separate .tar.xz unittest

* update PY2 compatibility

* [WIP] Add CircleCI for CI

* Make jobs only run on master

* Add initial CI

* [wip] testing if works

* Trying some basic GPU tests

* [WIP] maybe it will work?

* [WIP] One more try

* Pin versions

* Simplify and reuse

* Fix

* [WIP] testing windows

* [WIP] testing windows

* Try windows

* Try Windows

* Try windows

* Try windows

* Try windows

* Try windows

* Try windows

* Try windows

* Windows speedup

* Windows multicores

* Add parallel flags for Windows

* Skip some tests on Windows

* Sync config.yml and config.yml.in

* Regenerate

* Run all tests

* Limit python and cuda version for finding pytorch

* Skip darwin for previous check

* Add description

* Fix logic

* Remove space

* Add CUDA test back

* Add back .travis.yml for now and remove duplicate test

* Add newline

* introduced function to verify str arguments

* flake8

* added FIXME to VOC

* Fixed error message

* added test for verify_str_arg

* cleanup todos

* added option for custom error message

* fix VOC

* fixed Caltech

* Raise error during downloading

* Fix py2 error and lint

* Make tests work on fbcode

* Lint

* Fix rebase error

* Properly use get_file_path_2

* Fix wrong use of get_file_path_2 again

* Missing import

* added fake data

* fixed fake data

* renamed extract and download methods and added functionality

* added raw fake data

* refactored imagenet and added test

* flake8

* added fake devkit and mocked download_url

* reversed uncommenting

* added mock to CI

* fixed tests for imagefolder

* flake8

* WIP

* WIP: minor improvements

* Add tests

* Fix typo

* Use download_and_extract on caltech, cifar and omniglot

* Add a print message during extraction

* Remove EMNIST from test

* fixed check integrity

* stylistic changes

* added test for check_md5 and check_integrity

* flake8

* fix path to test file if not executed from test folder

* Fixes for PyTorch version of tqdm

* Flake

* Flake fix