GitLab

* fix check for exceeded quota on Google Drive

* safeguard quota exceed check

* Remove pandas dependecy for CelebA dataset

* address PR comments

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Added KITTI dataset

* Addressed review comments

* Changed type of target to List[Dict] and corrected the data types of the returned values.

* Updated unit test to rely on ImageDatasetTestCase

* Added kitti to dataset documentation

* Cleaned up test and some minor changes

* Made data_url a string instead of a list

* Removed unnecessary try and print
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* Added defusedxml to parse untrusted XML data

* Added typecheck disable for defusedxml
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* redirect the urls

* adds /edit

* Apply suggestions from code review

* fix lint

* undo urls
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* add tests for (Dataset|Image)Folder

* lint

* remove old tests

* cleanup

* more cleanup

* adapt tests

* fix make_dataset

* remove powerset

* readd import

* factor out find_classes

* use find_classes in video datasets

* adapt old tests

* added .item() for target inside getitem

* changed fakedataset's expected return type
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* remove caching from (Fashion|K)?MNIST

* remove unnecessary lazy import

* remove false check of binaries against the md5 of archives

* remove caching from EMNIST

* remove caching from QMNIST

* lint

* fix EMNIST

* streamline QMNIST download

* generalize extract_archive

* [test] re-enable extraction tests on windows

* add tests for detect_file_type

* add error messages to detect_file_type

* Revert "[test] re-enable extraction tests on windows"

This reverts commit 7fafebb0f6b4c49bd72c4b5e0a0b4b8c432bce57.

* add utility functions for better mock call checking

* add tests for decompress

* simplify logic by using pathlib

* lint

* Apply suggestions from code review
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* make decompress private

* remove unnecessary checks

* add error message

* fix mocking

* add remaining tests

* lint
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* use head request for redirects

* remove requests dependency

* add missing docs

* tries fixing docs

* fixes docs

* fixes ..code
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* add mirrors to (Fashion|K)MNIST

* fix download tests for MNIST

We are experiencing 403s when trying to download from the main mnist
site so lets fallback to our own mirror on failure.
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* add custom user agent for download_url

* fix progress bar

* lint

* [test] use repo instead of nightly for download tests

* add tests for PhotoTour dataset

* fix grayscale image generation

* fix test_feature_types for a examples of a single feature

* make image size variable instead of hard coding it

* make dataset length variable instead of hard coding it

* replace numpy with torch

* fix typo

* include intended structure of root directory in docstring of Kinetics400

* fix syntax
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: vfdev <vfdev.5@gmail.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Co-authored-by: vfdev <vfdev.5@gmail.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* use common download utils in VOC and SBDataset

* add tests for VOC

* use common base class for VOC datasets

* remove old voc test and fake data generation
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* fix STL10 folds

* use np.int64 over np.long

* Check if the file is located on Google Drive

download_file() detects if the provided URL contains a link to Google
Drive and passes the request through the download_file_from_google_drive()
if it is the case.

* remove lazy re import

* add guard clause

* use urlparse

* add tests

* Fixing lint.

* make id matching more safe

* Fixing type check.

* Remove AnyStr.
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <vvryniotis@fb.com>

* Fixing mypy errors.

* Fixing typing issue.

* Document undodcumented parameters

* remove setup.cfg changes

* Properly pass normalize down instead of deprecating it

* Fix flake8

* Add new CI check

* Fix type spec

* Leave normalize be part of kwargs
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Simplify code and remove used vars.

* Simplify expressions and remove used parenthesis.

* Jit fixes.

* Making check more readable.

* fixing styles

* Make download_url() follow redirects

Fix bug related to the incorrect processing of redirects.
Follow the redirect chain until the destination is reached or
the number of redirects exceeds the max allowed value (by default 10).

* Parametrize value of max allowed redirect number

Make max number of hops a function argument and assign its default value to 10

* Propagate the max number of hops to download_url()

Add the maximum number of redirect hops parameter to download_url()

* check file existence before redirect

* remove print

* remove recursion

* add tests

* Reducing max_redirect_hops
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Vasilis Vryniotis <vvryniotis@fb.com>

* 'make_dataset' as staticmethod of 'DatasetFolder'

* a better fix
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* initial commit of widerface dataset

* comment out old code

* improve parsing of annotation files

* code cleanup and fix docstring comments

* speed up check for quota exceeded

* cleanup print statements

* reformat code and remove print statements

* minor code cleanup and reformatting

* add more comments

* reuse variable

* reverse formatting changes

* fix flake8 errors

* add type annotations

* fix mypy errors

* add a base_folder to root directory

* some formatting fixes

* GDrive threshold does not throw 403 error

* testing new download logic

* cleanup logic for download and integrity check

* use a better variable name

* format fix

* reorder list in docstring

* initial widerface unit test - fails on MD5 check

* use list of dictionaries to store dataset

* fix docstring formatting

* remove unnecessary error checking

* fix type checker error

* revert typo fix

* rename var constants, use file context manager, verify str args

* fix flake8 error

* fix checking target_type argument values

* create uncompressed dataset folders

* cleanup unit tests for widerface

* use correct os function

* add more info to docstring

* disable unittests for windows

* fix _check_integrity logic

* update docstring

* remove citation

* remove target_type option

* fix formatting issue
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* remove comment and add more info to docstring

* update type annotations

* restart CI jobs
Co-authored-by: Joshua Bradley <jgbrad3@evoforge.org>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: vfdev <vfdev.5@gmail.com>