GitLab

* fix prototype dataset imagenet mock data

* [REVERT ME] temporarily add prototype datasets test to CI

* Revert "[REVERT ME] temporarily add prototype datasets test to CI"

This reverts commit 7cafb5f01487c7625bcffd88e76832349b3e5055.

* close streams in prototype datasets

* refactor prototype SBD to avoid closing demux streams at construction time

* mypy

* upgrade usort to

* Also update black

* Actually use 1.0.2

* Apply pre-commit
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* merge mock data preparation and loading

* address comments

* fix extra file creation

* remove tmp folder

* inline images meta creation in coco mock data

* rely on patched datasets home rather than passing it around

* add comment

* refactor prototype datasets to inherit from IterDataPipe (#5448)

* refactor prototype datasets to inherit from IterDataPipe

* depend on new architecture

* fix missing file detection

* remove unrelated file

* reinstante decorator for mock registering

* options -> config

* remove passing of info to mock data functions

* refactor categories file generation

* fix imagenet

* fix prototype datasets data loading tests (#5711)

* reenable serialization test

* cleanup

* fix dill test

* trigger CI

* patch DILL_AVAILABLE for pickle serialization

* revert CI changes

* remove dill test and traversable test

* add data loader test

* parametrize over only_datapipe

* draw one sample rather than exhaust data loader

* cleanup

* trigger CI

* migrate VOC prototype dataset (#5743)

* migrate VOC prototype dataset

* cleanup

* revert unrelated mock data changes

* remove categories annotations

* move properties to constructor

* readd homepage

* migrate CIFAR prototype datasets (#5751)

* migrate country211 prototype dataset (#5753)

* migrate CLEVR prototype datsaet (#5752)

* migrate coco prototype (#5473)

* migrate coco prototype

* revert unrelated change

* add kwargs to super constructor call

* remove unneeded changes

* fix docstring position

* make kwargs explicit

* add dependencies to docstring

* fix missing dependency message

* Migrate PCAM prototype dataset (#5745)

* Port PCAM

* skip_integrity_check

* Update torchvision/prototype/datasets/_builtin/pcam.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Address comments
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Migrate DTD prototype dataset (#5757)

* Migrate DTD prototype dataset

* Docstring

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Migrate GTSRB prototype dataset (#5746)

* Migrate GTSRB prototype dataset

* ufmt

* Address comments

* Apparently mypy doesn't know that __len__ returns ints. How cute.

* why is the CI not triggered??

* Update torchvision/prototype/datasets/_builtin/gtsrb.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* migrate CelebA prototype dataset (#5750)

* migrate CelebA prototype dataset

* inline split_id

* Migrate Food101 prototype dataset (#5758)

* Migrate Food101 dataset

* Added length

* Update torchvision/prototype/datasets/_builtin/food101.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Migrate Fer2013 prototype dataset (#5759)

* Migrate Fer2013 prototype dataset

* Update torchvision/prototype/datasets/_builtin/fer2013.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Migrate EuroSAT prototype dataset (#5760)

* Migrate Semeion prototype dataset (#5761)

* migrate caltech prototype datasets (#5749)

* migrate caltech prototype datasets

* resolve third party dependencies

* Migrate Oxford Pets prototype dataset (#5764)

* Migrate Oxford Pets prototype dataset

* Update torchvision/prototype/datasets/_builtin/oxford_iiit_pet.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* migrate mnist prototype datasets (#5480)

* migrate MNIST prototype datasets

* Update torchvision/prototype/datasets/_builtin/mnist.py
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* Migrate Stanford Cars prototype dataset (#5767)

* Migrate Stanford Cars prototype dataset

* Address comments

* fix category file generation (#5770)

* fix category file generation

* revert unrelated change

* revert unrelated change

* migrate cub200 prototype dataset (#5765)

* migrate cub200 prototype dataset

* address comments

* fix category-file-generation

* Migrate USPS prototype dataset (#5771)

* migrate SBD prototype dataset (#5772)

* migrate SBD prototype dataset

* reuse categories

* Migrate SVHN prototype dataset (#5769)

* add test to enforce __len__ is working on prototype datasets (#5742)

* reactivate special dataset tests

* add missing annotation

* Cleanup prototype dataset implementation (#5774)

* Remove Dataset2 class

* Move read_categories_file out of DatasetInfo

* Remove FrozenBunch and FrozenMapping

* Remove test_prototype_datasets_api.py and move missing dep test somewhere else

* ufmt

* Let read_categories_file accept names instead of paths

* Mypy

* flake8

* fix category file reading
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* update prototype dataset README (#5777)

* update prototype dataset README

* fix header level

* Apply suggestions from code review
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* [FEAT] Start implementing Food101 using the new datasets API. WIP.

* [FEAT] Generate Food101 categories and start the test mock.

* [FEAT] food101 dataset code seems to work now.

* [TEST] food101 mock update.

* [FIX] Some fixes thanks to running food101 tests.

* [FIX] Fix mypy checks for the food101 file.

* [FIX] Remove unused numpy.

* [FIX] Some changes thanks to code review.

* [ENH] More idomatic dataset code thanks to code review.

* [FIX] Remove unused cast.

* [ENH] Set decompress and extract to True for some performance gains.

* [FEAT] Use the preprocess=decompress keyword.

* [ENH] Use the train and test.txt file instead of the .json variants and simplify code + update mock data.

* [ENH] Better food101 mock data generation.

* [FIX] Remove a useless print.
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* fix PCAM prototype dataset

* update Zip and Tar archive loader datapipes

* only fail on warnings from the pytorch ecosystem

* Revert "only fail on warnings from the pytorch ecosystem"

This reverts commit 2bf3aa6f2d875a4055f7f3ed0b468316fc60d4f4.

* added usps dataset

* fixed type issues

* fix mobilnet norm layer test (#5643)

* xfail mobilnet norm layer test

* fix test

* More robust check in tests for 16 bits images (#5652)

* Prefer nvidia channel for conda builds (#5648)

To mitigate missing `libcupti.so` dependency

* fix torchdata CI installation (#5657)

* update urls for kinetics dataset (#5578)

* update urls for kinetics dataset

* update urls for kinetics dataset

* remove errors

* update the changes and add test option to split

* added test to valid values for split arg

* change .txt to .csv for annotation url of k600
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* Port Multi-weight support from prototype to main (#5618)

* Moving basefiles outside of prototype and porting Alexnet, ConvNext, Densenet and EfficientNet.

* Porting googlenet

* Porting inception

* Porting mnasnet

* Porting mobilenetv2

* Porting mobilenetv3

* Porting regnet

* Porting resnet

* Porting shufflenetv2

* Porting squeezenet

* Porting vgg

* Porting vit

* Fix docstrings

* Fixing imports

* Adding missing import

* Fix mobilenet imports

* Fix tests

* Fix prototype tests

* Exclude get_weight from models on test

* Fix init files

* Porting googlenet

* Porting inception

* porting mobilenetv2

* porting mobilenetv3

* porting resnet

* porting shufflenetv2

* Fix test and linter

* Fixing docs.

* Porting Detection models (#5617)

* fix inits

* fix docs

* Port faster_rcnn

* Port fcos

* Port keypoint_rcnn

* Port mask_rcnn

* Port retinanet

* Port ssd

* Port ssdlite

* Fix linter

* Fixing tests

* Fixing tests

* Fixing vgg test

* Porting Optical Flow, Segmentation, Video models (#5619)

* Porting raft

* Porting video resnet

* Porting deeplabv3

* Porting fcn and lraspp

* Fixing the tests and linter

* Porting docs, examples, tutorials and galleries (#5620)

* Fix examples, tutorials and gallery

* Update gallery/plot_optical_flow.py
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* Fix import

* Revert hardcoded normalization

* fix uncommitted changes

* Fix bug

* Fix more bugs

* Making resize optional for segmentation

* Fixing preset

* Fix mypy

* Fixing documentation strings

* Fix flake8

* minor refactoring
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* Resolve conflict

* Porting model tests (#5622)

* Porting tests

* Remove unnecessary variable

* Fix linter

* Move prototype to extended tests

* Fix download models job

* Update CI on Multiweight branch to use the new weight download approach (#5628)

* port Pad to prototype transforms (#5621)

* port Pad to prototype transforms

* use literal

* Bump up LibTorchvision version number for Podspec to release Cocoapods (#5624)
Co-authored-by: Anton Thomma <anton@pri.co.nz>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* pre-download model weights in CI docs build (#5625)

* pre-download model weights in CI docs build

* move changes into template

* change docs image

* Regenerated config.yml
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Anton Thomma <11010310+thommaa@users.noreply.github.com>
Co-authored-by: Anton Thomma <anton@pri.co.nz>

* Porting reference scripts and updating presets (#5629)

* Making _preset.py classes

* Remove support of targets on presets.

* Rewriting the video preset

* Adding tests to check that the bundled transforms are JIT scriptable

* Rename all presets from *Eval to *Inference

* Minor refactoring

* Remove --prototype and --pretrained from reference scripts

* remove  pretained_backbone refs

* Corrections and simplifications

* Fixing bug

* Fixing linter

* Fix flake8

* restore documentation example

* minor fixes

* fix optical flow missing param

* Fixing commands

* Adding weights_backbone support in detection and segmentation

* Updating the commands for InceptionV3

* Setting `weights_backbone` to its fully BC value (#5653)

* Replace default `weights_backbone=None` with its BC values.

* Fixing tests

* Fix linter

* Update docs.

* Update preprocessing on reference scripts.

* Change qat/ptq to their full values.

* Refactoring preprocessing

* Fix video preset

* No initialization on VGG if pretrained

* Fix warning messages for backbone utils.

* Adding star to all preset constructors.

* Fix mypy.
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Anton Thomma <11010310+thommaa@users.noreply.github.com>
Co-authored-by: Anton Thomma <anton@pri.co.nz>

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* use decompressor for extracting bz2

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* fixed lint fails

* added tests for USPS

* check image shape

* fix tests

* check shape on image directly

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* removed test and comments

* Update test/test_prototype_builtin_datasets.py
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Nikita Shulga <nshulga@fb.com>
Co-authored-by: Sahil Goyal <sahilgoyal6801@gmail.com>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Anton Thomma <11010310+thommaa@users.noreply.github.com>
Co-authored-by: Anton Thomma <anton@pri.co.nz>

* Moving `sequence_to_str` to `torchvision._utils`

* Fix linter

* Rename test_prototype_utils test to test_internal_utils

* [DEBUG] eurosat prototype on windows and macos

* print paths

* fix eurosat mock data setup

* revert changes

* minor cleanup

* basic stanford cars structure added.

* added class in the init.

* how to decode the images

* fix typo in init

* works partially

* formatting

* formatting

* formatting

* formatting

* formatting

* minor changes

* minor changes

* issue resolved

* both datapipes are in sync

* checksum dict keys changed

* formatting

* "ufmt formatting"

* "ufmt formatting"

* formatting

* minor change

* minor change

* test added

* lint (end of line issue)

* lint (end of line issue)

* minor changes

* mock dataset formatting

* minor changes

* hopefully final

* formatting

* prototype test

* prototype test

* prototype test

* prototype test

* prototype test

* prototype test

* prototype test

* prototype test

* Update builtin_dataset_mocks.py

* Update stanford_cars.py

* Update torchvision/prototype/datasets/_builtin/stanford_cars.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/prototype/datasets/_builtin/stanford_cars.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/prototype/datasets/_builtin/stanford_cars.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* Update torchvision/prototype/datasets/_builtin/stanford_cars.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* formatting and cleaning

* formatting and cleaning

* Update torchvision/prototype/datasets/_builtin/stanford_cars.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* generate catefories script

* generate categories script added

* sigh

* sigh

* sigh

* minor changes

* linting
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* add eurosat

* revert formatting

* port test and make style changes

* add eurosat to __init__

* fix pathlib error

* create dataset zipfile and revert pre commit changes

* remove unecessary variable in resources

* revert auto formatter changes and modify ufmt version

* revert change to contributing guide
Co-authored-by: Dbhasin1 <drishti_b@me.iitr.c.in>
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* add country211

* remove unused import

* map val to valid and use path comparator

* remove unused import

* resolve keyerror

* map split names in dataset mock
Co-authored-by: Dbhasin1 <drishti_b@me.iitr.c.in>
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* revamp prototype features (#5283)

* remove decoding from prototype datasets (#5287)

* remove decoder from prototype datasets

* remove unused imports

* cleanup

* fix readme

* use OneHotLabel in SEMEION

* improve voc implementation

* revert unrelated changes

* fix semeion mock data

* fix pcam

* readd functional transforms API to prototype (#5295)

* readd functional transforms

* cleanup

* add missing imports

* remove __torch_function__ dispatch

* readd repr

* readd empty line

* add test for scriptability

* remove function copy

* change import from functional tensor transforms to just functional

* fix import

* fix test

* fix prototype features and functional transforms after review (#5377)

* fix prototype functional transforms after review

* address features review

* make mypy more strict on prototype features

* make mypy more strict for prototype transforms

* fix annotation

* fix kernel tests

* add automatic feature type dispatch to functional transforms (#5323)

* add auto dispatch

* fix missing arguments error message

* remove pil kernel for erase

* automate feature specific parameter detection

* fix typos

* cleanup dispatcher call

* remove __torch_function__ from transform dispatch

* remove auto-generation

* revert unrelated changes

* remove implements decorator

* change register parameter order

* change order of transforms for readability

* add documentation for __torch_function__

* fix mypy

* inline check for support

* refactor kernel registering process

* refactor dispatch to be a regular decorator

* split kernels and dispatchers

* remove sentinels

* replace pass with ...

* appease mypy

* make single kernel dispatchers more concise

* make dispatcher signatures more generic

* make kernel checking more strict

* revert doc changes

* address Franciscos comments

* remove inplace

* rename kernel test module

* fix inplace

* remove special casing for pil and vanilla tensors

* address comments

* update docs

* cleanup features / transforms feature branch (#5406)

* mark candidates for removal

* align signature of resize_bounding_box with corresponding image kernel

* fix documentation of Feature

* remove interpolation mode and antialias option from resize_segmentation_mask

* remove or privatize functionality in features / datasets / transforms

* Add PCAM dataset to prototype area

* use BytesIO instead of writing file to disk

* Apply suggestions from code review
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* remove noqa

* typos

* Use _Resource namedtuple

* Add h5py to unittest_prototype job

* use .item() on target

* Forgot to call regenerate.py
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* use fixture for dataset root in tests

* fix home dir generation

* remove caching
Co-authored-by: Philip Meier <github.pmeier@posteo.de>

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

* add SVHN prototype dataset

* add test

* refactor prototype dataset tests to use public API for loading

* add explanation

* use loop alternative

* refactor prototype datasets tests

* skip tests with insufficient third party dependencies

* cleanup

* add tests for SBD prototype dataset

* add tests for SEMEION prototype dataset

* add tests for VOC prototype dataset

* add tests for CelebA prototype dataset

* add tests for DTD prototype dataset

* add tests for FER2013 prototype dataset

* add tests for CLEVR prototype dataset

* add tests for oxford-iiit-pet prototype dataset

* enforce tests for new datasets

* add missing archive generation for oxford-iiit-pet tests

* add tests for CUB200 prototype datasets

* fix split generation

* add capability to mark parametrization and xfail cub200 traverse tests

* refactor prototype datasets tests

* skip tests with insufficient third party dependencies

* add download functionality to prototype datasets

* fix annotation

* fix test

* remove iopath

* add comments

* improve COCO prototype

* test 2017 annotations

* add option to include captions

* fix categories and add tests

* cleanup

* add correct image size to bounding boxes

* fix annotation collation

* appease mypy

* add benchmark

* always use image as reference

* another refactor

* add support for segmentations

* add support for segmentations

* fix CI dependencies

* return features instead of vanilla tensors from prototype datasets

* fix tests

* remove inplace

* add explanation for __init_subclass__

* fix label for test split

* relax test

* remove pixels

* add test split for imagenet

* add infinite buffer size to shuffler

* add prototype features

* add some JIT tests

* refactor input data handling

* refactor tests

* cleanup tests

* add BoundingBox feature

* mypy

* xfail torchscript tests for now

* cleanup

* fix imports

* add tests for builtin prototype datasets

* fix caltech101

* fix emnist

* fix mnist and variants

* add iopath as test requirement

* fix MNIST warning

* fix qmnist data generation

* fix cifar data generation

* add tests for imagenet

* cleanup