GitLab

* add deepspeed scheduled test for amd

* fix image

* add dockerfile

* add comment

* enable tests

* trigger

* remove trigger for this branch

* trigger

* change runner env to trigger the docker build image test

* use new docker image

* remove test suffix from docker image tag

* replace test docker image with original image

* push new image

* Trigger

* add back amd tests

* fix typo

* add amd tests back

* fix

* comment until docker image build scheduled test fix

* remove deprecated deepspeed build option

* upgrade torch

* update docker & make tests pass

* Update docker/transformers-pytorch-deepspeed-amd-gpu/Dockerfile

* fix

* tmp disable test

* precompile deepspeed to avoid timeout during tests

* fix comment

* trigger deepspeed tests with new image

* comment tests

* trigger

* add sklearn dependency to fix slow tests

* enable back other tests

* final update

---------
Co-authored-by: Felix Marty <felix@hf.co>
Co-authored-by: Félix Marty <9808326+fxmarty@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Update CUDA versions

* For testing

* Allow for workflow dispatch

* Use newer image

* Revert workflow

* Revert workflow

* Push

* Other docker image

Update docker files to use torch 2.1
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Limit Pydantic to V1 in dependencies

Pydantic is about to release V2 release which will break a lot of things. This change prevents `transformers` to be used with Pydantic V2 to avoid breaking things.

* more

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* fix

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* fix

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* unpin tf prob

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* update docker files to use official torch 2.0.0

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Revert "Use `dash==2.8.1` for now for daily CI (#22227)"

This reverts commit 53218671.

Use dash 2.8.1 for now
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Deal with torch-tensorrt

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Use torch 1.13.1 in push/scheduled CI
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Fix docker image build issue

* remove comment

* Add comment

* Update docker/transformers-pytorch-deepspeed-latest-gpu/Dockerfile
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>

Recompile apex in DeepSpeed CI image
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Uninstall torch_tensorrt for now
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* use explicit torch version
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* nightly build pytorch CI

* fix working dir

* change time and event name
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* pre-build deepspeed
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* update versions
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* [trainer / deepspeed] fix hyperparameter_search

* require optuna

* style

* oops

* add dep in the right place

* create deepspeed-testing dep group

* Trigger CI