Limit Pydantic to V1 in dependencies (#24596)

* Limit Pydantic to V1 in dependencies

Pydantic is about to release V2 release which will break a lot of things. This change prevents `transformers` to be used with Pydantic V2 to avoid breaking things.

* more

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

docker/transformers-pytorch-deepspeed-latest-gpu/Dockerfile

@@ -50,5 +50,5 @@ RUN DS_BUILD_CPU_ADAM=1 DS_BUILD_FUSED_ADAM=1 DS_BUILD_UTILS=1 python3 -m pip in
 RUN cd transformers && python3 setup.py develop
 
 # The base image ships with `pydantic==1.8.2` which is not working - i.e. the next command fails
-RUN python3 -m pip install -U --no-cache-dir pydantic
+RUN python3 -m pip install -U --no-cache-dir pydantic<2
 RUN python3 -c "from deepspeed.launcher.runner import main"

setup.py

@@ -145,7 +145,7 @@ _deps = [
     "protobuf",
     "psutil",
     "pyyaml>=5.1",
-    "pydantic",
+    "pydantic<2",
     "pytest>=7.2.0",
     "pytest-timeout",
     "pytest-xdist",

src/transformers/dependency_versions_table.py

@@ -50,7 +50,7 @@ deps = {
     "protobuf": "protobuf",
     "psutil": "psutil",
     "pyyaml": "pyyaml>=5.1",
-    "pydantic": "pydantic",
+    "pydantic": "pydantic<2",
     "pytest": "pytest>=7.2.0",
     "pytest-timeout": "pytest-timeout",
     "pytest-xdist": "pytest-xdist",