[Build] Remove pyproject.toml

I haven't found an easy way to add torch as a build dependency in
pyproject.toml.
If we add torch in pyproject.toml, for some setup it would download a
different version of Pytorch before building.
If we don't add torch, lots of users report they get error when installing.

pyproject.toml

-[build-system]
-requires = ["ninja", "packaging", "setuptools", "wheel"]
-build-backend = "setuptools.build_meta"