[CI] Run unit test with non-editable installation (#845)

We have been running unit test with editable installation. (i.e. `python setup.py develop`), with which we missed issues like #842. 

This CC makes installation in CI non-editable, and change test directory structure so that the source code will not shadow the installed version of `torchaudio`. With simple `pytest test`, `pytest` modifies `sys.path` and prepend checked out repository, which shadows the installed version.

To remedy this, the whole test suites has been moved from `./test` to `./test/torchaudio_unittest`. This adds nice module structure to our test code and we can do absolute import in each test module, which makes it possible again to run test with `python -m unittest torchaudio_unittest/XXX.py`

This change does not affect the regular development process (`python setup.py develop` && `pytest test`)

.circleci/unittest/linux/scripts/install.sh

@@ -20,4 +20,4 @@ printf "Installing PyTorch with %s\n" "${cudatoolkit}"
 conda install -y -c pytorch-nightly pytorch "${cudatoolkit}"
 
 printf "* Installing torchaudio\n"
-BUILD_SOX=1 python setup.py develop
+BUILD_SOX=1 python setup.py install

.circleci/unittest/linux/scripts/run_test.sh

@@ -15,8 +15,15 @@ python -m torch.utils.collect_env
 export TORCHAUDIO_TEST_FAIL_IF_NO_EXTENSION=1
 export PATH="${PWD}/third_party/install/bin/:${PATH}"
 
+declare -a common_args=(
+    '--cov=torchaudio'
+    "--junitxml=${PWD}/test-results/junit.xml"
+    '--durations' '20'
+)
 if [ "${os}" == MacOSX ] ; then
-    pytest -q -n auto --dist=loadscope --cov=torchaudio --junitxml=test-results/junit.xml --durations 20 test
+    declare -a args=('-q' '-n' 'auto' '--dist=loadscope')
 else
-    pytest -v --cov=torchaudio --junitxml=test-results/junit.xml --durations 20 test
+    declare -a args=('-v')
 fi
+cd test
+pytest "${args[@]}" "${common_args[@]}" torchaudio_unittest

.circleci/unittest/windows/scripts/install.sh

@@ -20,4 +20,4 @@ printf "Installing PyTorch with %s\n" "${cudatoolkit}"
 conda install -y -c pytorch-nightly pytorch "${cudatoolkit}"
 
 printf "* Installing torchaudio\n"
-python setup.py develop
+python setup.py install

.circleci/unittest/windows/scripts/run_test.sh

@@ -6,5 +6,5 @@ eval "$(./conda/Scripts/conda.exe 'shell.bash' 'hook')"
 conda activate ./env
 
 python -m torch.utils.collect_env
-pytest --cov=torchaudio --junitxml=test-results/junit.xml -v --durations 20 test
-flake8 torchaudio test
+cd test
+pytest --cov=torchaudio --junitxml=../test-results/junit.xml -v --durations 20 torchaudio_unittest

test/README.md

@@ -29,39 +29,39 @@ The following is an overview of the tests and related modules for `torchaudio`.
 ### Purpose specific test suites
 
 #### Numerical compatibility agains existing software
-- [Librosa compatibility test](./test_librosa_compatibility.py)
+- [Librosa compatibility test](./torchaudio_unittest/librosa_compatibility_test.py)
     Test suite for numerical compatibility against librosa.
-- [SoX compatibility test](./test_sox_compatibility.py)
+- [SoX compatibility test](./torchaudio_unittest/sox_compatibility_test.py)
     Test suite for numerical compatibility against SoX.
-- [Kaldi compatibility test](./test_kaldi_compatibility.py)
+- [Kaldi compatibility test](./torchaudio_unittest/kaldi_compatibility_test.py)
     Test suite for numerical compatibility against Kaldi.
 
 #### Result consistency with PyTorch framework
-- [TorchScript consistency test](torchscript_consistency_impl.py)
+- [TorchScript consistency test](./torchaudio_unittest/torchscript_consistency_impl.py)
     Test suite to check 1. if an API is TorchScript-able, and 2. the results from Python and Torchscript match.
-- [Batch consistency test](./test_batch_consistency.py)
+- [Batch consistency test](./torchaudio_unittest/batch_consistency_test.py)
     Test suite to check if functionals/Transforms handle single sample input and batch input and return the same result.
 
 ### Module specific test suites
 
 The following test modules are defined for corresponding `torchaudio` module/functions.
 
-- [`torchaudio.datasets`](./test_datasets)
-- `torchaudio.functional` [CPU](./functional_cpu_test.py), [CUDA](./functional_cuda_test.py), [common](./functional_impl.py)
-- [`torchaudio.transforms`](./test_transforms.py)
-- [`torchaudio.compliance.kaldi`](./test_compliance_kaldi.py)
-- [`torchaudio.kaldi_io`](./test_kaldi_io.py)
+- [`torchaudio.datasets`](./torchaudio_unittest/datasets)
+- [`torchaudio.functional`](./torchaudio_unittest/functional)
+- [`torchaudio.transforms`](./torchaudio_unittest/transforms_test.py)
+- [`torchaudio.compliance.kaldi`](./torchaudio_unittest/compliance_kaldi_test.py)
+- [`torchaudio.kaldi_io`](./torchaudio_unittest/kaldi_io_test.py)
 - [`torchaudio.sox_effects`](test/sox_effects)
-- [`torchaudio.save`, `torchaudio.load`, `torchaudio.info`](test/test_io.py)
+- [`torchaudio.save`, `torchaudio.load`, `torchaudio.info`](./torchaudio_unittest/io_test.py)
 
 ### Test modules that do not fall into the above categories
-- [test_dataloader.py](./test_dataloader.py)
+- [test_dataloader.py](./torchaudio_unittest/dataloader_test.py)
     Simple test for loading data and applying preprocessing.
 
 ### Support files
-- [assets](./assets): Contain sample audio files.
-- [assets/kaldi](./assets/kaldi): Contains Kaldi format matrix files used in [./test_compliance_kaldi.py](./test_compliance_kaldi.py).
-- [compliance](./compliance): Scripts used to generate above Kaldi matrix files.
+- [assets](./torchaudio_unittest/assets): Contain sample audio files.
+- [assets/kaldi](./torchaudio_unittest/assets/kaldi): Contains Kaldi format matrix files used in [./torchaudio_unittest/test_compliance_kaldi.py](./torchaudio_unittest/test_compliance_kaldi.py).
+- [compliance](./torchaudio_unittest/compliance): Scripts used to generate above Kaldi matrix files.
 
 ### Waveforms for Testing Purposes
 
@@ -116,13 +116,13 @@ Files:
 
 The following is the current practice of torchaudio test suite.
 
-1. Unless the tests are related to I/O, use synthetic data. [`common_utils`](./common_utils.py) has some data generator functions.
+1. Unless the tests are related to I/O, use synthetic data. [`common_utils`](./torchaudio_unittest/common_utils) has some data generator functions.
 1. When you add a new test case, use `common_utils.TorchaudioTestCase` as base class unless you are writing tests that are common to CPU / CUDA.
   - Set class memeber `dtype`, `device` and `backend` for the desired behavior.
   - If you do not set `backend` value in your test suite, then I/O functions will be unassigned and attempt to load/save file will fail.
   - For `backend` value, in addition to available backends, you can also provide the value "default" and backend will be picked automatically based on availability.
-1. If you are writing tests that should pass on diffrent dtype/devices, write a common class inheriting `common_utils.TestBaseMixin`, then inherit `common_utils.PytorchTestCase` and define class attributes (`dtype` / `device` / `backend`) there. See [Torchscript consistency test implementation](./torchscript_consistency_impl.py) and test definitions for [CPU](./torchscript_consistency_cpu_test.py) and [CUDA](./torchscript_consistency_cuda_test.py) devices.
-1. For numerically comparing Tensors, use `assertEqual` method from `common_utils.PytorchTestCase` class. This method has a better support for a wide variety of Tensor types.
+1. If you are writing tests that should pass on diffrent dtype/devices, write a common class inheriting `common_utils.TestBaseMixin`, then inherit `common_utils.PytorchTestCase` and define class attributes (`dtype` / `device` / `backend`) there. See [Torchscript consistency test implementation](./torchaudio_unittest/torchscript_consistency_impl.py) and test definitions for [CPU](./torchaudio_unittest/torchscript_consistency_cpu_test.py) and [CUDA](./torchaudio_unittest/torchscript_consistency_cuda_test.py) devices.
+1. For numerically comparing Tensors, use `assertEqual` method from torchaudio_unittest.common_utils.PytorchTestCase` class. This method has a better support for a wide variety of Tensor types.
 
 When you add a new feature(functional/transform), consider the following