Update Makefile (espnet#2225)

* Update Makefile - Change to remove chainer from requirements of setup.py and intall it from Makefile - chainer=6.0.0 has incompatible requirement of protobuf to tensorboardX espnet#2188 - New variables: - CPU_ONLY: Perform on CPU mode - CONDA: You can specify existing anaconda path - CONDA_ENV_NAME - CHAINER_VERSION - USE_PIP: Use pip to install torch even if it's anaconda - Romoved: - CUDA_VERSION: Detect it from `nvcc` - CUPY_VERSION: Use CHAINER_VERSION instead - Makefile creates tools/activate_python.sh. `path.sh` invokes it. - I changed the default PyTorch to 1.4.0 - Move tools/install_*.sh -> tools/installers/install_*.sh - Implement tools/setup_cuda_env.sh - A util to setup CUDA environment variables * Update setup.py Co-authored-by: Shinji Watanabe <[email protected]> * Update doc/installation.md Co-authored-by: Shinji Watanabe <[email protected]> * Update doc/installation.md Co-authored-by: Shinji Watanabe <[email protected]> * Update doc/installation.md Co-authored-by: Shinji Watanabe <[email protected]> * Update doc/installation.md Co-authored-by: Shinji Watanabe <[email protected]> * fix nargs: setup_cuda_env.sh * modified: Makefile * modified for $CONDA: ../doc/installation.md * modified for $CONDA: ../doc/installation.md * more logging: Makefile Co-authored-by: Shinji Watanabe <[email protected]>
airenas · Jul 28, 2020 · 1cf49b7 · 1cf49b7
1 parent 929a2dc
commit 1cf49b7
Show file tree

Hide file tree

Showing 115 changed files with 381 additions and 638 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -59,7 +59,7 @@ jobs:
       - run:
           name: codecov
           command: |
-            source ./tools/venv/bin/activate
+            . ./tools/activate_python.sh
             bash <(curl -s https://codecov.io/bash)
       - store_artifacts:
           path: egs/mini_an4/asr1/exp

diff --git a/.gitignore b/.gitignore
@@ -62,3 +62,4 @@ tools/PESQ*
 tools/hts_engine_API*
 tools/open_jtalk*
 tools/pyopenjtalk*
+tools/activate_python.sh
diff --git a/README.md b/README.md
@@ -96,13 +96,23 @@ See [ESPnet2](https://espnet.github.io/espnet/espnet2_tutorial.html).
 ## Installation
 - If you intend to do full experiments including DNN training, then see [Installation](https://espnet.github.io/espnet/installation.html).
 - If you just need the Python module only:
-    ```bash
-    pip install torch  # Install some dependencies manually
+    ```sh
     pip install espnet
     # To install latest
     # pip install git+https://github.com/espnet/espnet
     ```
 
+    You need to install some packages.
+
+    ```sh
+    pip install torch
+    pip install chainer==6.0.0 cupy==6.0.0    # [Option] If you'll use ESPnet1
+    pip install torchaudio                    # [Option] If you'll use enhancement task
+    pip install torch_optimizer               # [Option] If you'll use additional optimizers in ESPnet2
+    ```
+
+    There are some required packages depending on each task other than above. If you meet ImportError, please intall them at that time.
+
 ## Usage
 See [Usage](https://espnet.github.io/espnet/tutorial.html).
 
@@ -408,7 +418,7 @@ Available pretrained vocoder models in the demo script are listed as follows:
 
 ### VC results
 
-The [Voice Conversion Challenge 2020](http://www.vc-challenge.org/) (VCC2020) adopts ESPnet to build an end-to-end based baseline system. In VCC2020, the objective is intra/cross lingual nonparallel VC. A cascade method of ASR+TTS is developed.  
+The [Voice Conversion Challenge 2020](http://www.vc-challenge.org/) (VCC2020) adopts ESPnet to build an end-to-end based baseline system. In VCC2020, the objective is intra/cross lingual nonparallel VC. A cascade method of ASR+TTS is developed.
 You can download converted samples [here](https://drive.google.com/drive/folders/1oeZo83GrOgtqxGwF7KagzIrfjr8X59Ue?usp=sharing).
 
 

diff --git a/ci/doc.sh b/ci/doc.sh
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-. tools/venv/bin/activate
+. tools/activate_python.sh
 
 if [ ! -e tools/kaldi ]; then
     git clone https://github.com/kaldi-asr/kaldi --depth 1 tools/kaldi

diff --git a/ci/install.sh b/ci/install.sh
@@ -19,10 +19,7 @@ $CXX -v
     make moses.done
     rm kaldi.done
 )
-if [ -z "${PS1:-}" ]; then
-    PS1=__dummy__
-fi
-. tools/venv/bin/activate
+. tools/activate_python.sh
 python --version
 
 pip install -U wheel

diff --git a/ci/test_flake8.sh b/ci/test_flake8.sh
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-. tools/venv/bin/activate
+. tools/activate_python.sh
 
 set -euo pipefail
 

diff --git a/ci/test_python.sh b/ci/test_python.sh
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-. tools/venv/bin/activate
+. tools/activate_python.sh
 
 set -euo pipefail
 
@@ -15,6 +15,6 @@ fi
 # flake8
 "$(dirname $0)"/test_flake8.sh
 # pycodestyle
-pycodestyle -r ${modules} --show-source --show-pep8 
+pycodestyle -r ${modules} --show-source --show-pep8
 
-LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}:$(pwd)/tools/chainer_ctc/ext/warp-ctc/build" pytest -q
+LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}:$(pwd)/tools/chainer_ctc/ext/warp-ctc/build" pytest -q
diff --git a/ci/test_shell.sh b/ci/test_shell.sh
@@ -14,7 +14,7 @@ if ! [ -x "$(command -v shellcheck)" ]; then
     wget https://storage.googleapis.com/shellcheck/shellcheck-stable.linux.x86_64.tar.xz
     tar -xvf shellcheck-stable.linux.x86_64.tar.xz
 fi
-. tools/venv/bin/activate
+. tools/activate_python.sh
 
 set -euo pipefail
 

diff --git a/doc/faq.md b/doc/faq.md
@@ -4,12 +4,12 @@
 Firstly, you definitely missed some installation processes. Please read [Installation](./installation.md) again before posting an issue. If you still have a problem, then you have to understand the following items.
 
 ### How does ESPnet depend on a python interpreter?
-1. `egs/<some-where>/<some-task>/run.sh`, and any other shell scripts, always invokes `egs/<some-where>/<some-task>/path.sh` 
+1. `egs/<some-where>/<some-task>/run.sh`, and any other shell scripts, always invokes `egs/<some-where>/<some-task>/path.sh`
 1. `path.sh` setups the python interpreter at `tools/venv`
 1. `tools/venv` is installed by `tools/Makefile`
 
 Note that **we normally don't use the system Python, but uses `tools/venv` installed by Makefile**.
-See [Step 3-A) installation of espnet](installation.md#step-3-a-installation-of-espnet). We intend to prepare a Python interpreter independently and we'd also like to avoid contaminating the other Python. We believe this is a good strategy for a researcher, but sometimes it causes an installation problem. 
+See [Step 2) installation of espnet](installation.md#step-2-installation-of-espnet). We intend to prepare a Python interpreter independently and we'd also like to avoid contaminating the other Python. We believe this is a good strategy for a researcher, but sometimes it causes an installation problem.
 
 Please also note that, if you just intend to use Python scripts directly without a shell script, e.g. `asr_recog.py`, you always need to activate the Python interpreter before using it.
 

diff --git a/doc/installation.md b/doc/installation.md
@@ -4,7 +4,7 @@
 - Python 3.6.1+
 - gcc 4.9+ for PyTorch1.0.0+
 - cmake3 for some extensions
-    ```bash
+    ```sh
     # e.g. Ubuntu
     $ sudo apt-get install cmake
     # e.g. Using anaconda (If you don't have sudo privilege, the installation from conda might be useful)
@@ -14,7 +14,7 @@
 We often use audio converter tools in several recipes:
 
 - sox
-    ```bash
+    ```sh
     # e.g. Ubuntu
     $ sudo apt-get install sox
     # e.g. CentOS
@@ -23,23 +23,23 @@ We often use audio converter tools in several recipes:
     $ conda install -c conda-forge sox
     ```
 - sndfile
-    ```bash
+    ```sh
     # e.g. Ubuntu
     $ sudo apt-get install libsndfile1-dev
     # e.g. CentOS
     $ sudo yum install libsndfile
     ```
-- ffmpeg (This is not required when insllataion, but used in some recipes)
-    ```bash
+- ffmpeg (This is not required when installataion, but used in some recipes)
+    ```sh
     # e.g. Ubuntu
     $ sudo apt-get install ffmpeg
     # e.g. CentOS
     $ sudo yum install ffmpeg
     # e.g. Using anaconda
     $ conda install -c conda-forge ffmpeg
     ```
-- flac (This is not required when insllataion, but used in some recipes)
-    ```bash
+- flac (This is not required when installataion, but used in some recipes)
+    ```sh
     # e.g. Ubuntu
     $ sudo apt-get install flac
     # e.g. CentOS
@@ -64,37 +64,7 @@ to prepare the appropriate environments
 - debian9
 
 
-### Step 1) setting of the environment for GPU support
-
-To use cuda (and cudnn), make sure to set paths in your `.bashrc` or `.bash_profile` appropriately.
-```
-CUDAROOT=/path/to/cuda
-
-export PATH=$CUDAROOT/bin:$PATH
-export LD_LIBRARY_PATH=$CUDAROOT/lib64:$LD_LIBRARY_PATH
-export CFLAGS="-I$CUDAROOT/include $CFLAGS"
-export CPATH=$CUDAROOT/include:$CPATH
-export CUDA_HOME=$CUDAROOT
-export CUDA_PATH=$CUDAROOT
-```
-
-If you want to use multiple GPUs, you should install [nccl](https://developer.nvidia.com/nccl)
-and set paths in your `.bashrc` or `.bash_profile` appropriately, for example:
-```
-CUDAROOT=/path/to/cuda
-NCCL_ROOT=/path/to/nccl
-
-export CPATH=$NCCL_ROOT/include:$CPATH
-export LD_LIBRARY_PATH=$NCCL_ROOT/lib/:$CUDAROOT/lib64:$LD_LIBRARY_PATH
-export LIBRARY_PATH=$NCCL_ROOT/lib/:$LIBRARY_PATH
-export CFLAGS="-I$CUDAROOT/include $CFLAGS"
-export CPATH=$CUDAROOT/include:$CPATH
-export CUDA_HOME=$CUDAROOT
-export CUDA_PATH=$CUDAROOT
-```
-
-
-### Step 2) Install Kaldi
+### Step 1) Install Kaldi
 Related links:
 - [Kaldi Github](https://github.com/kaldi-asr/kaldi)
 - [Kaldi Documentation](https://kaldi-asr.org/)
@@ -111,40 +81,40 @@ We also have [prebuilt Kaldi binaries](https://github.com/espnet/espnet/blob/mas
 
 1. Git clone kaldi
 
-    ```bash
+    ```sh
     $ cd <any-place>
     $ git clone https://github.com/kaldi-asr/kaldi
     ```
 1. Install tools
 
-    ```bash
+    ```sh
     $ cd <kaldi-root>/tools
     $ make -j <NUM-CPU>
     ```
     1. Select BLAS library from ATLAS, OpenBLAS, or MKL
 
     - OpenBLAS
 
-    ```bash
+    ```sh
     $ cd <kaldi-root>/tools
     $ ./extras/install_openblas.sh
     ```
     - MKL (You need sudo privilege)
 
-    ```bash
+    ```sh
     $ cd <kaldi-root>/tools
     $ sudo ./extras/install_mkl.sh
     ```
     - ATLAS (You need sudo privilege)
 
-    ```bash
+    ```sh
     # Ubuntu
     $ sudo apt-get install libatlas-base-dev
     ```
 
 1. Compile Kaldi & install
 
-    ```bash
+    ```sh
     $ cd <kaldi-root>/src
     # [By default MKL is used] ESPnet uses only feature extractor, so you can disable CUDA
     $ ./configure --use-cuda=no
@@ -155,28 +125,58 @@ We also have [prebuilt Kaldi binaries](https://github.com/espnet/espnet/blob/mas
     $ make -j clean depend; make -j <NUM-CPU>
     ```
 
-### Step 3-A) installation of espnet
+### Step 2) installation of espnet
 
-```bash
+```sh
 $ cd <any-place>
 $ git clone https://github.com/espnet/espnet
 ```
 
-#### create miniconda environment (default)
+Before installing ESPnet, set up some environment variables related to CUDA.
+
+```sh
+$ cd <espnet-root>/tools
+$ . ./setup_cuda_env.sh /usr/local/cuda
+# If you have NCCL (If you'll install pytorch from anaconda, NCCL is also bundled, so you don't need to give it)
+# $ . ./setup_cuda_env.sh /usr/local/cuda /usr/local/nccl
+```
 
-Install Python libraries and other required tools with [miniconda](https://conda.io/docs/glossary.html#miniconda-glossary)
+#### A.) create miniconda environment (default)
 ```sh
 $ cd <espnet-root>/tools
 $ make KALDI=<kaldi-root>
 ```
 
-You can also specify the Python (`PYTHON_VERSION` default 3.7), PyTorch (`TH_VERSION` default 1.0.0) and CUDA versions (`CUDA_VERSION` default 10.0), for example:
+You can also specify the Python and PyTorch version, for example:
+```sh
+$ cd <espnet-root>/tools
+$ make KALDI=<kaldi-root> PYTHON_VERSION=3.6 TH_VERSION=1.3.1
+```
+
+By default, the environment is named `espnet`. If you prefer the other name,
+
+```sh
+$ cd <espnet-root>/tools
+$ make KALDI=<kaldi-root> CONDA_ENV_NAME=<name>
+```
+
+#### B.) create environment in existing anaconda/ change the installation path of anaconda
+
+If you already have anaconda and you'll create an environment of ESPnet.
+
 ```sh
 $ cd <espnet-root>/tools
-$ make KALDI=<kaldi-root> PYTHON_VERSION=3.6 TH_VERSION=0.4.1 CUDA_VERSION=9.0
+$ CONDA_TOOLS_DIR=$(dirname ${CONDA_EXE})/..
+$ make KALDI=<kaldi-root> CONDA=${CONDA_TOOLS_DIR} CONDA_ENV_NAME=<name>
 ```
 
-#### create virtualenv from an existing python
+Before executing this command, check the existance of `${CONDA_TOOLS_DIR}/etc/profile.d/conda.sh`
+
+Note that
+- If there are no conda tools at the path, new conda is created there.
+- If there already exists conda and its environment, the creation of a new environment is skipped.
+
+#### C.) create virtualenv from an existing python
 
 If you do not want to use miniconda, you need to specify your python interpreter to setup `virtualenv`
 
@@ -185,27 +185,47 @@ $ cd <espnet-root>/tools
 $ make KALDI=<kaldi-root> PYTHON=/usr/bin/python3.6
 ```
 
-#### using the system Python without creating new python environment
-We prepare a new Python interpreter independently in the Makefile, but if you'd like to use the System Python, for example, Google Colab originally provides an independent environment and you may use the Python as it is, then just create an empty file at `tools/venv/bin/activate`:
+In this case, you can't use `PYTHON_VERSION` option, but you can still use `TH_VERSION`.
+
+
+```sh
+$ cd <espnet-root>/tools
+$ make KALDI=<kaldi-root> PYTHON=/usr/bin/python3.6 TH_VERSION=1.3.1
+```
+
+#### D.) using the existing Python/Anaconda without creating new environment
+You can skip the installation of new environment by creating `activate_python.sh`.
+
+```sh
+$ cd <espnet-root>/tools
+$ rm -f activate_python.sh; touch activate_python.sh
+$ make KALDI=<kaldi-root> USE_PIP=0
+```
+
+If your Python is anaconda, you don't need to provide `USE_PIP`.
 
 ```sh
 $ cd <espnet-root>/tools
-$ rm -rf venv; mkdir -p venv/bin; touch venv/bin/activate  # Create an empty file
-$ make KALDI=<kaldi-root> PYTHON=dummy
+$ rm -f activate_python.sh; touch activate_python.sh
+$ make KALDI=<kaldi-root>
 ```
 
-### Step 3-B) installation for CPU-only
+In this case, you can use `TH_VERSION` too.
 
-To install in a terminal that does not have a GPU installed, just clear the version of `CUPY` as follows:
+#### E) installation for CPU-only
+
+If you don't have `nvcc` command, packages are installed for CPU mode by default.
+If you'll turn it on manually, give `CPU_ONLY` option.
 
 ```sh
 $ cd <espnet-root>/tools
-$ make KALDI=<kaldi-root> CUPY_VERSION='' -j 10
+$ make KALDI=<kaldi-root> CPU_ONLY=0
 ```
 
 This option is enabled for any of the install configuration.
 
-### Step 4) installation check
+
+### Step 3) installation check
 
 You can check whether the install is succeeded via the following commands
 ```sh

diff --git a/egs/aidatatang_200zh/asr1/path.sh b/egs/aidatatang_200zh/asr1/path.sh
@@ -9,11 +9,7 @@ export PATH=$PWD/utils/:$KALDI_ROOT/tools/openfst/bin:$KALDI_ROOT/tools/sctk/bin
 export LC_ALL=C
 
 export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$MAIN_ROOT/tools/chainer_ctc/ext/warp-ctc/build
-if [ -e $MAIN_ROOT/tools/venv/etc/profile.d/conda.sh ]; then
-    source $MAIN_ROOT/tools/venv/etc/profile.d/conda.sh && conda deactivate && conda activate
-else
-    source $MAIN_ROOT/tools/venv/bin/activate
-fi
+. $MAIN_ROOT/tools/activate_python.sh
 export PATH=$MAIN_ROOT/utils:$MAIN_ROOT/espnet/bin:$PATH
 
 export OMP_NUM_THREADS=1

diff --git a/egs/aishell/asr1/path.sh b/egs/aishell/asr1/path.sh
@@ -9,11 +9,7 @@ export PATH=$PWD/utils/:$KALDI_ROOT/tools/openfst/bin:$KALDI_ROOT/tools/sctk/bin
 export LC_ALL=C
 
 export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$MAIN_ROOT/tools/chainer_ctc/ext/warp-ctc/build
-if [ -e $MAIN_ROOT/tools/venv/etc/profile.d/conda.sh ]; then
-    source $MAIN_ROOT/tools/venv/etc/profile.d/conda.sh && conda deactivate && conda activate
-else
-    source $MAIN_ROOT/tools/venv/bin/activate
-fi
+. $MAIN_ROOT/tools/activate_python.sh
 export PATH=$MAIN_ROOT/utils:$MAIN_ROOT/espnet/bin:$PATH
 
 export OMP_NUM_THREADS=1