Add pre-built packages with combination of (cuda, torch) versions

Summary: Pull Request resolved: fairinternal/detectron2#410

Test Plan:
github
{F238145512}

readthedocs:
{F238145513}

Reviewed By: rbgirshick

Differential Revision: D21710420

Pulled By: ppwwyyxx

fbshipit-source-id: da055598e77d1ec095c685666c6a02f73f0fe403

GETTING_STARTED.md

@@ -13,8 +13,8 @@ For more advanced tutorials, refer to our [documentation](https://detectron2.rea
 ### Inference Demo with Pre-trained Models
 
 1. Pick a model and its config file from
-	[model zoo](MODEL_ZOO.md),
-	for example, `mask_rcnn_R_50_FPN_3x.yaml`.
+  [model zoo](MODEL_ZOO.md),
+  for example, `mask_rcnn_R_50_FPN_3x.yaml`.
 2. We provide `demo.py` that is able to run builtin standard models. Run it with:
 ```
 cd demo/
@@ -47,24 +47,24 @@ then run:
 ```
 cd tools/
 ./train_net.py --num-gpus 8 \
-	--config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml
+  --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml
 ```
 
 The configs are made for 8-GPU training.
 To train on 1 GPU, you may need to [change some parameters](https://arxiv.org/abs/1706.02677), e.g.:
 ```
 ./train_net.py \
-	--config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml \
-	--num-gpus 1 SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025
+  --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml \
+  --num-gpus 1 SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025
 ```
 
 For most models, CPU training is not supported.
 
 To evaluate a model's performance, use
 ```
 ./train_net.py \
-	--config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml \
-	--eval-only MODEL.WEIGHTS /path/to/checkpoint_file
+  --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml \
+  --eval-only MODEL.WEIGHTS /path/to/checkpoint_file
 ```
 For more options, see `./train_net.py -h`.
 

INSTALL.md

@@ -9,7 +9,7 @@ also installs detectron2 with a few simple commands.
 - Linux or macOS with Python ≥ 3.6
 - PyTorch ≥ 1.4
 - [torchvision](https://github.com/pytorch/vision/) that matches the PyTorch installation.
-	You can install them together at [pytorch.org](https://pytorch.org) to make sure of this.
+  You can install them together at [pytorch.org](https://pytorch.org) to make sure of this.
 - OpenCV, optional, needed by demo and visualization
 - pycocotools: `pip install cython; pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'`
 
@@ -34,20 +34,34 @@ To __rebuild__ detectron2 that's built from a local clone, use `rm -rf build/ **
 old build first. You often need to rebuild detectron2 after reinstalling PyTorch.
 
 ### Install Pre-Built Detectron2 (Linux only)
-```
-# for CUDA 10.2:
-python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/index.html
-```
-For other cuda versions, replace cu102 with "cu{101,92}" or "cpu".
+
+Choose from this table:
+
+<table class="docutils"><tbody><th width="80"> CUDA </th><th valign="bottom" align="left" width="100">torch 1.5</th><th valign="bottom" align="left" width="100">torch 1.4</th> <tr><td align="left">10.2</td><td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.5/index.html
+</code></pre> </details> </td> <td align="left"> </td> </tr> <tr><td align="left">10.1</td><td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html
+</code></pre> </details> </td> <td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.4/index.html
+</code></pre> </details> </td> </tr> <tr><td align="left">10.0</td><td align="left"> </td> <td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cu100/torch1.4/index.html
+</code></pre> </details> </td> </tr> <tr><td align="left">9.2</td><td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cu92/torch1.5/index.html
+</code></pre> </details> </td> <td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cu92/torch1.4/index.html
+</code></pre> </details> </td> </tr> <tr><td align="left">cpu</td><td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/torch1.5/index.html
+</code></pre> </details> </td> <td align="left"><details><summary> install </summary><pre><code>python -m pip install detectron2 -f \
+  https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/torch1.4/index.html
+</code></pre> </details> </td> </tr></tbody></table>
+
 
 Note that:
-1. Such installation has to be used with certain version of official PyTorch release.
-   See [releases](https://github.com/facebookresearch/detectron2/releases) for requirements.
+1. The pre-built package has to be used with corresponding version of CUDA and official PyTorch release.
    It will not work with a different version of PyTorch or a non-official build of PyTorch.
-	 The CUDA version used by PyTorch and detectron2 has to match as well.
 2. Such installation is out-of-date w.r.t. master branch of detectron2. It may not be
-	 compatible with the master branch of a research project that uses detectron2 (e.g. those in
-	 [projects](projects) or [meshrcnn](https://github.com/facebookresearch/meshrcnn/)).
+   compatible with the master branch of a research project that uses detectron2 (e.g. those in
+   [projects](projects) or [meshrcnn](https://github.com/facebookresearch/meshrcnn/)).
 
 ### Common Installation Issues
 
@@ -119,25 +133,25 @@ Two possibilities:
 
   To check whether it is the case,
   use `python -m detectron2.utils.collect_env` to find out inconsistent CUDA versions.
-	In the output of this command, you should expect "Detectron2 CUDA Compiler", "CUDA_HOME", "PyTorch built with - CUDA"
-	to contain cuda libraries of the same version.
+  In the output of this command, you should expect "Detectron2 CUDA Compiler", "CUDA_HOME", "PyTorch built with - CUDA"
+  to contain cuda libraries of the same version.
 
-	When they are inconsistent,
-	you need to either install a different build of PyTorch (or build by yourself)
-	to match your local CUDA installation, or install a different version of CUDA to match PyTorch.
+  When they are inconsistent,
+  you need to either install a different build of PyTorch (or build by yourself)
+  to match your local CUDA installation, or install a different version of CUDA to match PyTorch.
 
 * Detectron2 or PyTorch/torchvision is not built for the correct GPU architecture (compute compatibility).
 
-	The GPU architecture for PyTorch/detectron2/torchvision is available in the "architecture flags" in
-	`python -m detectron2.utils.collect_env`.
+  The GPU architecture for PyTorch/detectron2/torchvision is available in the "architecture flags" in
+  `python -m detectron2.utils.collect_env`.
 
-	The GPU architecture flags of detectron2/torchvision by default matches the GPU model detected
-	during compilation. This means the compiled code may not work on a different GPU model.
-	To overwrite the GPU architecture for detectron2/torchvision, use `TORCH_CUDA_ARCH_LIST` environment variable during compilation.
+  The GPU architecture flags of detectron2/torchvision by default matches the GPU model detected
+  during compilation. This means the compiled code may not work on a different GPU model.
+  To overwrite the GPU architecture for detectron2/torchvision, use `TORCH_CUDA_ARCH_LIST` environment variable during compilation.
 
-	For example, `export TORCH_CUDA_ARCH_LIST=6.0,7.0` makes it compile for both P100s and V100s.
-	Visit [developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus) to find out
-	the correct compute compatibility number for your device.
+  For example, `export TORCH_CUDA_ARCH_LIST=6.0,7.0` makes it compile for both P100s and V100s.
+  Visit [developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus) to find out
+  the correct compute compatibility number for your device.
 
 </details>
 
@@ -165,10 +179,11 @@ to match your local CUDA installation, or install a different version of CUDA to
 C++ compilation errors from NVCC
 </summary>
 <br/>
+
 1. NVCC version has to match the CUDA version of your PyTorch.
 
 2. NVCC has compatibility issues with certain versions of gcc. You may need a different
-	 version of gcc. The version used by PyTorch can be found by `print(torch.__config__.show())`.
+   version of gcc. The version used by PyTorch can be found by `print(torch.__config__.show())`.
 </details>
 
 

detectron2/utils/collect_env.py

@@ -71,18 +71,39 @@ def collect_env_info():
         )
     except ImportError:
         data.append(("detectron2", "failed to import"))
+
+    try:
+        from detectron2 import _C
+    except ImportError:
+        data.append(("detectron2._C", "failed to import"))
+
+        # print system compilers when extension fails to build
+        if sys.platform != "win32":  # don't know what to do for windows
+            try:
+                # this is how torch/utils/cpp_extensions.py choose compiler
+                cxx = os.environ.get("CXX", "c++")
+                cxx = subprocess.check_output("'{}' --version".format(cxx), shell=True)
+                cxx = cxx.decode("utf-8").strip().split("\n")[0]
+            except subprocess.SubprocessError:
+                cxx = "Not found"
+            data.append(("Compiler", cxx))
+
+            if has_cuda and CUDA_HOME is not None:
+                try:
+                    nvcc = os.path.join(CUDA_HOME, "bin", "nvcc")
+                    nvcc = subprocess.check_output("'{}' -V".format(nvcc), shell=True)
+                    nvcc = nvcc.decode("utf-8").strip().split("\n")[-1]
+                except subprocess.SubprocessError:
+                    nvcc = "Not found"
+                data.append(("CUDA compiler", nvcc))
     else:
-        try:
-            from detectron2 import _C
-        except ImportError:
-            data.append(("detectron2._C", "failed to import"))
-        else:
-            data.append(("detectron2 compiler", _C.get_compiler_version()))
-            data.append(("detectron2 CUDA compiler", _C.get_cuda_version()))
-            if has_cuda:
-                data.append(
-                    ("detectron2 arch flags", detect_compute_compatibility(CUDA_HOME, _C.__file__))
-                )
+        # print compilers that are used to build extension
+        data.append(("Compiler", _C.get_compiler_version()))
+        data.append(("CUDA compiler", _C.get_cuda_version()))
+        if has_cuda:
+            data.append(
+                ("detectron2 arch flags", detect_compute_compatibility(CUDA_HOME, _C.__file__))
+            )
 
     data.append(get_env_module())
     data.append(("PyTorch", torch.__version__ + " @" + os.path.dirname(torch.__file__)))
@@ -100,15 +121,6 @@ def collect_env_info():
 
         data.append(("CUDA_HOME", str(CUDA_HOME)))
 
-        if CUDA_HOME is not None and os.path.isdir(CUDA_HOME):
-            try:
-                nvcc = os.path.join(CUDA_HOME, "bin", "nvcc")
-                nvcc = subprocess.check_output("'{}' -V | tail -n1".format(nvcc), shell=True)
-                nvcc = nvcc.decode("utf-8").strip()
-            except subprocess.SubprocessError:
-                nvcc = "Not Available"
-            data.append(("NVCC", nvcc))
-
         cuda_arch_list = os.environ.get("TORCH_CUDA_ARCH_LIST", None)
         if cuda_arch_list:
             data.append(("TORCH_CUDA_ARCH_LIST", cuda_arch_list))

dev/packaging/build_all_wheels.sh

@@ -1,10 +1,14 @@
 #!/bin/bash -e
 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
 
-PYTORCH_VERSION=1.5
+[[ -d "dev/packaging" ]] || {
+  echo "Please run this script at detectron2 root!"
+  exit 1
+}
 
-build_for_one_cuda() {
+build_one() {
   cu=$1
+  pytorch_ver=$2
 
   case "$cu" in
     cu*)
@@ -29,29 +33,27 @@ build_for_one_cuda() {
 
     cat <<EOF | docker exec -i $container_name sh
       export CU_VERSION=$cu D2_VERSION_SUFFIX=+$cu PYTHON_VERSION=$py
-      export PYTORCH_VERSION=$PYTORCH_VERSION
+      export PYTORCH_VERSION=$pytorch_ver
       cd /detectron2 && ./dev/packaging/build_wheel.sh
 EOF
 
-#     if [[ "$cu" == "cu101" ]]; then
-#       # build wheel without local version
-#       cat <<EOF | docker exec -i $container_name sh
-#         export CU_VERSION=$cu D2_VERSION_SUFFIX= PYTHON_VERSION=$py
-#         export PYTORCH_VERSION=$PYTORCH_VERSION
-#         cd /detectron2 && ./dev/packaging/build_wheel.sh
-# EOF
-#     fi
-
     docker exec -i $container_name rm -rf /detectron2/build/$cu
     docker container stop $container_name
     docker container rm $container_name
   done
 }
 
-if [[ -n "$1" ]]; then
-  build_for_one_cuda "$1"
+
+if [[ -n "$1" ]] && [[ -n "$2" ]]; then
+  build_one "$1" "$2"
 else
-  for cu in cu102 cu101 cu92 cpu; do
-    build_for_one_cuda "$cu"
-  done
+  build_one cu102 1.5
+  build_one cu101 1.5
+  build_one cu92 1.5
+  build_one cpu 1.5
+
+  build_one cu101 1.4
+  build_one cu100 1.4
+  build_one cu92 1.4
+  build_one cpu 1.4
 fi

dev/packaging/build_wheel.sh

@@ -17,16 +17,13 @@ setup_cuda
 setup_wheel_python
 yum install ninja-build -y && ln -sv /usr/bin/ninja-build /usr/bin/ninja
 
-export TORCH_VERSION_SUFFIX="+$CU_VERSION"
-if [[ "$CU_VERSION" == "cu102" ]]; then
-	export TORCH_VERSION_SUFFIX=""
-fi
 pip_install pip numpy -U
-pip_install "torch==$PYTORCH_VERSION$TORCH_VERSION_SUFFIX" \
-	-f https://download.pytorch.org/whl/$CU_VERSION/torch_stable.html
+# use --no-index to avoid mistakenly download wrong cuda version from pypi
+pip_install "torch==$PYTORCH_VERSION" \
+	--no-index -f https://download.pytorch.org/whl/"$CU_VERSION"/torch_stable.html
 
 # use separate directories to allow parallel build
-BASE_BUILD_DIR=build/$CU_VERSION/$PYTHON_VERSION
+BASE_BUILD_DIR=build/cu$CU_VERSION-py$PYTHON_VERSION-pt$PYTORCH_VERSION
 python setup.py \
-  build -b $BASE_BUILD_DIR \
-  bdist_wheel -b $BASE_BUILD_DIR/build_dist -d wheels/$CU_VERSION
+  build -b "$BASE_BUILD_DIR" \
+  bdist_wheel -b "$BASE_BUILD_DIR/build_dist" -d "wheels/$CU_VERSION/torch$PYTORCH_VERSION"

dev/packaging/gen_install_table.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+
+template = """<details><summary> install </summary><pre><code>python -m pip install detectron2 -f \\
+  https://dl.fbaipublicfiles.com/detectron2/wheels/{cuda}/torch{torch}/index.html
+</code></pre> </details>"""
+CUDA_SUFFIX = {"10.2": "cu102", "10.1": "cu101", "10.0": "cu100", "9.2": "cu92", "cpu": "cpu"}
+
+
+def gen_header(torch_versions):
+    return '<table class="docutils"><tbody><th width="80"> CUDA </th>' + "".join(
+        [
+            '<th valign="bottom" align="left" width="100">torch {}</th>'.format(t)
+            for t in torch_versions
+        ]
+    )
+
+
+if __name__ == "__main__":
+    all_versions = [("1.4", k) for k in ["10.1", "10.0", "9.2", "cpu"]] + [
+        ("1.5", k) for k in ["10.2", "10.1", "9.2", "cpu"]
+    ]
+
+    torch_versions = sorted({k[0] for k in all_versions}, key=float, reverse=True)
+    cuda_versions = sorted(
+        {k[1] for k in all_versions}, key=lambda x: float(x) if x != "cpu" else 0, reverse=True
+    )
+
+    table = gen_header(torch_versions)
+    for cu in cuda_versions:
+        table += f""" <tr><td align="left">{cu}</td>"""
+        cu_suffix = CUDA_SUFFIX[cu]
+        for torch in torch_versions:
+            if (torch, cu) in all_versions:
+                cell = template.format(cuda=cu_suffix, torch=torch)
+            else:
+                cell = ""
+            table += f"""<td align="left">{cell} </td> """
+        table += "</tr>"
+    table += "</tbody></table>"
+    print(table)

dev/packaging/gen_wheel_index.sh

@@ -8,20 +8,38 @@ if [[ -z "$root" ]]; then
   exit
 fi
 
+export LC_ALL=C  # reproducible sort
+# NOTE: all sort in this script might not work when xx.10 is released
+
 index=$root/index.html
 
 cd "$root"
 for cu in cpu cu92 cu100 cu101 cu102; do
-  cd $cu
+  cd "$root/$cu"
   echo "Creating $PWD/index.html ..."
-  for whl in *.whl; do
+  # First sort by torch version, then stable sort by d2 version with unique.
+  # As a result, the latest torch version for each d2 version is kept.
+  for whl in $(find -type f -name '*.whl' -printf '%P\n' \
+    | sort -k 1 -r  | sort -t '/' -k 2 --stable -r --unique); do
     echo "<a href=\"${whl/+/%2B}\">$whl</a><br>"
   done > index.html
-  cd "$root"
+
+
+  for torch in torch*; do
+    cd "$root/$cu/$torch"
+
+    # list all whl for each cuda,torch version
+    echo "Creating $PWD/index.html ..."
+    for whl in $(find . -type f -name '*.whl' -printf '%P\n' | sort -r); do
+      echo "<a href=\"${whl/+/%2B}\">$whl</a><br>"
+    done > index.html
+  done
 done
 
+cd "$root"
+# Just list everything:
 echo "Creating $index ..."
-for whl in $(find . -type f -name '*.whl' -printf '%P\n' | sort); do
+for whl in $(find . -type f -name '*.whl' -printf '%P\n' | sort -r); do
   echo "<a href=\"${whl/+/%2B}\">$whl</a><br>"
 done > "$index"
 

docker/Dockerfile-circleci

@@ -11,7 +11,6 @@ RUN wget -q https://bootstrap.pypa.io/get-pip.py && \
 	rm get-pip.py
 
 # install dependencies
-# See https://pytorch.org/ for other options if you use a different version of CUDA
 RUN pip install tensorboard cython
 RUN pip install torch==1.5+cu101 torchvision==0.6+cu101 -f https://download.pytorch.org/whl/torch_stable.html
 RUN pip install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'