SINGA-89 Add Docker support

Merge branch 'feature-docker'

tool/docker/README.md

@@ -0,0 +1,189 @@
+This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes. 
+
+1. [Build SINGA base](#build_base)
+2. [Build SINGA with Mesos and Hadoop](#build_mesos)
+3. [Pre-built images](#pre_built)
+4. [Launch and stop SINGA (stand alone mode)](#launch_stand_alone)
+5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo)
+6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed)
+
+---
+
+<a name="build_base"></a>
+#### Build SINGA base image
+
+````
+$ cd tool/docker/singa
+$ sudo docker build -t singa/base . 
+$ sudo docker images
+REPOSITORY             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
+singa/base             latest              XXXX                XXX                 2.01 GB
+````
+
+The result is the image containing a built version of SINGA. 
+
+   ![singa/base](http://www.comp.nus.edu.sg/~dinhtta/files/images_base.png)
+
+   *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.*
+
+--
+
+<a name="build_mesos"></a>
+#### Build SINGA with Mesos and Hadoop
+````
+$ cd tool/docker/mesos
+$ sudo docker build -t singa/mesos .
+$ sudo docker images
+REPOSITORY             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
+singa/mesos             latest              XXXX                XXX                 4.935 GB
+````
+   ![singa/mesos](http://www.comp.nus.edu.sg/~dinhtta/files/images_mesos.png#1)
+
+   *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on
+top of SINGA. The default namenode address for Hadoop is `node0:9000`*
+
+**Notes** A common failure observed during the build process is caused by network failure occuring when downloading dependencies. Simply re-run the build command. 
+
+--
+
+<a name="pre_built"></a>
+#### Pre-built images on epiC cluster
+For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes:
+
+      ciidaa-c18
+      ciidaa-c19
+
+The available images at those nodes are:
+````
+REPOSITORY             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
+singa/base             latest              XXXX                XXX                 2.01 GB
+singa/mesos            latest              XXXX                XXX                 4.935 GB
+weaveworks/weaveexec   1.1.1               XXXX                11 days ago         57.8 MB
+weaveworks/weave       1.1.1               XXXX                11 days ago         17.56 MB
+````
+
+--
+
+<a name="launch_stand_alone"></a>
+#### Launch and stop SINGA in stand-alone mode
+To launch a test environment for a single-node SINGA training, simply start a container from `singa/base` image. The following starts a container called
+`XYZ`, then launches a shell in the container: 
+
+````
+$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord
+$ sudo docker exec -it XYZ /bin/bash
+````
+
+   ![](http://www.comp.nus.edu.sg/~dinhtta/files/images_standalone.png#1)
+
+   *Figure 3. Launch SINGA in stand-alone mode: single node training*
+
+Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa`. 
+
+**Stopping the container**
+
+````
+$ sudo docker stop XYZ
+$ sudo docker rm ZYZ
+````
+
+--
+
+<a name="launch_pseudo"></a>
+#### Launch SINGA on pseudo-distributed mode (single node)
+To simulate a distributed environment on a single node, one can repeat the
+previous step multiple times, each time giving a different name to the
+container.  Network connections between these containers are already supported,
+thus SINGA instances/nodes in these container can readily communicate with each
+other. 
+
+The previous approach requires the user to start SINGA instances individually
+at each container. Although there's a bash script for that, we provide a better
+way. In particular, multiple containers can be started from `singa/mesos` image
+which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to
+launch, stop and monitor the distributed execution from a single container.
+Figure 4 shows `N+1` containers running concurrently at the local host. 
+
+````
+# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord
+# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord
+...
+````
+   ![](http://www.comp.nus.edu.sg/~dinhtta/files/images_pseudo.png#1)
+
+   *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine*
+
+**Starting SINGA distributed training**
+
+Refer to the [Mesos
+guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md)
+for details of how to start training with multiple SINGA instances. 
+
+**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately. 
+
+--
+
+<a name="launch_distributed"></a>
+#### Launch SINGA on fully distributed mode (multiple nodes)
+The previous section has explained how to start a distributed environment on a
+single node. But running many containers on one node does not scale. When there
+are multiple physical hosts available, it is better to distribute the
+containers over them. 
+
+The only extra requirement for the fully distributed mode, as compared with the
+pseudo distributed mode, is that the containers from different hosts are able
+to transparently communicate with each other. In the pseudo distributed mode,
+the local docker engine takes care of such communication. Here, we rely on
+[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the
+communication transparent. The resulting architecture is shown below.  
+
+   ![](http://www.comp.nus.edu.sg/~dinhtta/files/images_full.png#1)
+
+   *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines*
+
+**Install Weave at all hosts**
+```
+$ curl -L git.io/weave -o /usr/local/bin/weave
+$ chmod a+x /usr/local/bin/weave
+```
+
+**Starting Weave**
+
+Suppose `node0` will be launched at host with IP `111.222.111.222`.
+
+* At host `111.222.111.222`:
+````
+$ weave launch
+$ eval "$(weave env)"  //if there's error, do `sudo -s` and try again
+````
+* At other hosts:
+````
+$ weave launch 111.222.111.222
+$ eval "$(weave env)" //if there's error, do `sudo -s` and try again
+````
+
+**Starting containers**
+
+The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example). 
+
+**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example,
+there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit
+`/etc/hosts` file:
+
+````
+# modified by weave
+...
+X.Y.Z	node0 node0.bridge  //<- REMOVE this line
+..
+````
+This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default,
+containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use
+addressed given by Weave.  
+
+
+**Starting SINGA distributed training**
+
+Refer to the [Mesos
+guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md)
+for details of how to start training with multiple SINGA instances. 
+

tool/docker/mesos/.bashrc

@@ -0,0 +1,10 @@
+export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH
+export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
+export PATH=/opt/jdk1.8.0_60/bin:/opt/bin:$PATH
+export HADOOP_HOME=/opt/hadoop-2.6.0
+export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
+export JAVA_HOME=/opt/jdk1.8.0_60
+alias ls="ls --color=always"
+export SINGA_HOME=/root/incubator-singa
+export PATH=$PATH:$SINGA_HOME/bin

tool/docker/mesos/Dockerfile

@@ -0,0 +1,21 @@
+#build Mesos+Hadoop on top of SINGA
+FROM singa/base
+MAINTAINER Anh Dinh<[email protected]>
+
+COPY .bashrc /root/.bashrc
+COPY install.sh /opt/install.sh
+
+RUN cd /opt && source ./install.sh
+
+RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev
+
+RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install
+
+#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master)
+#RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make 
+
+RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make
+
+COPY *.xml /opt/hadoop-2.6.0/etc/hadoop/
+
+RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config

tool/docker/mesos/core-site.xml

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+  <property>
+    <name>fs.default.name</name>
+    <value>hdfs://node0:9000</value>
+  </property>
+</configuration>

tool/docker/mesos/hdfs-site.xml

@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+<property>
+    <name>dfs.replication</name>
+    <value>1</value>
+  </property>
+
+<property>
+    <name>dfs.name.dir</name>
+    <value>file:///var/lib/hadoop/hdfs/namenode</value>
+  </property>
+
+  <property>
+    <name>dfs.data.dir</name>
+    <value>file:///var/lib/hadoop/hdfs/datanode</value>
+  </property>
+</configuration>

tool/docker/mesos/install.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+source /root/.bashrc
+#download
+cd /opt
+wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz
+wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch 
+tar -zxvf mesos-0.22.0.tar.gz
+
+#patch and install mesos
+cd /opt/mesos-0.22.0
+patch -p5 < ../mesos_patch
+mkdir build; cd build
+../configure
+make
+sudo make install
+

tool/docker/mesos/mapred-site.xml

@@ -0,0 +1,22 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+  <property>
+    <name>mapreduce.framework.name</name>
+    <value>yarn</value>
+  </property>
+</configuration>

tool/docker/mesos/yarn-site.xml

@@ -0,0 +1,32 @@
+<?xml version="1.0"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+<configuration>
+
+<!-- Site specific YARN configuration properties -->
+  <property>
+    <name>yarn.nodemanager.aux-services</name>
+    <value>mapreduce_shuffle</value>
+  </property>
+
+  <property>
+    <name>yarn.resourcemanager.hostname</name>
+    <value>node0</value>
+  </property>
+
+  <property>
+    <name>yarn.resourcemanager.bind-host</name>
+    <value>0.0.0.0</value>
+  </property>
+</configuration>

tool/docker/singa/.bashrc

@@ -0,0 +1,8 @@
+export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH
+export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
+export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin
+export JAVA_HOME=/opt/jdk1.8.0_60
+export SINGA_HOME=/root/incubator-singa
+export PATH=$PATH:$SINGA_HOME/bin
+alias ls="ls --color=always"

tool/docker/singa/.vimrc

@@ -0,0 +1,7 @@
+set autoindent              " always set autoindenting on
+set smartindent             " use smart indent if there is no indent file
+set tabstop=2               " <tab> inserts 4 spaces
+set shiftwidth=2            " but an indent level is 2 spaces wide.
+set softtabstop=2           " <BS> over an autoindent deletes both spaces.
+"set expandtab               " Use spaces, not tabs, for autoindent/tab key.
+set shiftround              " rounds indent to a multiple of shiftwidth

tool/docker/singa/Dockerfile

@@ -0,0 +1,15 @@
+#build singa from scratch
+FROM ubuntu:14.04
+MAINTAINER Anh Dinh<[email protected]>
+RUN rm /bin/sh && ln -s /bin/bash /bin/sh
+
+RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor
+
+RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie"  http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz
+
+COPY ssh.conf /etc/supervisor/conf.d/
+COPY .bashrc /root/.bashrc
+copy .vimrc /root/.vimrc
+
+RUN cd && source /root/.bashrc && git clone https://github.com/apache/incubator-singa.git && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start
+

tool/docker/singa/ssh.conf

@@ -0,0 +1,5 @@
+[supervisord]
+nodaemon=true
+
+[program:ssh]
+command=/usr/sbin/sshd -D