Merge remote-tracking branch 'upstream/master' into feat/specify-list…

…-of-sync

.gitignore

@@ -41,6 +41,7 @@ build-iPhoneSimulator/
 /_yardoc/
 /doc/
 /rdoc/
+/docs/_build/
 
 ## Environment normalization:
 /.bundle/

.travis.yml

@@ -1,8 +1,9 @@
 language: ruby
 
 rvm:
-  - 2.0
-  - 2.1
+# Removed those to since bundler is to old on those 2
+#  - 2.0
+#  - 2.1
   - 2.2
   - 2.3
   - 2.4

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    docker-sync (0.5.10)
+    docker-sync (0.5.11)
       daemons (~> 1.2, >= 1.2.3)
       docker-compose (~> 1.1, >= 1.1.7)
       dotenv (~> 2.1, >= 2.1.1)

README.md

@@ -13,7 +13,7 @@ If you are eager to help and improve docker-sync
  - Help [here](https://github.com/EugenMayer/docker-sync/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) if you are a coder
  - Help [here](https://github.com/EugenMayer/docker-sync/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3A%22help%20wanted%22%20%20label%3A%22documentation%22%20) with the docs no matter what skill set you have
 
-Anything else under [http://docker-sync.io](http://docker-sync.io) or the [wiki](https://github.com/EugenMayer/docker-sync/wiki)
+Anything else under [http://docker-sync.io](http://docker-sync.io) or the [documentation](https://docker-sync.readthedocs.io/en/latest/index.html#)
 
 Main Contributors:
  - [ignatiusreza](https://github.com/ignatiusreza)

VERSION

@@ -1 +1 @@
-0.5.10
+0.5.11

docs/Makefile

@@ -0,0 +1,22 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+livehtml:
+	sphinx-autobuild -b html "$(SOURCEDIR)" "$(BUILDDIR)/html" $(SPHINXOPTS) $(O)
+
+.PHONY: help livehtml Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/README.md

@@ -0,0 +1,14 @@
+# Setting up docs generator locally
+
+To re-generate the docs on your local machine, simply follow the commands below.
+
+```shell
+# Install required dependencies
+pip install sphinx sphinx_rtd_theme
+
+# Spins up the worker that will watch and re-generate the docs when changed
+make livehtml
+```
+
+Navigate to `http://localhost:8000`. You'll see the latest docs there.
+The pages will re-generate and reload itself when a file is changed.

docs/advanced/how-it-works.rst

@@ -0,0 +1,73 @@
+************
+How it works
+************
+
+Architecture
+============
+
+On the host, a thor based ruby task is started, this starts:
+
+    - Every sync will start an own docker-container with a rsync/unison-daemon watching for connections.
+    - The data gets pre-synced on sync-start
+    - A fswatch cli-task gets setup, to run rsync/unison on each file-change in the source-folder you defined
+
+Done. No magic. But its roadrunner fast! And it has no pre-conditions on your actual stack.
+
+----
+
+.. _native_osx in depth:
+
+native_osx in depth
+===================
+
+Under The Hood
+--------------
+
+First, take a look at this diagram:
+
+.. image:: /_static/native_osx.png
+  :alt: DockerSync native_osx strategy overview
+
+There are some important keypoints to notice here:
+
+1. We use OSXFS to mount your local host folder into the sync-container.
+2. We do not mount this in the app-container directly, since this would lead to `infamously horrible performance`_.
+3. Instead of directly mounting ``/host_sync`` in the app-container we setup a **2-way-sync** inside the sync-container using Unison_. This ensures that the actual READ/WRITE performance on the ``/app_sync`` folder is native-speed fast.
+4. This makes all operations on ``/app_sync`` be asynchronous with ``/host_sync``, since writing and reading on ``/app_sync`` does not rely on any OSXFS operation directly, **but shortly delayed and asynchronous**.
+5. We mount ``/app_sync`` to your app_container - since this happens in hyperkit, it's a **Docker LINUX-based** native  mount, thus running at native-speed.
+6. Your application now runs like there was no sync at all.
+
+.. _infamously horrible performance: https://docs.docker.com/docker-for-mac/osxfs/#performance-issues-solutions-and-roadmap
+.. _Unison: http://www.cis.upenn.edu/~bcpierce/unison/
+
+FAQ
+---
+
+Why use OSXFS in the first place (instead of the ``unison`` strategy) to sync from the host to the sync-container
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are several reasons, one of the most important being the performance. Since MacOS/OSX has very bad filesystem events support on HFS/APFS, watching the file-system for changes using ``unox`` or ``fswatch`` was causing a heavy CPU load. This CPU load is very significant, even on modern high-end CPUs (like a i7 4770k / 3.5GHz).
+
+The second issue was dependencies. With native_osx you do not need to install anything on your host OS except the docker-sync gem. So no need to compile unox or install unison manually, deploy with brew and fail along the way - just keeping you system clean.
+
+Is this strategy absolutely bullet proof?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+No, it is not. But it has been pretty battle proven already - the main issue is https://github.com/EugenMayer/docker-sync/issues/410 - so sometimes OSXFS just stops triggering FS events in Hyperkit, thus in the sync-container. This leads to an issue with our sync, since the ``unison`` daemon inside the app-sync  container relies on those events to sync the changes (it does not have the ability to poll, which would be disastrous performance-wise, anyway).
+
+Advanced Monitoring for ``native_osx``
+--------------------------------------
+
+Background
+^^^^^^^^^^
+
+Monit_ is a utility which can be used to monitor the health of the ``unison`` process which runs in the container for the ``native_osx`` strategy. If it detects that ``unison`` is unhealthy, Monit automatically restarts ``unison``. This improves the stability of the ``native_osx`` container in cases where the ``unison`` process is misbehaving but does not necessarily crash. Currently, there is only one check for CPU usage implemented, but in the future more checks may be added, such as memory usage. It is currently turned off by default and can be turned on in the configuration:
+
+https://github.com/EugenMayer/docker-sync/blob/master/example/docker-sync.yml#L120-L126
+
+.. _Monit: https://mmonit.com/monit/
+
+Monitoring CPU usage
+^^^^^^^^^^^^^^^^^^^^
+
+One instance which ``unison`` has been seen to misbehave is when quickly creating and deleting a file while it is processing it. ``unison`` may hang, using a high amount of cpu time: https://github.com/EugenMayer/docker-sync/issues/497. ``monit`` detects this high cpu usage (>50%) and automatically restarts ``unison`` to recover it. By default this happens within 10 seconds, but the tolerance can be configured in case there are normal spikes in cpu usage during successful syncs.

docs/advanced/scripting.rst

@@ -0,0 +1,54 @@
+Scripting
+=========
+
+We use docker-sync as a library in our own docker-stack startup script. It starts the docker-compose stack using a Ruby gem `EugenMayer/docker-compose`_ all this wrapped into a thor task. So:
+
+ - Start docker-sync
+ - Start a docker-compose stack based on some arguments like --dev and load the specific docker-compose files for that using `xeger/docker-compose`_
+
+docker-sync-stack is actually an example already, just see here:
+
+1. You run the sync manager with run : https://github.com/EugenMayer/docker-sync/blob/master/tasks/stack/stack.thor#L37
+2. But you do not call .join_threads after that like her https://github.com/EugenMayer/docker-sync/blob/master/tasks/sync/sync.thor#L36
+3. Then you just continue doing what you want to script, in my case, i start a new blocking task - docker-compose. But you could do anything.
+
+.. _EugenMayer/docker-compose: https://github.com/EugenMayer/docker-compose
+.. _xeger/docker-compose: https://github.com/xeger/docker-compose
+
+
+Simple scripting example
+------------------------
+
+.. code-block:: ruby
+
+    require 'docker-sync/sync_manager'
+    require 'docker-sync/dependencies'
+    require 'docker-sync/config/project_config'
+
+    # load the project config
+    config = DockerSync::ProjectConfig.new(config_path: nil)
+    DockerSync::Dependencies.ensure_all!(config)
+    # now start the sync
+    @sync_manager = Docker_sync::SyncManager.new(:config_path => config_path)
+    @sync_manager.run() # do not call .join_threads now
+
+    #### your stuff here
+    @sync_manager.watch_stop()
+    system('my-bash-script.sh')
+    some_ruby_logic()
+    system('other-tasks.sh')
+    @sync_manager.sync()
+    @sync_manager.watch_start()
+    ### debootsrapping
+    begin
+      @sync_manager.join_threads
+    rescue SystemExit, Interrupt
+      say_status 'shutdown', 'Shutting down...', :blue
+
+      @sync_manager.stop
+    rescue Exception => e
+
+      puts "EXCEPTION: #{e.inspect}"
+      puts "MESSAGE: #{e.message}"
+
+    end

docs/advanced/sync-strategies.rst

@@ -0,0 +1,100 @@
+Sync strategies
+===============
+
+The sync strategies depend on the OS, so not all strategies are available on all operating system
+
+- OSX: native_osx, unison, rsync
+- Windows: unison
+- Linux: native_linux, unison
+
+----
+
+.. _strategies-native-osx:
+
+native_osx (OSX)
+----------------
+
+.. image:: /_static/native_osx.png
+  :alt: DockerSync native_osx strategy overview
+
+For advanced understanding, please read :ref:`native_osx in depth`.
+
+Native-OSX is a combination of two concepts, `OSXFS only`_ and Unison together. We use OSX to sync the file-system into a sync-container to /host_sync. In that sync container we sync from /host_sync to /app_sync using Unison. /app_sync is exposed as a named volume mount and consumed in the app. You ask yourself, why? Its fairly simple.
+
+By having this extra layer of unison on linux, we detach the actual write/read performance of your application from the actual OSXFS performance - running at native speed. Still, we using OSXFS, a about up to 1 second delayed, to synchronize changes both ways. So we have a 2-way sync.
+
+What is different to plain Unison you might ask. The first big issue with Unison is, how bad it performs under OSX due to the bad FS-Events of OSX, implemented in macfsevents and alikes. It uses a lot of CPU for watching files, it can lose some events and miss a sync - but also, it adds extra dependencies on the OSX hosts.
+
+All that is eliminated by native_osx - we use Unison in the Linux container, where it performs great due to inotify-event based watchers.
+
+Pros
+ - Far more reliable due to a low-level implementation of the sync using `OSXFS only`_
+ - Uses far less CPU to sync files from the host to the sync container - will handle a lot more files
+ - No daemon to run, control, restart and sync on the OSX host. Makes sleep/hibernate/reboot much more robust
+ - No dependencies on the OSX-Host at all
+ - A lot easier installation since we just need ``gem install docker-sync`` and that on runs under system ruby. Anything else is in containers
+ - It performs at native speed
+ - It will be much easier to support Windows this way
+
+Cons
+ - Initial start can take longer as on unison, since the first sync is more expensive. But this is one time only
+ - It works under Docker for Mac only - missing file system events under vbox/fusion. See `native_osx does not work with docker-machine vbox / fusion`_
+
+.. _OSXFS only: https://github.com/EugenMayer/docker-sync/issues/346
+.. _native_osx does not work with docker-machine vbox / fusion: https://github.com/EugenMayer/docker-sync/issues/346
+
+----
+
+unison (OSX/Windows/Linux)
+--------------------------
+
+This strategy has the biggest drive to become the new default player out of the current technologies. It seems to work very well with huge codebases too. It generally is build to handle 2 way sync, so syncs back changes from the container to the host.
+
+Pros
+ - Offers 2 way sync (please see unison-dualside why this is misleading here)
+ - Still very effective and works for huge projects
+ - Native speed in for the application
+
+Cons
+ - Can be unreliable with huge file counts (> 30.000) and bad hardware (events gets stuck)
+ - The daemon on OSX needs extra care for sleep/hibernate.
+ - Extra dependencies we need on OSX, in need to install unison and unox natively - brew dependencies
+
+**Initial startup delays with unison**
+
+On initial start, Unison sync needs to build a catalog of the files in the synced folder before sync begins. As a result, when syncing folders with large numbers of relatively large files (for example, 40k+ files occupying 4G of space) using unison, you may see a significant delay (even 20+ minutes) between the initial ``ok  Starting unison`` message and the ``ok  Synced`` message. This is not a bug. Performance in these situations can be improved by moving directories with a large number of files into a separate ``rsync`` strategy sync volume, and using unison only on directories where two-way sync is necessary.
+
+----
+
+rsync (OSX)
+-----------
+
+This strategy is probably the simplest one and probably the most robust one for now, while it has some limitations. rsync-syncs are known to be pretty fast, also working very efficient on huge codebases - no need to be scared of 20k files. rsync will easily rsync the changes ( diff ) very fast.
+
+Pros
+ - Fast re-syncing huge codebases - sync diffs (faster then unison? proof?)
+ - Well tested and known to be robust
+ - Best handling of user-permission handling ( mapped into proper users in the app container )
+
+Cons
+ - Implements a one way sync only, means only changes of your codebase on the host are transferred to the app-container. Changes of the app-container are not synced back at all
+ - Deleting files on the host does yet not delete them on the container, since we do not use --delete, see `#347`_
+
+Example: On the docker-sync-boilerplate_
+
+.. _#347: https://github.com/EugenMayer/docker-sync/issues/37
+.. _docker-sync-boilerplate: https://github.com/EugenMayer/docker-sync-boilerplate/tree/master/rsync
+
+----
+
+native_linux
+------------
+
+Native linux is actually no real implementation, it just wraps docker-sync around the plain native docker mounts which do work perfectly on linux. So there are no sync-containers, no strategies or what so ever. This strategy is mainly used just to have the whole team use docker-sync, even on linux, to have the same interface. If you use default sync_strategy in the docker-sync.yml, under Linux, native_linux is picked automatically
+
+----
+
+Sync Flags or Options
+---------------------
+
+You find the available options for each strategy in :doc:`../getting-started/configuration`.

docs/advanced/tips-and-tricks.rst

@@ -0,0 +1,71 @@
+***************
+Tips and tricks
+***************
+
+HTTP Proxy and DNS
+==================
+
+The HTTP Proxy and DNS used in dinghy_ is available as a standalone project dinghy-http-proxy_.  The proxy is based on jwilder's excellent
+nginx-proxy_ project, with modifications to make it more suitable for local development work. A DNS resolver is also added. By default it will resolve all ``*.docker`` domains to the Docker VM, but this can be changed.
+
+.. _dinghy: https://github.com/codekitchen/dinghy
+.. _dinghy-http-proxy: https://github.com/codekitchen/dinghy-http-proxy
+.. _nginx-proxy: https://github.com/jwilder/nginx-proxy
+
+SSH-Agent Forwarding
+====================
+
+If you need to access some private git repos or ssh servers, it could be useful to use have a ssh-agent accessible from your containers. `whilp/ssh-agent`_ helps you to do so easily.
+
+.. _whilp/ssh-agent: https://github.com/whilp/ssh-agent
+
+Running composer or other tools like if they were on the host
+=============================================================
+
+If you run composer and other tools directly in containers, you could use a combination of the autoenv_ zsh plugin and a simple wrapper script to run it easily directly from the host. In your project folder, create a ``.autoenv.zsh`` file with the name of your container:
+
+.. code-block:: shell
+
+    autostash COMPOSER_CONTAINER='project_fpm_1'
+
+then create a simple function in your ``.zshrc``:
+
+
+.. code-block:: shell
+
+    composer () {
+        if [ ! -z $COMPOSER_CONTAINER ]; then
+          docker exec -it ${COMPOSER_CONTAINER} /usr/local/bin/composer --working-dir=/src -vv "$@"
+        else
+          /usr/local/bin/composer "$@"
+        fi
+    }
+
+.. _autoenv: https://github.com/Tarrasch/zsh-autoenv
+
+Ignore files in your IDE
+========================
+
+It's a good idea to add the temporary sync files to your IDE's ignore list to prevent your IDE from indexing them all the time or showing up in search results. In case of unison and PHPStorm for example just go to Preferences -> File Types -> Ignore files and folders and add ``.unison`` to the pattern.
+
+Don't sync everything
+=====================
+
+You should only sync files that you really need on both the host and client side. You will see that the sync performance will improve drastically when you ignore unnecessary files. How and which files to ignore depends on your syncing strategy (rsync/unison/...) and your project.
+
+Example for a PHP Symfony project using unison:
+
+.. code-block:: yaml
+
+    # docker-sync.yml
+    syncs:
+      appcode-unison-sync:
+        ...
+        sync_args:
+            - "-ignore='Path .idea'"          # no need to send PHPStorm config to container
+            - "-ignore='Path .git'"           # ignore the main .git repo
+            - "-ignore='BelowPath .git'"      # also ignore .git repos in subfolders such as in composer vendor dirs
+            - "-ignore='Path var/cache/*'"    # don't share the cache
+            - "-ignore='Path var/sessions/*'" # we don't need the sessions locally
+            - "-ignore='Path node_modules/*'" # remove this if you need code completion
+            # - "-ignore='Path vendor/*'"     # we could ignore the composer vendor folder, but then you won't have code completion in your IDE