nodeodm consumes /var/lib/docker

[plutext]

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Chrome 69 on Linux

What is the problem?

nodeodm consumes /var/lib/docker

In my overlay2 dir, one of the entries is 20GB:

# du -h -s *
2.1G    309660212bf55b4d7870c6fce6dae824da7da95f36703e11bab46373bd0e782c
3.1G    6709e473af643be80e544dbffd4682b6770fc3a37e2e13369c19f5668c1c8a92
20G     a15dd8544cc35ae2a2df099de8ab9fa5d1319facb417ff7b253c84ba04330b3f
2.9G    b6052d7ad2f21cef3d4c9edafc6c1dbfded960b3871c629a0a8b57762cac73da
2.9G    cbe268cdec9dfad29fa4fafb1fd93691fb85d32fd6da0c1db43783191ae0e4ff

This was after processing 150 images.

Now to identify the container causing this:

# docker container ls
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                     NAMES
95ae357c37c1        opendronemap/webodm_webapp   "/bin/bash -c 'chmod…"   24 hours ago        Up 24 hours         0.0.0.0:8000->8000/tcp    webapp
bed43950b21b        opendronemap/webodm_webapp   "/bin/bash -c '/webo…"   24 hours ago        Up 24 hours                                   worker
3f21e4ef1007        redis                        "docker-entrypoint.s…"   24 hours ago        Up 24 hours         6379/tcp                  broker
839eb5925a60        opendronemap/nodeodm         "/usr/bin/nodejs /va…"   24 hours ago        Up 24 hours         0.0.0.0:32772->3000/tcp   webodm_node-odm_1
d964227ea525        opendronemap/webodm_db       "docker-entrypoint.s…"   24 hours ago        Up 24 hours         0.0.0.0:32773->5432/tcp   db

Using docker inspect, we can see nodeodm and webodm_db are both implicated, but that nodeodm looks to be to blame:

# docker inspect 839eb5925a60 | grep a15
                "LowerDir": "/var/lib/docker/overlay2/a15dd8544cc35ae2a2df099de8ab9fa5d1319facb417ff7b253c84ba04330b3f-init/diff:/var/lib/docker/overlay2/454d4695f1d09b3d60746f808dde441dcee8310bd1ae859e935c6beb32bac42a/diff:/var/lib/docker/overlay2/745ae95e1888191bcfd0a72c8fdf849f28b19e953a4d3e8ce85b456eb8d04366/diff:/var/lib/docker/overlay2/aa5b34d47e3c27b32db71aa1119b522f9f908320d39dca9bd44c024975ca92d1/diff:/var/lib/docker/overlay2/001038e4e7bc630a01a793413a9c48a64eaa2829e7db32bb2d298635f764db6f/diff:/var/lib/docker/overlay2/3b72c51627bc5cd4fc00d455d510e30d0e1030c57d941c6f2ca3678718a09b83/diff:/var/lib/docker/overlay2/6709e473af643be80e544dbffd4682b6770fc3a37e2e13369c19f5668c1c8a92/diff:/var/lib/docker/overlay2/c11139095057467be351e9255cf01feb8078435d9fc73bf4f8d95f7a73c83326/diff:/var/lib/docker/overlay2/1178f10eb61e00078210920bfcab5ddf3358d7b6774d45ac373152853d9b7e37/diff:/var/lib/docker/overlay2/54e080e475cfd3db5258efe15fbcdbb53e50ebb62e60944ab4b9197ccb07ec21/diff:/var/lib/docker/overlay2/786d3ee0fc9d8742128b5e7955f0901d9de6931730702817f7cd0c9bba883c39/diff:/var/lib/docker/overlay2/d5f2296f6bd013e521ff2e8d70166a145d5a5ff0dc7a16bf0b8c726ec5163507/diff:/var/lib/docker/overlay2/5fad82d1cff366a4fbc205c97be3acd41245bb84afe60a54628e2baf602bceb7/diff:/var/lib/docker/overlay2/e7b001b7a625f19c3f9ef4ff3e95f7706aa04e8ca6bdce4a4f31562693d3ab3a/diff:/var/lib/docker/overlay2/7416d07b818eb97d381ed2f873dd8010314a075fa03eb81ed8935058f3481c39/diff:/var/lib/docker/overlay2/58a3f208ed84e5d5c0314699583e911f0e25fc829774b16b63224abef6be7b97/diff:/var/lib/docker/overlay2/42a6f85af2bf507a0fe2a9e820e2b46ab4d86782c05c58eced8e9973e3877558/diff",
                "MergedDir": "/var/lib/docker/overlay2/a15dd8544cc35ae2a2df099de8ab9fa5d1319facb417ff7b253c84ba04330b3f/merged",
                "UpperDir": "/var/lib/docker/overlay2/a15dd8544cc35ae2a2df099de8ab9fa5d1319facb417ff7b253c84ba04330b3f/diff",
                "WorkDir": "/var/lib/docker/overlay2/a15dd8544cc35ae2a2df099de8ab9fa5d1319facb417ff7b253c84ba04330b3f/work"

# docker inspect d964227ea525 | grep a15
                "LowerDir": "/var/lib/docker/overlay2/e16267d4b2747a41ea6a0e54dff238e542afd35be69e07473851585fdef1112c-init/diff:/var/lib/docker/overlay2/099d7d6d1f062a10d487195dd2fa9630cb2437c37a4c288e85f4c840ef73b279/diff:/var/lib/docker/overlay2/edce50294eea2605d0413c9f1a9efebca0f2f4d12887cfa404d36972dc15ca77/diff:/var/lib/docker/overlay2/56c874258c831f3b2144d4d3f11269c08aff2a29342ccd665df783e3ba267f4a/diff:/var/lib/docker/overlay2/f4f1392c5016fc1f1c94df24d069bebdddcc5ac69dc61d40ebecca8d32924c66/diff:/var/lib/docker/overlay2/40595c7ba87289f601778eab49f93286de4ef270020430b66be89486bab90338/diff:/var/lib/docker/overlay2/0b1562fb81b1572eb031446820f2585910dd477771436c388dc465fceff98109/diff:/var/lib/docker/overlay2/a4d4d49baaca18a2976148dd683b90cf1824773367169fea60546327ed6f5067/diff:/var/lib/docker/overlay2/64a88e9c6d785be18a8ddd72f85d3d85a16efaa153d7ab1e095a6e67acd62dfb/diff:/var/lib/docker/overlay2/8e112638007091f780d57ec725a526c0736e3bb996dd7691df0e5d3cc600c82d/diff:/var/lib/docker/overlay2/072eca82e9bdff13517c682fb2387fffe81d515f05b6090ef478d51424fa3dc4/diff:/var/lib/docker/overlay2/d6d7bc6a9e2d7e3d68cd1f20d9fcbf79e79b98ae6b5146a496c4136dcaa00b42/diff:/var/lib/docker/overlay2/e2d380ba756c59c172f25d9a8a0a7f0209fdf89bb0c7a669afb3d6fbef0ef4d4/diff:/var/lib/docker/overlay2/080bc0dcd573a6bfbf53dcba4c087aad09d9ccc869edf7c2e7accc826d69fa30/diff:/var/lib/docker/overlay2/3a5b5199d01194d99b3c58c58864739c21e2e2b1610f2f39071081689944a4c8/diff:/var/lib/docker/overlay2/261688cf2db0fedc71d1f9c658ad7edeb52d3d8c0c01a8660e647686eb7aa2ad/diff:/var/lib/docker/overlay2/fdac587f4330961520bec46121bbd32a21c31c469ba27d7b50b763f8afe463b2/diff",

See also #776

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? Please include multiple screenshots of the problem! If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

Process some images and observe the resource usage of nodeodm. This issue shows how to do that if you are using overlay2

[pierotofy]

A docker system prune might help free disk space. If one is worried about space, the correct fix here would be to volume mount /var/www/data from the nodeodm container onto a path on the host. I don't think this is something that WebODM should do, however, because there's no good cross-platform way to do it without possibly hindering performance (Windows).

[pierotofy]

I'll leave #776 open in regard to possibly offering a flag to specify a mount directory explicitly (like --media-dir and --db-dir).