Docker file configuration. #200
Conversation
jwils
requested review from
myronmarston and
BrianSigafoos-SQ
as code owners
There was a problem hiding this comment.
Thanks for getting this off the ground @jwils!
Unfortunately, I keep getting no space left on device errors:
~/Development/elasticgraph/ [joshuaw/onestep/dockerfile] VERSION=8.16.1 ENV=local docker compose up
[+] Running 2/3
[+] Building 46.8s (28/30) docker:desktop-linux
✔ Service kibana Built 1.2s
=> [kibana internal] load build definition from UI-Dockerfile 0.0s
[+] Building 73.5s (30/30) FINISHED docker:desktop-linux
=> [kibana internal] load build definition from UI-Dockerfile 0.0s
=> => transferring dockerfile: 82B 0.0s
=> [elasticsearch internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 148B 0.0s
=> [kibana internal] load metadata for docker.io/library/kibana:8.16.1 1.0s
=> [elasticsearch internal] load metadata for docker.io/library/elasticsearch:8.16.1 1.1s
=> [elasticsearch auth] library/elasticsearch:pull token for registry-1.docker.io 0.0s
=> [kibana auth] library/kibana:pull token for registry-1.docker.io 0.0s
=> [kibana internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [elasticsearch internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [kibana 1/1] FROM docker.io/library/kibana:8.16.1@sha256:e18c1f6d92e819b1c577a1af9a02bfcae6e8b63596368eec3b40e9ad98fa3caa 0.0s
=> [elasticsearch 1/2] FROM docker.io/library/elasticsearch:8.16.1@sha256:e5ee5f8dacbf18fa3ab59a098cc7d4d69f73e61637eb45f1c029e74b1cb200a1 0.0s
=> CACHED [elasticsearch 2/2] RUN bin/elasticsearch-plugin install mapper-size analysis-icu 0.0s
=> [kibana] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:c58110f9cc41fda2c8ebc14a648ce7db689c16f0b425f1bd6caac7cc4e0231ac 0.0s
=> => naming to docker.io/library/elasticgraph-kibana 0.0s
=> [elasticsearch] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:8abb8047e2fca67b01d5848cc325875153647f95e11166049a3d3feba0a7d5e5 0.0s
=> => naming to docker.io/library/elasticgraph-elasticsearch 0.0s
=> [kibana] resolving provenance for metadata file 0.0s
=> [elasticsearch] resolving provenance for metadata file 0.0s
=> [elasticgraph internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.20kB 0.0s
=> [elasticgraph internal] load metadata for docker.io/library/ruby:3.4 0.7s
=> [elasticgraph auth] library/ruby:pull token for registry-1.docker.io 0.0s
=> [elasticgraph internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [elasticgraph internal] load build context 7.1s
=> => transferring context: 12.21MB 6.7s
=> [elasticgraph stage-0 1/7] FROM docker.io/library/ruby:3.4@sha256:b6c3947ee95eb6bea5e331ac288beb2a48b98bb30d15353b6c5ffe2482828572 0.0s
=> CACHED [elasticgraph stage-0 3/7] RUN mkdir -p ~/.gnupg && chmod 700 ~/.gnupg && apt-get update -qq && apt-get install -y --no-install-recommends build-essential git nodejs && apt-get clean && rm -rf /var/lib/apt/lists/* && mkdir -p log tmp 0.0s
=> CACHED [elasticgraph stage-0 2/7] WORKDIR /app 0.0s
=> [elasticgraph stage-0 4/7] COPY . . 11.3s
=> [elasticgraph stage-0 5/7] RUN find . ! -name "Gemfile" ! -name "*.gemspec" -mindepth 2 -maxdepth 2 -print | xargs rm -rf 5.7s
=> [elasticgraph stage-0 6/7] RUN find . ! -name "Gemfile*" ! -name "*.gemspec" -maxdepth 1 -type f | xargs rm 0.1s
=> [elasticgraph stage-0 7/7] COPY elasticgraph-support/lib/elastic_graph/version.rb ./elasticgraph-support/lib/elastic_graph/version.rb 0.0s
=> [elasticgraph stage-1 3/5] COPY --from=0 /app . 0.0s
=> [elasticgraph stage-1 4/5] RUN bundle install 39.8s
[+] Running 2/3ticgraph stage-1 5/5] COPY . . 6.9s
✔ Service kibana Built 1.2s
✔ Service elasticsearch Built 1.2s
⠼ Service elasticgraph Building 72.4s
failed to solve: ResourceExhausted: failed to copy files: userspace copy failed: write /app/tmp/coverage/index.html: no space left on device
I've tried deleting all the volumes/images/containers I have and also I've restarted docker desktop...and I still get it. Did you run into that? If so please share how you solved it!
Dockerfile
Outdated
|
|
||
|
|
||
| # Set the default command to run the application | ||
| CMD ["bundle", "exec", "rake", "boot_in_container[9393,--host=0.0.0.0,true]"] |
There was a problem hiding this comment.
This boots the development/test schema of the ElasticGraph codebase itself. The test schema used in EG isn't a good candidate for using as a demonstration of what EG can do, because it's got all sorts of intentional weird edge cases built into it so we can exercise them in our tests suite.
Instead, I'd like to use the artists schema which is featured at https://block.github.io/elasticgraph/query-api/ and is also what you get when you do gem exec elasticgraph new. In fact I think we can use the new elasticgraph CLI for this so that this basically just bootstraps and runs a project just like a user will. Here's what I'm thinking:
- Let's add a new
elasticgraph democommand which:- Runs
elasticgraph newand bootstraps in a temp directory. cds into that directory- Runs
rake boot_locallyin that directory in order to boot EG - With all that done,
gem exec elasticgraph demowill be a nice one-liner for folks who have a ruby dev environment.
- Runs
- The
Dockerfilehere can then be simplified to just rungem exec elasticgraph demorather than needing to copy over the entire source tree, etc.
Thoughts?
| @@ -467,9 +466,13 @@ def define_other_tasks | |||
| desc "Boots ElasticGraph locally from scratch: boots #{datastore_to_boot}, configures it, indexes fake data, and boots GraphiQL" | |||
| task :boot_locally, [:port, :rackup_args, :no_open] => ["#{datastore_to_boot.downcase}:local:daemon", *index_fake_data_tasks, "boot_graphiql"] | |||
|
|
|||
| desc "Boots ElasticGraph locally from and connects to #{datastore_to_boot} cluster: configures it, indexes fake data, and boots GraphiQL" | |||
| task :boot_in_container, [:port, :rackup_args, :no_open] => [*index_fake_data_tasks, "boot_graphiql"] | |||
There was a problem hiding this comment.
It's a little weird to expose a rake task which is only for use when running the "demonstrate EG" one-liner, and isn't itself useful when working in an EG project. (And also, defining a task here defines it for ALL EG projects!).
Can we instead improve boot_locally to tolerate the case where the datastore is already running? Maybe the elasticsearch:local:daemon task can be a no-op if the datastore is already running. Or if need be we can use an ENV var to enable that behavior or something...
There was a problem hiding this comment.
Curious what you think of reworking local to be able to run everything i a container like this? It makes a nice easy way to have multiple version of all the components. I saw the comment about elasticsearch nodes accidentally forming a cluster. We can make that impossible if they don't share a network.
There was a problem hiding this comment.
Curious what you think of reworking local to be able to run everything i a container like this?
I'm open to it as an option, particularly if it simplifies things. However, I wouldn't want to change things to require that ElasticGraph projects run the Ruby bits in a docker container...and I have a hard time thinking of situations where running the Ruby pieces in a docker container makes for a better development experience, outside of this one use case (offering a way to boot an example ElasticGraph project locally for people checking out ElasticGraph who may not have a working Ruby development environment).
Maybe this is just my lack of experience with docker, but while Docker's great for dependencies written in other languages that need to run as a separate process (e.g. Elasticsearch, OpenSearch, DynamoDB, LocalStack, etc), I've never used docker to run a project's "main" language environment--e.g. in a Ruby project I run Ruby directly on my dev machine, and in a java/kotlin project I run the JVM outside docker directly on my dev machine. Running the main language environment inside a docker container while developing feels like it would have lots of downsides to me--e.g. it'd be more difficult to attach a debugger, IDE features might work differently (or not work at all), start up time to run a test or some other short-lived task would be greater, etc.
| - elasticsearch:/usr/share/elasticsearch/data | ||
| ports: | ||
| - ${PORT:-9200}:9200 | ||
| kibana: |
There was a problem hiding this comment.
We don't actually need kibana for this.
There was a problem hiding this comment.
I figured it doesn't hurt to give people access to kibana, but I can remove it. Looking to figure out how to have conditional containers run so we can choose elasticsearch vs opensearch if I do I'll make kibana optional.
There was a problem hiding this comment.
This duplicates a bunch of https://github.com/block/elasticgraph/blob/main/elasticgraph-local/lib/elastic_graph/local/elasticsearch/docker-compose.yaml. Can we have this reference/load that some how?
There was a problem hiding this comment.
There may be able to simplify both but the depends on logic only works if both are in the same compose file and it's a very nice way to make sure elasticsearch is running before we bootstrap elasticgraph.
| healthcheck: | ||
| interval: 10s | ||
| retries: 80 | ||
| test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:9200/ |
There was a problem hiding this comment.
Interesting, this isn't something we have in https://github.com/block/elasticgraph/blob/main/elasticgraph-local/lib/elastic_graph/local/elasticsearch/docker-compose.yaml.
I wonder if we should add it...
There was a problem hiding this comment.
Probably a good idea to add, but the value is mostly that this is used to ensure elasticsearch is running before we startup elasticgraph.
jwils
force-pushed
the
joshuaw/onestep/dockerfile
branch
from
92897ba to
e156d6d
Compare
There was a problem hiding this comment.
There may be able to simplify both but the depends on logic only works if both are in the same compose file and it's a very nice way to make sure elasticsearch is running before we bootstrap elasticgraph.
| healthcheck: | ||
| interval: 10s | ||
| retries: 80 | ||
| test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:9200/ |
There was a problem hiding this comment.
Probably a good idea to add, but the value is mostly that this is used to ensure elasticsearch is running before we startup elasticgraph.
| - elasticsearch:/usr/share/elasticsearch/data | ||
| ports: | ||
| - ${PORT:-9200}:9200 | ||
| kibana: |
There was a problem hiding this comment.
I figured it doesn't hurt to give people access to kibana, but I can remove it. Looking to figure out how to have conditional containers run so we can choose elasticsearch vs opensearch if I do I'll make kibana optional.
| @@ -467,9 +466,13 @@ def define_other_tasks | |||
| desc "Boots ElasticGraph locally from scratch: boots #{datastore_to_boot}, configures it, indexes fake data, and boots GraphiQL" | |||
| task :boot_locally, [:port, :rackup_args, :no_open] => ["#{datastore_to_boot.downcase}:local:daemon", *index_fake_data_tasks, "boot_graphiql"] | |||
|
|
|||
| desc "Boots ElasticGraph locally from and connects to #{datastore_to_boot} cluster: configures it, indexes fake data, and boots GraphiQL" | |||
| task :boot_in_container, [:port, :rackup_args, :no_open] => [*index_fake_data_tasks, "boot_graphiql"] | |||
There was a problem hiding this comment.
Curious what you think of reworking local to be able to run everything i a container like this? It makes a nice easy way to have multiple version of all the components. I saw the comment about elasticsearch nodes accidentally forming a cluster. We can make that impossible if they don't share a network.
jwils
force-pushed
the
joshuaw/onestep/dockerfile
branch
from
e156d6d to
3a24ebb
Compare
jwils
force-pushed
the
joshuaw/onestep_docker
branch
2 times, most recently
from
79cd252 to
c89298f
Compare
This PR still needs some cleanup as it breaks the local rake tasks and overrides the development config but it's a proof of concept with a working docker compose file.
Feel free to try it out. No dependency on local systems ruby.
From the root directory
VERSION=8.16.1 ENV=local docker compose up