Improve configuration and service management (ory#1314)

This patch significantly refactors internal configuration and service management with the goal of making configuration changes possible without service restarts. This patch prepares the possibility to configure ORY Hydra from a remote source (etcd, consul) and watch for changes. This patch also introduces the possibility to configure ORY Hydra from a configuration file on top of environment variables.

The following issues have been fixed as well:

- Key rotation of the system secret is now much simpler (closes ory#1316)
- Database connectivity parameters such as max_conns no longer cause issues with older postgres versions (closes ory#1327)
- Non-existing routes now return a JSON error instead of `text/plain` when `application/json` was requested (clsoes ory#1244)
- We now push tags latest, X, X.Y, X.Y.Z to docker hub (closes ory#1289)
- The quickstart guide no longer builds the source code but instead pulls latest tag (closes ory#1309)
- We moved to goreleaser and godownloader for release management (closes ory#1107)
- The quickstart Docker Compose files are now reusable (closes ory#1196)

The following issues are also resolved:

- Closes ory#1121 

Signed-off-by: aeneasr <[email protected]>

.circleci/config.yml

@@ -135,60 +135,6 @@ jobs:
 #    See https://github.com/ory/hydra/issues/1179
 #    - run: OAUTH2_ACCESS_TOKEN_STRATEGY=jwt DATABASE_URL=memory ./scripts/test-e2e.sh
 
-# This test is really useless because there are always changes (usually timestamps in the generated code)
-#  generators:
-#    docker:
-#      - image: circleci/golang:1.12
-#    working_directory: /go/src/github.com/ory/hydra
-#    steps:
-#      - checkout
-#      - run: sudo apt-get update -y
-#      - run: sudo apt-get install -y default-jdk
-#      - run: make init
-#      - run:
-#          name: Enable go1.12 modules
-#          command: |
-#            echo 'export GO111MODULE=on' >> $BASH_ENV
-#            source $BASH_ENV
-#      - run: curl -L https://git.io/vp6lP | sh
-#      - run: mv ./bin/* $GOPATH/bin
-#      - run: go mod download
-#      - run: go mod vendor
-#      - run: GO111MODULE=off make gen
-#      - run: git add -A
-#      - run: git diff --cached --exit-code
-
-  build-docker:
-    docker:
-      - image: library/docker:17.10
-    steps:
-      - checkout
-      - setup_remote_docker:
-          version: 17.10.0-ce
-      - run: docker build -f Dockerfile -t hydra-test .
-      - run: docker build -f Dockerfile-alpine -t hydra-test-alpine .
-      - run: docker run hydra-test help
-
-  release-docker:
-    docker:
-      - image: circleci/golang:1.12
-    working_directory: /go/src/github.com/ory/hydra
-    steps:
-      - run:
-          name: Enable go1.12 modules
-          command: |
-            echo 'export GO111MODULE=on' >> $BASH_ENV
-            source $BASH_ENV
-      - checkout
-      - setup_remote_docker:
-          version: 17.10.0-ce
-      # Build and push docker image
-      - run: docker build --build-arg git_tag=$(git describe --tags) --build-arg git_commit=$(git rev-parse HEAD) -f Dockerfile -t oryd/hydra:$(echo $CIRCLE_TAG | tr '+' '_') .
-      - run: docker build --build-arg git_tag=$(git describe --tags) --build-arg git_commit=$(git rev-parse HEAD) -f Dockerfile-alpine -t oryd/hydra:$(echo $CIRCLE_TAG | tr '+' '_')-alpine .
-      - run: docker login --username "$DOCKER_USERNAME" --password "$DOCKER_PASSWORD"
-      - run: docker push oryd/hydra:$(echo $CIRCLE_TAG | tr '+' '_')
-      - run: docker push oryd/hydra:$(echo $CIRCLE_TAG | tr '+' '_')-alpine
-
   release-npm:
     docker:
       - image: circleci/node:8.9.3
@@ -200,22 +146,6 @@ jobs:
           npm version -f --no-git-tag-version $CIRCLE_TAG
       - run: npm publish --access public
 
-  release-binaries:
-    docker:
-      - image: circleci/golang:1.12
-    working_directory: /go/src/github.com/ory/hydra
-    steps:
-      - run:
-          name: Enable go1.12 modules
-          command: |
-            echo 'export GO111MODULE=on' >> $BASH_ENV
-            source $BASH_ENV
-      - checkout
-      - run: go get -u github.com/mitchellh/gox github.com/tcnksm/ghr
-      - run: |
-          gox -parallel=2 -ldflags "-X github.com/ory/hydra/cmd.Version=`git describe --tags` -X github.com/ory/hydra/cmd.BuildTime=`TZ=UTC date -u '+%Y-%m-%dT%H:%M:%SZ'` -X github.com/ory/hydra/cmd.GitHash=`git rev-parse HEAD`" -output "dist/{{.Dir}}-{{.OS}}-{{.Arch}}";
-      - run: ghr -t $GITHUB_TOKEN -u $CIRCLE_PROJECT_USERNAME -r $CIRCLE_PROJECT_REPONAME --replace `git describe --tags` dist/
-
   release-changelog:
     docker:
       - image: circleci/ruby:2.4-node
@@ -262,12 +192,28 @@ jobs:
       - run: go get -u github.com/rakyll/hey
       - run: go install .
       - run: ./scripts/run-bench.sh
-      - run: ./scripts/run-appendix.sh
+      - run: ./scripts/run-configuration.sh
       - run: "git clone https://arekkas:[email protected]/ory/docs.git ../docs"
       - run: "cp BENCHMARKS.md ../docs/docs/performance/hydra.md"
-      - run: "cp appendix.md ../docs/docs/hydra/appendix.md"
+      - run: "cp configuration.md ../docs/docs/hydra/configuration.md"
       - run: "(cd ../docs && git add -A && git commit -a -m \"Updates ORY Hydra autogenerated docs\" && git push origin) || exit 0"
 
+  release:
+    docker:
+      - image: circleci/golang:1.12
+    working_directory: /go/src/github.com/ory/hydra
+    steps:
+      - run:
+          name: Enable go1.12 modules
+          command: |
+            echo 'export GO111MODULE=on' >> $BASH_ENV
+            source $BASH_ENV
+      - checkout
+      - setup_remote_docker
+      - run: docker login --username "$DOCKER_USERNAME" --password "$DOCKER_PASSWORD"
+      - run: cp ./.releaser/LICENSE.txt ./LICENSE.txt
+      - run: curl -sL https://git.io/goreleaser | bash
+
 workflows:
   version: 2
   "test, build, push, and deploy":
@@ -311,50 +257,30 @@ workflows:
           filters:
             branches:
               only: master
-#      - generators:
-#          filters:
-#            tags:
-#              only: /.*/
-      - build-docker:
+      - release:
           requires:
             - test
-#            - generators
             - format
             - test-e2e-opaque
             - test-e2e-plugin
             - test-e2e-jwt
           filters:
             tags:
               only: /.*/
-      - release-binaries:
-          requires:
-            - build-docker
-          filters:
-            tags:
-              only: /.*/
-            branches:
-              ignore: /.*/
-      - release-docker:
-          requires:
-            - build-docker
-          filters:
-            tags:
-              only: /.*/
             branches:
               ignore: /.*/
       - release-npm:
           requires:
-            - build-docker
+            - release
           filters:
             tags:
               only: /.*/
             branches:
               ignore: /.*/
       - release-changelog:
           requires:
-            - release-docker
+            - release
             - release-npm
-            - release-binaries
           filters:
             tags:
               only: /.*/

.gitignore

@@ -17,3 +17,5 @@ plugin-*.so
 hydra-docker-bin
 cookies.txt
 vendor/
+LICENSE.txt
+hydra

.goreleaser.yml

@@ -0,0 +1,73 @@
+# This is an example goreleaser.yaml file with some sane defaults.
+# Make sure to check the documentation at http://goreleaser.com
+project_name: hydra
+
+before:
+  hooks:
+    - go mod download
+    - go mod tidy
+builds:
+  -
+    binary: hydra
+    env:
+      - CGO_ENABLED=0
+    goarch:
+      - amd64
+      - 386
+      - arm
+      - arm64
+    goarm:
+      - 5
+      - 6
+      - 7
+    goos:
+      - freebsd
+      - linux
+      #    - plan9
+      #    - solaris
+      #    - netbsd
+      #    - openbsd
+      - windows
+      - darwin
+archive:
+  replacements:
+    darwin: macOS
+    linux: Linux
+    windows: Windows
+    386: 32-bit
+    amd64: 64-bit
+  format_overrides:
+    - goos: windows
+      format: zip
+  files:
+    -  LICENSE.txt
+snapshot:
+  name_template: "{{ .Tag }}-next"
+changelog:
+  sort: asc
+brew:
+  github:
+    owner: ory
+    name: homebrew-ory-hydra
+  homepage:  https://www.ory.sh
+  commit_author:
+    name: ory
+    email: [email protected]
+
+scoop:
+  bucket:
+    owner: ory
+    name: scoop-ory-hydra
+  homepage:  https://www.ory.sh
+  commit_author:
+    name: ory
+    email: [email protected]
+
+dockers:
+  - image_templates:
+      - "oryd/hydra:{{ .Tag }}"
+      - "oryd/hydra:v{{ .Major }}"
+      - "oryd/hydra:v{{ .Major }}.{{ .Minor }}"
+      - "oryd/hydra:latest"
+    extra_files:
+      - .releaser/LICENSE.txt

.releaser/LICENSE.txt

@@ -0,0 +1,84 @@
+ORY Hydra
+Copyright (c) 2019 ORY GmbH
+
+*** END USER LICENSE AGREEMENT ***
+
+IMPORTANT: PLEASE READ THIS LICENSE CAREFULLY BEFORE USING THIS SOFTWARE.
+
+1. LICENSE
+
+By receiving, opening the file package, and/or using ORY Hydra ("Software") containing this software, you agree that
+this End User User License Agreement(EULA) is a legally binding and valid contract and agree to be bound by it.
+You agree to abide by the intellectual property laws and all of the terms and conditions of this Agreement.
+
+Unless you have a different license agreement signed by ORY GmbH your use of ORY Hydra indicates
+your acceptance of this license agreement and warranty.
+
+Subject to the terms of this Agreement, ORY GmbH grants to you a limited, non-exclusive, non-transferable
+license, without right to sub-license, to use ORY Hydra in accordance with this Agreement and any other written
+agreement with ORY GmbH. ORY GmbH does not transfer the title of ORY Hydra to you; the license granted to you is not a
+sale. This agreement is a binding legal agreement between ORY GmbH and the purchasers or users of ORY Hydra.
+
+If you do not agree to be bound by this agreement, remove ORY Hydra from your computer now and, if applicable,
+promptly return to ORY GmbH by mail any copies of ORY Hydra and related documentation and packaging in your possession.
+
+2. DISTRIBUTION
+
+ORY Hydra and the license herein granted shall not be copied, shared, distributed, re-sold, offered for re-sale,
+transferred or sub-licensed in whole or in part except that you may make one copy for archive purposes only. For
+information about redistribution of ORY Hydra contact ORY GmbH.
+
+3. USER AGREEMENT
+
+3.1 Use
+
+Your license to use ORY Hydra is limited to the number of licenses purchased by you. You shall not allow others to use,
+copy or evaluate copies of ORY Hydra.
+
+3.2 Use Restrictions
+
+You shall use ORY Hydra in compliance with all applicable laws and not for any unlawful purpose. Without limiting the
+foregoing, use, display or distribution of ORY Hydra together with material that is pornographic, racist, vulgar,
+obscene, defamatory, libelous, abusive, promoting hatred, discriminating or displaying prejudice based on religion,
+ethnic heritage, race, sexual orientation or age is strictly prohibited.
+
+Each licensed copy of ORY Hydra may be used on one single computer location by one user. Use of ORY Hydra means that you have loaded, installed, or run ORY Hydra on a computer or similar device. If you install ORY Hydra onto a multi-user platform, server or network, each and every individual user of ORY Hydra must be licensed separately.
+
+You may make one copy of ORY Hydra for backup purposes, providing you only have one copy installed on one computer being used by one person. Other users may not use your copy of ORY Hydra . The assignment, sublicense, networking, sale, or distribution of copies of ORY Hydra are strictly forbidden without the prior written consent of ORY GmbH. It is a violation of this agreement to assign, sell, share, loan, rent, lease, borrow, network or transfer the use of ORY Hydra. If any person other than yourself uses ORY Hydra registered in your name, regardless of whether it is at the same time or different times, then this agreement is being violated and you are responsible for that violation!
+
+3.3 Copyright Restriction
+
+This Software contains copyrighted material, trade secrets and other proprietary material. You shall not, and shall not attempt to, modify, reverse engineer, disassemble or decompile ORY Hydra. Nor can you create any derivative works or other works that are based upon or derived from ORY Hydra in whole or in part.
+
+ORY GmbH's name, logo and graphics file that represents ORY Hydra shall not be used in any way to promote products developed with ORY Hydra . ORY GmbH retains sole and exclusive ownership of all right, title and interest in and to ORY Hydra and all Intellectual Property rights relating thereto.
+
+Copyright law and international copyright treaty provisions protect all parts of ORY Hydra, products and services. No program, code, part, image, audio sample, or text may be copied or used in any way by the user except as intended within the bounds of the single user program. All rights not expressly granted hereunder are reserved for ORY GmbH.
+
+3.4 Limitation of Responsibility
+
+You will indemnify, hold harmless, and defend ORY GmbH , its employees, agents and distributors against any and all claims, proceedings, demand and costs resulting from or in any way connected with your use of ORY GmbH's Software.
+
+In no event (including, without limitation, in the event of negligence) will ORY GmbH , its employees, agents or distributors be liable for any consequential, incidental, indirect, special or punitive damages whatsoever (including, without limitation, damages for loss of profits, loss of use, business interruption, loss of information or data, or pecuniary loss), in connection with or arising out of or related to this Agreement, ORY Hydra or the use or inability to use ORY Hydra or the furnishing, performance or use of any other matters hereunder whether based upon contract, tort or any other theory including negligence.
+
+ORY GmbH's entire liability, without exception, is limited to the customers' reimbursement of the purchase price of the Software (maximum being the lesser of the amount paid by you and the suggested retail price as listed by ORY GmbH ) in exchange for the return of the product, all copies, registration papers and manuals, and all materials that constitute a transfer of license from the customer back to ORY GmbH.
+
+3.5 Warranties
+
+Except as expressly stated in writing, ORY GmbH makes no representation or warranties in respect of this Software and expressly excludes all other warranties, expressed or implied, oral or written, including, without limitation, any implied warranties of merchantable quality or fitness for a particular purpose.
+
+3.6 Governing Law
+
+This Agreement shall be governed by the law of the Germany applicable therein. You hereby irrevocably attorn and submit to the non-exclusive jurisdiction of the courts of Germany therefrom. If any provision shall be considered unlawful, void or otherwise unenforceable, then that provision shall be deemed severable from this License and not affect the validity and enforceability of any other provisions.
+
+3.7 Termination
+
+Any failure to comply with the terms and conditions of this Agreement will result in automatic and immediate termination of this license. Upon termination of this license granted herein for any reason, you agree to immediately cease use of ORY Hydra and destroy all copies of ORY Hydra supplied under this Agreement. The financial obligations incurred by you shall survive the expiration or termination of this license.
+
+4. DISCLAIMER OF WARRANTY
+
+THIS SOFTWARE AND THE ACCOMPANYING FILES ARE SOLD "AS IS" AND WITHOUT WARRANTIES AS TO PERFORMANCE OR MERCHANTABILITY OR ANY OTHER WARRANTIES WHETHER EXPRESSED OR IMPLIED. THIS DISCLAIMER CONCERNS ALL FILES GENERATED AND EDITED BY ORY Hydra AS WELL.
+
+5. CONSENT OF USE OF DATA
+
+You agree that ORY GmbH may collect and use information gathered in any manner as part of the product support services provided to you, if any, related to ORY Hydra.ORY GmbH may also use this information to provide notices to you which may be of use or interest to you.
+

Dockerfile

@@ -1,29 +1,15 @@
-FROM golang:1.12.1-alpine
+# To compile this image manually run:
+#
+# $ GO111MODULE=on GOOS=linux GOARCH=amd64 go build && docker build -t oryd/hydra:v1.0.0-rc.7_oryOS.10 . && rm hydra
+FROM alpine:3.9
 
-ARG git_tag
-ARG git_commit
-
-RUN apk add --no-cache git build-base
-
-WORKDIR /go/src/github.com/ory/hydra
-
-ENV GO111MODULE=on
-
-ADD ./go.mod ./go.mod
-ADD ./go.sum ./go.sum
-
-RUN go mod download
-
-ADD . .
-
-RUN go mod verify
-RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-s -X github.com/ory/hydra/cmd.Version=$git_tag -X github.com/ory/hydra/cmd.BuildTime=`TZ=UTC date -u '+%Y-%m-%dT%H:%M:%SZ'` -X github.com/ory/hydra/cmd.GitHash=$git_commit" -a -installsuffix cgo -o hydra
+RUN apk add -U --no-cache ca-certificates
 
 FROM scratch
 
 COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
-COPY --from=0 /go/src/github.com/ory/hydra/hydra /usr/bin/hydra
+COPY hydra /usr/bin/hydra
+COPY .releaser/LICENSE.txt /LICENSE.txt
 
 ENTRYPOINT ["hydra"]
-
 CMD ["serve", "all"]