From d7a412f32e390e49d94dccd503bceb69c7c67c1f Mon Sep 17 00:00:00 2001 From: sam boyer Date: Mon, 9 Jul 2018 03:25:43 -0400 Subject: [PATCH] dep: Update docs to reflect vendor verification --- docs/Gopkg.lock.md | 41 +++++++++++++++++++++++++++++----------- docs/daily-dep.md | 2 +- docs/ensure-mechanics.md | 2 -- docs/env-vars.md | 6 +----- docs/glossary.md | 2 +- 5 files changed, 33 insertions(+), 20 deletions(-) diff --git a/docs/Gopkg.lock.md b/docs/Gopkg.lock.md index 7b7876e210..1a930fcccb 100644 --- a/docs/Gopkg.lock.md +++ b/docs/Gopkg.lock.md @@ -1,5 +1,5 @@ --- -title: Gopkg.lock + title: Gopkg.lock --- The `Gopkg.lock` file is generated by `dep ensure` and `dep init`. It is the output of [the solving function](ensure-mechanics.md#functional-flow): a transitively complete snapshot of a project's dependency graph, expressed as a series of `[[project]]` stanzas. That means: @@ -8,7 +8,7 @@ The `Gopkg.lock` file is generated by `dep ensure` and `dep init`. It is the out * Plus any [`required`](Gopkg.toml.md#required) packages * Less any [`ignored`](Gopkg.toml.md#ignored) packages -`Gopkg.lock` also contains some metadata about the algorithm used to arrive at the final graph, under `[solve-meta]`. +`Gopkg.lock` also contains some metadata about the algorithm and inputs used to arrive at the final graph, under `[solve-meta]`. `Gopkg.lock` always includes a `revision` for all listed dependencies, as the semantics of `revision` guarantee them to be immutable. Thus, the `Gopkg.lock` acts as a reproducible build list - as long as the upstream remains available, all dependencies can be precisely reproduced. @@ -28,6 +28,8 @@ These are all the properties that can appear in a `[[projects]]` stanza, and whe | `revision` | Y | | `version` | N | | `branch` | N | +| `pruneopts` | Y | +| `digest` | Y | ### `name` @@ -47,6 +49,27 @@ In general, this is the set of packages that were found to be participants in th * Being imported by a package from either the current project or a different dependency * Being imported by a package from within this project that, directly or transitively, is imported by a package from a different project +### `pruneopts` + +A compactly-encoded form of the [prune options designated in `Gopkg.toml`](Gopkg.toml.md#prune) . Each character represents one of the three possible rules: + +| Character | Pruning Rule in `Gopkg.toml` | +| --------- | ---------------------------- | +| `N` | `non-go` | +| `U` | `unused-packages` | +| `T` | `go-tests` | + +If the character is present in `pruneopts`, the pruning rule is enabled for that project. Thus, `NUT` indicates that all three pruning rules are active. + +### `digest` + +The hash digest of the contents of `vendor/` for this project, _after_ pruning rules have been applied. The digest is versioned, by way of a colon-delimited prefix; the string is of the form `:` . The hashing algorithm corresponding to version 1 is SHA256, as implemented in the stdlib package `crypto/sha256`. + +There are some tweaks that differentiate the hasher apart from a naive filesystem tree hashing implementation: + +* Symlinks are ignored. +* Line endings are normalized to LF (using an algorithm similar to git's) in order to ensure digests do not vary across platforms. + ### Version information: `revision`, `version`, and `branch` In order to provide reproducible builds, it is an absolute requirement that every project stanza contain a `revision`, no matter what kinds of constraints were encountered in `Gopkg.toml` files. It is further possible that exactly one of either `version` or `branch` will _additionally_ be present. @@ -61,6 +84,10 @@ More details on "analyzer" and "solver" follow, but the versioning principle is By bumping versions only on solution set contractions, but not expansions, it allows us to avoid having to bump constantly (which could make using dep across teams awkward), while still making it likely that when the solver and version numbers match between `Gopkg.lock` and a running version of dep, what's recorded in the file is acceptable by the running version's rules. +### `input-imports` + +A sorted list of all the import inputs that were present at the time the `Gopkg.lock` was computed. This list includes both actual `import` statements from the project, as well as any `required` import paths listed in `Gopkg.toml`. + ### `analyzer-name` and `analyzer-version` The analyzer is an internal dep component responsible for interpreting the contents of `Gopkg.toml` files, as well as metadata files from any tools dep knows about: `glide.yaml`, `vendor.json`, etc. @@ -75,12 +102,4 @@ The solver is the algorithm behind [the solving function](ensure-mechanics.md#fu The solver is named because, like the analyzer, it is pluggable; an alternative algorithm could be written that applies different rules to achieve the same goal. The one dep uses, "gps-cdcl", is named after [the general class of SAT solving algorithm it most resembles](https://en.wikipedia.org/wiki/Conflict-Driven_Clause_Learning), though the algorithm is actually a specialized, domain-specific [SMT solver](https://en.wikipedia.org/wiki/Satisfiability_modulo_theories). -The same general principles of version-bumping apply to the solver version: if the solver starts enforcing [Go 1.4 import path comments](https://golang.org/cmd/go/#hdr-Import_path_checking), that entails a bump, because it can only narrow the solution set. If it were to later relax that requirement, it would not require a bump, as that can only expand the solution set. - -### `inputs-digest` - -A SHA256 hash digest of all the [inputs to the solving function](ensure-mechanics.md#functional-flow). Those inputs can be shown directly with the hidden command `dep hash-inputs`, allowing this value to be generated directly: - -``` -dep hash-inputs | tr -d “\n” | shasum -a256 -``` +The same general principles of version-bumping apply to the solver version: if the solver starts enforcing [Go 1.4 import path comments](https://golang.org/cmd/go/#hdr-Import_path_checking), that entails a bump, because it can only narrow the solution set. If it were to later relax that requirement, it would not require a bump, as that can only expand the solution set. \ No newline at end of file diff --git a/docs/daily-dep.md b/docs/daily-dep.md index 7ef4432d01..7bf0066e37 100644 --- a/docs/daily-dep.md +++ b/docs/daily-dep.md @@ -83,7 +83,7 @@ $ dep ensure -update ### Adding and removing `import` statements -As noted in [the section on adding dependencies](#adding-a-new-dependency), dep relies on the import statements in your code to figure out which dependencies your project actually needs. Thus, when you add or remove import statements, dep might need to care about it. +As noted in [the section on adding dependencies](#adding-a-new-dependency), dep relies on the `import` statements in your code to figure out which dependencies your project actually needs. Thus, when you add or remove import statements, dep might need to care about it. It's only "might," though, because most of the time, adding or removing imports doesn't matter to dep. Only if one of the following has occurred will a `dep ensure` be necessary to bring the project back in sync: diff --git a/docs/ensure-mechanics.md b/docs/ensure-mechanics.md index 7d813f82a8..78b80b6359 100644 --- a/docs/ensure-mechanics.md +++ b/docs/ensure-mechanics.md @@ -51,9 +51,7 @@ The four state system, and these functional flows through it, are the foundation One of dep's design goals is that both of its "functions" minimize both the work they do, and the change they induce in their respective results. (Note: "minimize" is not currently formally defined with respect to a cost function.) Consequently, both functions peek ahead at the pre-existing result to understand what work actually needs to be done: * The solving function checks the existing `Gopkg.lock` to determine if all of its inputs (project import statements + `Gopkg.toml` rules) are satisfied. If they are, the solving function can be bypassed entirely. If not, the solving function proceeds, but attempts to change as few of the selections in `Gopkg.lock` as possible. - * WIP: The current implementation's check relies on a coarse heuristic check that can be wrong in some cases. There is a [plan to fix this](https://github.com/golang/dep/issues/1496). * The vendoring function hashes each discrete project already in `vendor/` to see if the code present on disk is what `Gopkg.lock` indicates it should be. Only projects that deviate from expectations are written out. - * WIP: the hashing check is generally referred to as "vendor verification," and [is not yet complete](https://github.com/golang/dep/issues/121). Without this verification, dep is blind to whether code in `vendor/` is correct or not; as such, dep must defensively re-write all projects to ensure the state of `vendor/` is correct. Of course, it's possible that, in peeking ahead, either function might discover that the pre-existing result is already correct - so no work need be done at all. Either way, when each function completes, we can be sure that the output, changed or not, is correct with respect to the inputs. In other words, the inputs and outputs are "in sync." Indeed, being in sync is the "known good state" of dep; `dep ensure` (without flags) guarantees that if it exits 0, all four states in the project are in sync. diff --git a/docs/env-vars.md b/docs/env-vars.md index 83aeed9218..7f38fef194 100644 --- a/docs/env-vars.md +++ b/docs/env-vars.md @@ -25,8 +25,4 @@ This is primarily useful if you're not using the standard `go` toolchain as a co ### `DEPNOLOCK` -By default, dep creates an `sm.lock` file at `$DEPCACHEDIR/sm.lock` in order to -prevent multiple dep processes from interacting with the [local -cache](glossary.md#local-cache) simultaneously. Setting this variable will -bypass that protection; no file will be created. This can be useful on certain -filesystems; VirtualBox shares in particular are known to misbehave. +By default, dep creates an `sm.lock` file at `$DEPCACHEDIR/sm.lock` in order to prevent multiple dep processes from interacting with the [local cache](glossary.md#local-cache) simultaneously. Setting this variable will bypass that protection; no file will be created. This can be useful on certain filesystems; VirtualBox shares in particular are known to misbehave. diff --git a/docs/glossary.md b/docs/glossary.md index 9c491aa211..5b78e980d3 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -69,7 +69,7 @@ An `import` statement that points to a package in a project other than the one i ### GPS -Stands for "Go packaging solver", it is [a subtree of library-style packages within dep](https://godoc.org/github.com/golang/dep/gps), and is the engine around which dep is built. Most commonly referred to as "gps." +Acronym for "Go packaging solver", it is [a subtree of library-style packages within dep](https://godoc.org/github.com/golang/dep/gps), and is the engine around which dep is built. Most commonly referred to as "gps." ### Local cache