Skip to content

Commit

Permalink
refactor(artifacts): Extract artifact resolution into ArtifactResolve…
Browse files Browse the repository at this point in the history
…r class (spinnaker#3377)

* refactor(artifacts): Add ArtifactResolver class

The ArtifactUtils class handles a lot of functionality and is difficult
to understand. Simplify this by creating an ArtifactResolver class to
handle resolving artifacts, so that functionality can be removed from
ArtifactUtils.

One important change in this class is that it does not mutate any of the
Artifact or ExpectedArtifact instances it receives; it returns the
results of its work (containing new instances) which the caller should
then use as appropriate. This will help reduce bugs where the caller is
depending on a mutation happening several calls below the call stack,
and brings us closer to making Artifact and ExpectedArtifact immutable.

* refactor(artifacts): Update callers to use ArtifactResolver

This commit updates all artifact resolving code to use the new
ArtifactResolver class.

A few things of note in the changes:
* FindArtifactFromExecutionTaskTest was mocking resolveExpectedArtifacts
to return resolvedArtifacts; this was subtly different than the actual
function because the actual resolvedArtifacts function mutated the
input expectedArtifacts to add a boundArtifact to each. As the
resolveExpectedArtifacts method has been moved to a simpler class that
we don't need to mock, the test is calling the actual method now and
so we need to update the expected result to account for the fact that
the resolvedExpectedArtifacts now have a bound artifact.
* All of the tests deleted from ArtifactUtilsSpec have equivalent new
tests in ArtifactResolverTest which test the new implementation of
artifact resolving.
* The one changed test in ArtifactUtilsSpec (...sets the bound
artifact...) is a result of fixing what was a bug in the prior code.
As the prior code was mutating each ExpectedArtifact, we were mutating
the expected artifacts on the pipeline itself (and testing that).
resolveArtifacts is storing the resolved artifacts in the trigger,
under resolvedArtifacts; this is where downstream stages look to find
bound artifacts and where we should be testing that the bound artifact
is set. The test now checks that the bound artifacts are set in the
trigger.

* refactor(artifacts): Clean up some stream code

The code to map input artifacts in resolveArtifacts() is more complex
and verbose than necessary.  Simplify it by moving the .orElse before
the map operation (which means we don't need to have a nested stream)
and also inline a two streams that are used only once.

* perf(artifacts): Lazily compute past pipeline artifacts

The resolveArtifacts function always passes the prior artifacts
to the ArtifactResolver (except when it short-circuits in the
event of no expected artifacts). Looking up artifacts from prior
executions can be expensive, so ideally we'd defer looking these
up until they are actually needed.

This change updates the constructor for ArtifactResolver to accept
a Supplier of prior artifacts; it will memoize this supplier and
only invoke it if the resolution requires looking up prior artifacts
(that is, the first time it encounters an expected artifact with
usePriorAritfact set to true that does not match one of the current
artifacts).

This will also allow us in the next commit to remove the
short-circuiting for the case where there are no expected artifacts
without worrying about negatively affecting performance.

* refactor(artifacts): A few minor refactors of resolveArtifacts

Replace the List<ExpectedArtifact> with an ImmutableList<> as we
never modify it. (And there's a tiny performance benefit that
ArtifactResolver.getInstance should short-circuit the copy.)

Replace .distinct() collecting to a List<> of receivedArtifacts
with directly collecting to toImmutableSet() as it will remove
duplicates and return a set whose iteration order is the encounter
order in the stream.

Remove the short-circuit that returns early if there are no expected
artifacts; the ArtifactResolver is a no-op that returns empty
lists for resolvedArtifacts and resolvedExpectedArtifacts in that
case, so there is no need to special-case it.

Replace the LinkedHashSet code that creates a deduplicated list of
artifacts in the supplied order with an ImmutableSet that does the same.

* fix(artifacts): Fix typo in comment

Co-Authored-By: Maggie Neterval <[email protected]>

* refactor(artifacts): Make static factory parameters non-null

We never actually need currentArtifacts to be null, so remove the
nullable annotation. To handle priorArtifacts, add an overload that
omits the parameter.

Also update the resolveExpectedArtifacts parameter to be non-null.
In order to ensure a safe non-null chain of custody, also added
Nonnull to the artifacts field in FindArtifactFromExecutionContext,
and while there made it immutable.  (Also slightly changed the
constructor as prior to this the case where both expectedArtifact
and expectedArtifacts were null create a singleton array with a null
element that was likely to create downstream hard-to-debug issues.)

* refactor(artifacts): Improve method order in ArtifactResolver

Move resolveExpectedArtifacts above the private methods it uses.

* style(artifacts): Add parameter comments for constant parameters

Add documentation of the parameter name for all of the usages of the
requireUniqueMatches parameter.

Co-authored-by: Ethan Rogers <[email protected]>
Co-authored-by: Maggie Neterval <[email protected]>
  • Loading branch information
3 people authored and mergify[bot] committed Jan 22, 2020
1 parent 2e78355 commit db7988f
Show file tree
Hide file tree
Showing 9 changed files with 587 additions and 269 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -17,34 +17,40 @@
package com.netflix.spinnaker.orca.clouddriver.tasks.artifacts;

import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.common.collect.ImmutableList;
import com.netflix.spinnaker.kork.artifacts.model.ExpectedArtifact;
import com.netflix.spinnaker.orca.pipeline.persistence.ExecutionRepository.ExecutionCriteria;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Optional;
import javax.annotation.Nonnull;
import javax.annotation.ParametersAreNullableByDefault;
import lombok.Data;
import lombok.Getter;

@Getter
public class FindArtifactFromExecutionContext extends HashMap<String, Object> {
private final ExecutionOptions executionOptions;
private final List<ExpectedArtifact> expectedArtifacts;
@Nonnull private final ImmutableList<ExpectedArtifact> expectedArtifacts;
private final String pipeline;

// There does not seem to be a way to auto-generate a constructor using our current version of
// Lombok (1.16.20) that
// Jackson can use to deserialize.
// Lombok (1.16.20) that Jackson can use to deserialize.
@ParametersAreNullableByDefault
public FindArtifactFromExecutionContext(
@JsonProperty("executionOptions") ExecutionOptions executionOptions,
@JsonProperty("expectedArtifact") ExpectedArtifact expectedArtifact,
@JsonProperty("expectedArtifacts") List<ExpectedArtifact> expectedArtifacts,
@JsonProperty("pipeline") String pipeline) {
this.executionOptions = executionOptions;
// Previously, this stage accepted only one expected artifact
this.expectedArtifacts =
Optional.ofNullable(expectedArtifacts).orElse(Collections.singletonList(expectedArtifact));
if (expectedArtifacts != null) {
this.expectedArtifacts = ImmutableList.copyOf(expectedArtifacts);
} else if (expectedArtifact != null) {
this.expectedArtifacts = ImmutableList.of(expectedArtifact);
} else {
this.expectedArtifacts = ImmutableList.of();
}
this.pipeline = pipeline;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,19 @@

package com.netflix.spinnaker.orca.clouddriver.tasks.artifacts;

import com.google.common.collect.ImmutableList;
import com.netflix.spinnaker.kork.artifacts.model.Artifact;
import com.netflix.spinnaker.kork.artifacts.model.ExpectedArtifact;
import com.netflix.spinnaker.orca.ExecutionStatus;
import com.netflix.spinnaker.orca.Task;
import com.netflix.spinnaker.orca.TaskResult;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import com.netflix.spinnaker.orca.pipeline.util.ArtifactResolver;
import com.netflix.spinnaker.orca.pipeline.util.ArtifactUtils;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import javax.annotation.Nonnull;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Component;
Expand All @@ -46,7 +47,7 @@ public TaskResult execute(@Nonnull Stage stage) {
FindArtifactFromExecutionContext context = stage.mapTo(FindArtifactFromExecutionContext.class);
Map<String, Object> outputs = new HashMap<>();
String pipeline = context.getPipeline();
List<ExpectedArtifact> expectedArtifacts = context.getExpectedArtifacts();
ImmutableList<ExpectedArtifact> expectedArtifacts = context.getExpectedArtifacts();
FindArtifactFromExecutionContext.ExecutionOptions executionOptions =
context.getExecutionOptions();

Expand All @@ -64,11 +65,12 @@ public TaskResult execute(@Nonnull Stage stage) {
artifactUtils.getArtifactsForPipelineId(pipeline, executionOptions.toCriteria());
}

Set<Artifact> matchingArtifacts =
artifactUtils.resolveExpectedArtifacts(expectedArtifacts, priorArtifacts, null, false);
ArtifactResolver.ResolveResult resolveResult =
ArtifactResolver.getInstance(priorArtifacts, /* requireUniqueMatches= */ false)
.resolveExpectedArtifacts(expectedArtifacts);

outputs.put("resolvedExpectedArtifacts", expectedArtifacts);
outputs.put("artifacts", matchingArtifacts);
outputs.put("resolvedExpectedArtifacts", resolveResult.getResolvedExpectedArtifacts());
outputs.put("artifacts", resolveResult.getResolvedArtifacts());

return TaskResult.builder(ExecutionStatus.SUCCEEDED).context(outputs).outputs(outputs).build();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import static org.mockito.Mockito.*;

import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.netflix.spinnaker.kork.artifacts.model.Artifact;
import com.netflix.spinnaker.kork.artifacts.model.ExpectedArtifact;
import com.netflix.spinnaker.orca.TaskResult;
Expand All @@ -32,7 +31,6 @@
import com.netflix.spinnaker.orca.pipeline.util.ArtifactUtils;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.Map;
import org.junit.jupiter.api.Test;
import org.junit.platform.runner.JUnitPlatform;
Expand All @@ -54,7 +52,6 @@ final class FindArtifactFromExecutionTaskTest {
void findsSingleArtifact() {
ImmutableList<ExpectedArtifact> expectedArtifacts = ImmutableList.of(EXPECTED_ARTIFACT_A);
ImmutableList<Artifact> pipelineArtifacts = ImmutableList.of(ARTIFACT_A, ARTIFACT_B);
LinkedHashSet<Artifact> resolvedArtifacts = new LinkedHashSet<>(ImmutableSet.of(ARTIFACT_A));
Stage stage =
new Stage(
mock(Execution.class), "findArtifactFromExecution", getStageContext(expectedArtifacts));
Expand All @@ -63,15 +60,14 @@ void findsSingleArtifact() {
when(artifactUtils.getArtifactsForPipelineId(
eq(PIPELINE), any(ExecutionRepository.ExecutionCriteria.class)))
.thenReturn(pipelineArtifacts);
when(artifactUtils.resolveExpectedArtifacts(expectedArtifacts, pipelineArtifacts, null, false))
.thenReturn(resolvedArtifacts);

FindArtifactFromExecutionTask task = new FindArtifactFromExecutionTask(artifactUtils);
TaskResult result = task.execute(stage);

Collection<ExpectedArtifact> resolvedExpectedArtifacts =
(Collection<ExpectedArtifact>) result.getContext().get("resolvedExpectedArtifacts");
assertThat(resolvedExpectedArtifacts).containsExactly(EXPECTED_ARTIFACT_A);
assertThat(resolvedExpectedArtifacts)
.containsExactly(withBoundArtifact(EXPECTED_ARTIFACT_A, ARTIFACT_A));

Collection<Artifact> artifacts = (Collection<Artifact>) result.getContext().get("artifacts");
assertThat(artifacts).containsExactly(ARTIFACT_A);
Expand All @@ -82,8 +78,6 @@ void findsMultipleArtifacts() {
ImmutableList<ExpectedArtifact> expectedArtifacts =
ImmutableList.of(EXPECTED_ARTIFACT_A, EXPECTED_ARTIFACT_B);
ImmutableList<Artifact> pipelineArtifacts = ImmutableList.of(ARTIFACT_A, ARTIFACT_B);
LinkedHashSet<Artifact> resolvedArtifacts =
new LinkedHashSet<>(ImmutableSet.of(ARTIFACT_A, ARTIFACT_B));
Stage stage =
new Stage(
mock(Execution.class), "findArtifactFromExecution", getStageContext(expectedArtifacts));
Expand All @@ -92,15 +86,16 @@ void findsMultipleArtifacts() {
when(artifactUtils.getArtifactsForPipelineId(
eq(PIPELINE), any(ExecutionRepository.ExecutionCriteria.class)))
.thenReturn(pipelineArtifacts);
when(artifactUtils.resolveExpectedArtifacts(expectedArtifacts, pipelineArtifacts, null, false))
.thenReturn(resolvedArtifacts);

FindArtifactFromExecutionTask task = new FindArtifactFromExecutionTask(artifactUtils);
TaskResult result = task.execute(stage);

Collection<ExpectedArtifact> resolvedExpectedArtifacts =
(Collection<ExpectedArtifact>) result.getContext().get("resolvedExpectedArtifacts");
assertThat(resolvedExpectedArtifacts).containsExactly(EXPECTED_ARTIFACT_A, EXPECTED_ARTIFACT_B);
assertThat(resolvedExpectedArtifacts)
.containsExactly(
withBoundArtifact(EXPECTED_ARTIFACT_A, ARTIFACT_A),
withBoundArtifact(EXPECTED_ARTIFACT_B, ARTIFACT_B));

Collection<Artifact> artifacts = (Collection<Artifact>) result.getContext().get("artifacts");
assertThat(artifacts).containsExactly(ARTIFACT_A, ARTIFACT_B);
Expand All @@ -118,4 +113,9 @@ private static Map<String, Object> getStageContext(

return context;
}

private static ExpectedArtifact withBoundArtifact(
ExpectedArtifact expectedArtifact, Artifact artifact) {
return expectedArtifact.toBuilder().boundArtifact(artifact).build();
}
}
1 change: 1 addition & 0 deletions orca-core/orca-core.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ dependencies {
testImplementation(project(":orca-test"))
testImplementation(project(":orca-test-groovy"))
testImplementation("org.junit.jupiter:junit-jupiter-api")
testImplementation("org.junit.platform:junit-platform-runner")
testImplementation("org.assertj:assertj-core")

testRuntimeOnly("org.junit.jupiter:junit-jupiter-api")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@
import com.netflix.spinnaker.orca.Task;
import com.netflix.spinnaker.orca.TaskResult;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import com.netflix.spinnaker.orca.pipeline.util.ArtifactResolver;
import com.netflix.spinnaker.orca.pipeline.util.ArtifactUtils;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import javax.annotation.Nonnull;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
Expand Down Expand Up @@ -59,11 +59,12 @@ public TaskResult execute(@Nonnull Stage stage) {
}

List<Artifact> artifacts = artifactUtils.getArtifacts(stage);
Set<Artifact> resolvedArtifacts =
artifactUtils.resolveExpectedArtifacts(expectedArtifacts, artifacts, false);
ArtifactResolver.ResolveResult resolveResult =
ArtifactResolver.getInstance(artifacts, /* requireUniqueMatches= */ false)
.resolveExpectedArtifacts(expectedArtifacts);

outputs.put("artifacts", resolvedArtifacts);
outputs.put("resolvedExpectedArtifacts", expectedArtifacts);
outputs.put("artifacts", resolveResult.getResolvedArtifacts());
outputs.put("resolvedExpectedArtifacts", resolveResult.getResolvedExpectedArtifacts());

return TaskResult.builder(ExecutionStatus.SUCCEEDED).context(outputs).outputs(outputs).build();
}
Expand Down
Loading

0 comments on commit db7988f

Please sign in to comment.