fix: reverted NodeGetInfo response as it breaks Nomad clusters #776
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We modified the response for `NodeGetInfo` to return an additional Topology Segment. We assumed that this only “adds” new info, but in practice it breaks the spec. When trying to schedule a volume to nodes, the CO should verify that the Node fulfills at least one Accessible Topology of the Node, where “fulfills” means that all supplied segments match. This is not implemented in the same way between Kubernetes and Nomad. - **Kubernetes**: requirements are fulfilled if the volume specifies a subset of the Nodes topology - **Nomad**: requirements are fulfilled if the volume specifies all of the Nodes topology We made these changes to work around a bug in the Kubernetes scheduler where nodes without the CSI Plugin would still be considered for scheduling, but then creating and attaching the volume fails with no automatic reconciliation of this error.
apricote
approved these changes
Nov 12, 2024
This was referenced Nov 12, 2024
lukasmetzner
pushed a commit
that referenced
this pull request
Nov 12, 2024
###⚠️ Removed Feature from v2.10.0 We have reverted a workaround for an upstream issue in the Kubernetes scheduler where nodes without the CSI Plugin (e.g. Robot servers) would still be considered for scheduling, but then creating and attaching the volume fails with no automatic reconciliation of the this error. Due to variations in the CSI specification implementation, these changes disrupted Nomad clusters, requiring us to revert them. We are actively working on placing this workaround behind a feature flag, allowing Kubernetes users to bypass the upstream issue. This affects you, if you have set the Helm value `allowedTopologyCloudServer` in v2.10.0. If you are affected by the Kubernetes upstream issue, we will provide a fix in the next minor version v2.11.0. Learn more about this in [#400](#400) and [#771](#771). ### Bug Fixes - reverted NodeGetInfo response as it breaks Nomad clusters (#776) Co-authored-by: releaser-pleaser <>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We modified the response for
NodeGetInfoto return an additional Topology Segment. We assumed that this only “adds” new info, but in practice it breaks the spec.When trying to schedule a volume to nodes, the container orchestration systems should verify that the Node fulfills at least one Accessible Topology of the Node, where “fulfills” means that all supplied segments match.
This is not implemented in the same way between Kubernetes and Nomad.
We made these changes to work around a bug in the Kubernetes scheduler (here) where nodes without the CSI Plugin would still be considered for scheduling, but then creating and attaching the volume fails with no automatic reconciliation of this error.