.NET Core Linux package feeds (RPM and Debian packages) have expected outages when any package is pushed to them #4167
Assignees
Labels
area-setup
Issues related to installing .NET Core
Comments
dagood
changed the title
|
@dagood Do you know what it is in particular about .net core that triggers thsi with these package managers? I've never seen this with a single other package.
is there an ongoing thread somewhere you can point us to to the communication you're having? |
|
It's all about https://packages.microsoft.com/. (VSCode and PowerShell use it too, for example.) We don't understand and don't yet have transparency into how the service works and why. The Microsoft linux package repository team has no public presence, so unfortunately there's nothing I can point to. We're going to post info here when we have something to share. |
It's very frustrating given the downstream effects that can last hours. Note that this is not a new phenomenon. |
|
The Microsoft linux packaging team should look into other packages distributed via apt because I’ve never seen any other package behave this way. Not even the more actively released packages. |
4 tasks
traversaro
added a commit
to robotology/idyntree
that referenced
this issue
Jan 25, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
traversaro
added a commit
to robotology/robotology-superbuild
that referenced
this issue
Jan 25, 2020
Microsoft apt repos break apt-get update for several days once in a while. As there is currently no plan to fix these outages from Microsoft side (see dotnet/core#4167) and we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323
Nicogene
added a commit
to robotology/icub-basic-demos
that referenced
this issue
Jan 27, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
Nicogene
added a commit
to robotology/icub-tests
that referenced
this issue
Jan 27, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
GiulioRomualdi
pushed a commit
to GiulioRomualdi/idyntree
that referenced
this issue
Jan 28, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
Nicogene
added a commit
to robotology/icub-basic-demos
that referenced
this issue
Jan 28, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
Nicogene
added a commit
to robotology/icub-basic-demos
that referenced
this issue
Jan 28, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
|
Hey folks, We're engaging with the packages.microsoft.com owners (face to face meeting next Thurs.) so that they clearly understand the issues and pain we're encountering. After that discussion we'll have a better understanding of their current service level definitions and roadmap to getting that to an acceptable level. I will update this issue after the meeting. |
Nicogene
added a commit
to robotology/icub-basic-demos
that referenced
this issue
Jan 29, 2020
Microsoft apt repos break apt-get update for several days at every update, and there is no plan to fix these. As we are not using the repos, the best course of action is just to remove them. See: * actions/runner-images#323 * dotnet/core#4167
This was referenced Jan 30, 2020
|
Hi @leecow do you have an update here? I've faced issues with this yesterday afternoon and this morning. Thanks in advance |
|
Hey @armitagemderivitec, if you're still hitting problems please file an issue with some details. We did meet with the team that owns the service on Feb 6th and we'll post an update as soon as we can, but update or not, we do still rely on everyone here telling us about problems so we can pass them along. |
|
Hey folks, As @dagood mentions, we met with the team to understand their current status, goals, and timelines for improving reliability of the service. They have a maintenance rollout occurring today which includes some reliability enhancements. We'll keep a close eye on things over the next day or so to see if we can determine performance increases and your input on reliability issues will be an important data point. We expressed that repo reliability was the top priority to which they readily agreed. They are working on an overall roadmap for larger work which will span a bit of time and we'll share as much as we can as soon as we see the planning info. |
|
Is there a technical write up anywhere discussing what you guys are doing so differently from others that this was/is a problem? Doesn’t have to be a white paper just a couple of sentences so everyone (including your customers) learns from this experience.
|
jrfnl
added a commit
to PHPCompatibility/PHPCompatibilityParagonie
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to PHPCompatibility/PHPCompatibilityPasswordCompat
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to PHPCompatibility/PHPCompatibilitySymfony
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to PHPCompatibility/PHPCompatibilityWP
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to sirbrillig/phpcs-variable-analysis
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to WordPress/WordPress-Coding-Standards
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to Yoast/yoastcs
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
This was referenced Apr 24, 2024
Merged
Merged
Merged
Merged
Merged
jrfnl
added a commit
to PHPCompatibility/PHPCompatibility
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
sirbrillig
pushed a commit
to sirbrillig/phpcs-variable-analysis
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167 Co-authored-by: jrfnl <[email protected]>
jrfnl
added a commit
to PHPCompatibility/PHPCompatibilityParagonie
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to PHPCSStandards/PHP_CodeSniffer
that referenced
this issue
Apr 24, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
jrfnl
added a commit
to WordPress/WordPress-Coding-Standards
that referenced
this issue
Apr 25, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
rebeccahum
pushed a commit
to Automattic/VIP-Coding-Standards
that referenced
this issue
May 2, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
grogy
pushed a commit
to php-parallel-lint/PHP-Code-Style
that referenced
this issue
May 20, 2024
Okay, so apparently, there is a long-standing bug in the Microsoft package deploy process which caused `apt-get update` to fail in the first half hour after Microsoft has deployed a package. The failure looks like this: ``` E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?) ``` As this only happens intermittently (after a MS package deploy), the chance of running into this bug are slim, but guess what: today I ran into it. This change to the workflow is intended to prevent the next person running into this issue from having to waste time on figuring this out. By splitting the "Install xmllint" step into two steps: one doing the `apt-get update` and one doing the actual install and making the first step one which is allowed to `continue-on-error`, this issue should hopefully not crop up anymore. Any errors in the `apt-get update` step will now be ignored and as most errors which could potentially come from that step are irrelevant for the rest of the job anyway, this is fine. If a relevant error would be surfaced, the next step (the xmllint install), will fail the job anyway. Refs: * actions/runner-images#3410 * dotnet/core#4167
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Linux package repositories service has a temporary outage whenever packages are pushed to the feed. It depends on how fast syncing can occur, which includes signing the repo data.
Debian packages: up to 30 minutes is expected
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2 File has unexpected size (72896 != 72828). Mirror sync in progress? [IP: ...]RPM packages: up to 2 hours is expected
Signature verification failed for file 'repomd.xml' from repository 'packages-microsoft-com-prod'.If you hit this issue and it doesn't resolve itself after the expected time has passed, please file an issue in this repo with the error message and the distro/version you're using.
There unfortunately isn't any status page that gets updated when outages occur.
We are tracking adding this known issue to the docs for more visibility: dotnet/docs#16670.
We are trying to work with the Linux package repository team to improve quality of the service. We (.NET Core) and other Microsoft teams depend on this common infrastructure to make our Linux packages available.
I'm planning to leave this issue open to track progress working with the Linux package repository team, and close instances of the problem as a duplicate of this issue.
/cc @dleeapho @leecow @NikolaMilosavljevic @MichaelSimons
The text was updated successfully, but these errors were encountered: