Add HEALTHCHECK support #421
Conversation
WalkthroughThis pull request introduces a new container health monitoring mechanism for the Tailscale project. A health check command is added to the Dockerfile, a new healthcheck script is implemented to assess the Tailscale service's state via JSON status, and the service management script is updated to include the “Stopped” state in its condition checks. The overall container setup and environment configurations remain unchanged. Changes
Sequence Diagram(s)sequenceDiagram
participant D as Docker Healthcheck
participant H as Healthcheck Script
participant T as Tailscale Service
D->>H: Execute healthcheck command
H->>T: Run 'tailscale status'
T-->>H: Return JSON status
H->>D: Return health status (exit code)
sequenceDiagram
participant S as Service Manager
participant T as Tailscale Backend
S->>T: Query current backend state
T-->>S: Return state ("Running", "NeedsLogin", "Stopped")
S->>S: Evaluate condition to proceed based on state
Suggested labels
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
lmagyar
force-pushed
the
pr-healthcheck
branch
from
d124525 to
5a3ea4d
Compare
|
This behaves as expected in live environments and no issues reported by real users. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (6)
tailscale/rootfs/usr/bin/healthcheck (1)
25-25: Useprintfwith format strings to prevent unintended behaviorIn lines 25, 36, 49, and 57, you're using
printfwithout specifying a format string. This can lead to unexpected behavior if the variables contain format specifiers or special characters. It's safer to provide a format string.Apply this diff to fix the issue:
-printf "${STARTED_TIMESTAMP}" > /var/run/s6/container_environment/STARTED_TIMESTAMP +printf '%s\n' "${STARTED_TIMESTAMP}" > /var/run/s6/container_environment/STARTED_TIMESTAMP -printf "${LAST_ONLINE_TIMESTAMP}" > /var/run/s6/container_environment/LAST_ONLINE_TIMESTAMP +printf '%s\n' "${LAST_ONLINE_TIMESTAMP}" > /var/run/s6/container_environment/LAST_ONLINE_TIMESTAMP -printf "UNHEALTHY" > /var/run/s6/container_environment/LAST_REPORTED_HEALTH_STATE +printf '%s\n' "UNHEALTHY" > /var/run/s6/container_environment/LAST_REPORTED_HEALTH_STATE -printf "HEALTHY" > /var/run/s6/container_environment/LAST_REPORTED_HEALTH_STATE +printf '%s\n' "HEALTHY" > /var/run/s6/container_environment/LAST_REPORTED_HEALTH_STATEAlso applies to: 36-36, 49-49, 57-57
tailscale/translations/en.yaml (1)
47-55: Minor grammatical improvements in descriptionsConsider the following minor changes to enhance clarity:
- Line 47: Add "the" before "timeout": "This option allows you to set the timeout in seconds..."
- Line 54: Change "ie." to "i.e.": "The minimum value is 900, i.e., 15 minutes."
- Line 55: Rephrase for clarity: "You may set it to several minutes or even hours."
Apply this diff to implement the changes:
- This option allows you to set timeout in seconds for Tailscale to be offline + This option allows you to set the timeout in seconds for Tailscale to be offline - after (re)start. The minimum value is 900, ie. 15 minutes. Set it to several + after (re)start. The minimum value is 900, i.e., 15 minutes. You may set it to severaltailscale/DOCS.md (4)
73-74: Consider documenting the rationale for example timeout valuesThe example configuration shows specific timeout values (110s for offline and 3600s for restart) without explaining why these particular values were chosen. Consider adding a brief comment to help users understand these choices.
healthcheck_offline_timeout: 110 # Time to wait before restarting after going offline healthcheck_restart_timeout: 3600 # Time to wait for recovery after restart (1 hour)
194-215: Fix abbreviation formatting and enhance clarityThe documentation for
healthcheck_offline_timeoutis comprehensive but needs some grammatical improvements:
- Fix abbreviation formatting:
-Tailscale was once online, ie. it won't have any effect when +Tailscale was once online, i.e., it won't have any effect when -Tailscale's status is eg. Starting +Tailscale's status is e.g., Starting -Assistant in each 30s, ie. the effective +Assistant in each 30s, i.e., the effective
- Consider adding an example scenario to make it more concrete:
+For example, if set to 300 seconds (5 minutes), the add-on will restart if Tailscale goes offline and doesn't recover within 5 minutes of detecting the offline state.🧰 Tools
🪛 LanguageTool
[uncategorized] ~204-~204: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...s only after Tailscale was once online, ie. it won't have any effect when Tailscale...(I_E)
[uncategorized] ~205-~205: The abbreviation “e.g.” (= for example) requires two periods.
Context: ...e any effect when Tailscale's status is eg. Starting, NeedsLogin or NeedsMachineAut...(E_G)
[uncategorized] ~213-~213: The abbreviation “i.e.” (= that is) requires two periods.
Context: ... checked by Home Assistant in each 30s, ie. the effective resolution of this option...(I_E)
216-241: Fix abbreviation formatting and clarify timeout behaviorThe documentation for
healthcheck_restart_timeoutneeds grammatical improvements:Fix abbreviation formatting:
-The minimum value is 900, ie. 15 minutes +The minimum value is 900, i.e., 15 minutes -Tailscale is starting, ie. it won't have +Tailscale is starting, i.e., it won't have -Assistant in each 30s, ie. the effective +Assistant in each 30s, i.e., the effective🧰 Tools
🪛 LanguageTool
[uncategorized] ~221-~221: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...r (re)start. The minimum value is 900, ie. 15 minutes, to always give you enough t...(I_E)
[uncategorized] ~229-~229: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...appens only when Tailscale is starting, ie. it won't have any effect after Tailscal...(I_E)
[uncategorized] ~239-~239: The abbreviation “i.e.” (= that is) requires two periods.
Context: ... checked by Home Assistant in each 30s, ie. the effective resolution of this option...(I_E)
194-241: Consider adding a visual timeline diagramThe documentation thoroughly explains the health check mechanisms, but a visual timeline diagram could help users better understand how
healthcheck_offline_timeoutandhealthcheck_restart_timeoutinteract with different Tailscale states.Example diagram structure:
Timeline: [Starting] -> [Online] -> [Offline] -> [Restarting] -> [Online] | | | | | | | | restart_timeout | offline_timeout restart_timeout🧰 Tools
🪛 LanguageTool
[uncategorized] ~204-~204: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...s only after Tailscale was once online, ie. it won't have any effect when Tailscale...(I_E)
[uncategorized] ~205-~205: The abbreviation “e.g.” (= for example) requires two periods.
Context: ...e any effect when Tailscale's status is eg. Starting, NeedsLogin or NeedsMachineAut...(E_G)
[uncategorized] ~213-~213: The abbreviation “i.e.” (= that is) requires two periods.
Context: ... checked by Home Assistant in each 30s, ie. the effective resolution of this option...(I_E)
[uncategorized] ~221-~221: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...r (re)start. The minimum value is 900, ie. 15 minutes, to always give you enough t...(I_E)
[uncategorized] ~229-~229: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...appens only when Tailscale is starting, ie. it won't have any effect after Tailscal...(I_E)
[uncategorized] ~239-~239: The abbreviation “i.e.” (= that is) requires two periods.
Context: ... checked by Home Assistant in each 30s, ie. the effective resolution of this option...(I_E)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (6)
tailscale/DOCS.md(2 hunks)tailscale/Dockerfile(1 hunks)tailscale/config.yaml(1 hunks)tailscale/rootfs/etc/s6-overlay/s6-rc.d/post-tailscaled/run(1 hunks)tailscale/rootfs/usr/bin/healthcheck(1 hunks)tailscale/translations/en.yaml(1 hunks)
🧰 Additional context used
🪛 LanguageTool
tailscale/DOCS.md
[uncategorized] ~204-~204: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...s only after Tailscale was once online, ie. it won't have any effect when Tailscale...
(I_E)
[uncategorized] ~205-~205: The abbreviation “e.g.” (= for example) requires two periods.
Context: ...e any effect when Tailscale's status is eg. Starting, NeedsLogin or NeedsMachineAut...
(E_G)
[uncategorized] ~213-~213: The abbreviation “i.e.” (= that is) requires two periods.
Context: ... checked by Home Assistant in each 30s, ie. the effective resolution of this option...
(I_E)
[uncategorized] ~221-~221: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...r (re)start. The minimum value is 900, ie. 15 minutes, to always give you enough t...
(I_E)
[uncategorized] ~229-~229: The abbreviation “i.e.” (= that is) requires two periods.
Context: ...appens only when Tailscale is starting, ie. it won't have any effect after Tailscal...
(I_E)
[uncategorized] ~239-~239: The abbreviation “i.e.” (= that is) requires two periods.
Context: ... checked by Home Assistant in each 30s, ie. the effective resolution of this option...
(I_E)
🔇 Additional comments (3)
tailscale/config.yaml (1)
39-39: Verify the schema definition for healthcheck_restart_timeout
The schema defines healthcheck_restart_timeout as int(900,)?. Please ensure that this syntax correctly enforces a minimum value of 900 as intended. If the schema requires a different format to enforce a minimum value, consider updating it accordingly.
tailscale/Dockerfile (1)
38-40: HEALTHCHECK command added correctly
The addition of the HEALTHCHECK command appears to be correct and properly implemented.
tailscale/rootfs/etc/s6-overlay/s6-rc.d/post-tailscaled/run (1)
93-93: Verify inclusion of "Stopped" in backend state condition
Including .BackendState == "Stopped" in the condition may affect the script's logic. Please verify that considering the "Stopped" state here is intentional and ensures correct behavior during service startup and monitoring.
|
There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days. Thank you for your contributions. |
github-actions
bot
added
the
stale
|
Not stale. |
github-actions
bot
removed
the
stale
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tailscale/DOCS.md (1)
348-358: Fix formatting according to PrettierThe pipeline indicates formatting issues. Please run Prettier to ensure consistent formatting across the documentation.
🧰 Tools
🪛 GitHub Actions: CI
[info] File was modified by Prettier formatting
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tailscale/DOCS.md(1 hunks)
🧰 Additional context used
🪛 GitHub Actions: CI
tailscale/DOCS.md
[info] File was modified by Prettier formatting
🔇 Additional comments (1)
tailscale/DOCS.md (1)
348-358: LGTM! Clear and comprehensive health check documentation.The new Healthcheck section effectively communicates:
- The automatic health check mechanism (30s intervals)
- The restart policy (3 consecutive failures)
- Specific timeout conditions (5 min offline, 1 hour startup)
The timeouts are well-balanced for real-world scenarios, where network dropouts of 2-3 minutes are common.
🧰 Tools
🪛 GitHub Actions: CI
[info] File was modified by Prettier formatting
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tailscale/DOCS.md (1)
348-359: LGTM! Consider enhancing the documentation with additional context.The health check documentation is clear and well-structured. To make it even more helpful for users:
- Consider adding context about the real-world scenario this addresses (WAN switch-over causing Tailscale to become permanently offline).
- Explain the rationale behind the timeout values (5min offline, 1hr startup).
- Clarify whether these timeouts are configurable.
Add the following paragraph after line 359 to provide more context:
- after a (re)start can't get online for longer than 1 hour +These timeout values were chosen based on real-world experiences where network +changes (e.g., WAN switch-over) could cause Tailscale to become permanently +offline. The 5-minute offline timeout accommodates typical network disruptions, +while the 1-hour startup timeout allows for extended network instability during +recovery.
lmagyar
force-pushed
the
pr-healthcheck
branch
from
7368076 to
15681cc
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tailscale/DOCS.md (2)
359-360: Fix formatting and punctuationThe bullet points need proper formatting and punctuation:
-once it was online and gets offline for longer than 5 minutes -after a (re)start can't get online for longer than 1 hour +- once it was online and gets offline for longer than 5 minutes +- after a (re)start can't get online for longer than 1 hour.🧰 Tools
🪛 LanguageTool
[uncategorized] ~360-~360: Possible missing comma found.
Context: ...for longer than 5 minutes - after a (re)start can't get online for longer than 1 hour...(AI_HYDRA_LEO_MISSING_COMMA)
[uncategorized] ~360-~360: A period might be missing here.
Context: ...tart can't get online for longer than 1 hour ## Changelog & Releases This reposito...(AI_EN_LECTOR_MISSING_PUNCTUATION_PERIOD)
352-356: Clarify health check behaviorThe documentation should clarify that the health check is performed by the add-on itself, and Home Assistant only acts on the reported health status.
Tailscale is quite resilient and can recover from nearly any network change. In case it fails to recover, the add-on's health is set unhealthy. The add-on's -health is checked by Home Assistant in each 30s, and if it reports itself 3 -times unhealthy in a row, the add-on will be restarted. +health is monitored internally and reported to Home Assistant every 30 seconds. +If the add-on reports itself as unhealthy 3 times in a row, Home Assistant will +automatically restart it.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
tailscale/DOCS.md(1 hunks)tailscale/Dockerfile(1 hunks)tailscale/rootfs/etc/s6-overlay/s6-rc.d/post-tailscaled/run(1 hunks)tailscale/rootfs/usr/bin/healthcheck(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- tailscale/rootfs/etc/s6-overlay/s6-rc.d/post-tailscaled/run
- tailscale/Dockerfile
- tailscale/rootfs/usr/bin/healthcheck
🧰 Additional context used
🪛 LanguageTool
tailscale/DOCS.md
[uncategorized] ~360-~360: Possible missing comma found.
Context: ...for longer than 5 minutes - after a (re)start can't get online for longer than 1 hour...
(AI_HYDRA_LEO_MISSING_COMMA)
[uncategorized] ~360-~360: A period might be missing here.
Context: ...tart can't get online for longer than 1 hour ## Changelog & Releases This reposito...
(AI_EN_LECTOR_MISSING_PUNCTUATION_PERIOD)
✅ Actions performedReview triggered.
|
lmagyar
force-pushed
the
pr-healthcheck
branch
from
b1d902d to
fc27de2
Compare
|
Removed the log and the docs, now this is always workign in the background. |
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
tailscale/rootfs/usr/bin/healthcheck (1)
38-46: 🛠️ Refactor suggestionAdd logging to explain why the health check failed.
When the health check fails, it should log the specific reason for better observability.
Add failure reason logging:
if [[ "${backend_state}" == "Stopped" ]] || \ (bashio::var.has_value "${LAST_ONLINE_TIMESTAMP-}" && \ (( $(date +"%s") - ${LAST_ONLINE_TIMESTAMP} > ${HEALTHCHECK_OFFLINE_TIMEOUT} )) ) || \ (! bashio::var.has_value "${LAST_ONLINE_TIMESTAMP-}" && \ (( $(date +"%s") - ${STARTED_TIMESTAMP} > ${HEALTHCHECK_RESTART_TIMEOUT} )) ) then - # Unhealthy + if [[ "${backend_state}" == "Stopped" ]]; then + bashio::log.error "Health check failed: Backend state is Stopped" + elif bashio::var.has_value "${LAST_ONLINE_TIMESTAMP-}" && \ + (( $(date +"%s") - ${LAST_ONLINE_TIMESTAMP} > ${HEALTHCHECK_OFFLINE_TIMEOUT} )); then + bashio::log.error "Health check failed: Service offline for more than ${HEALTHCHECK_OFFLINE_TIMEOUT} seconds" + else + bashio::log.error "Health check failed: Service never online for more than ${HEALTHCHECK_RESTART_TIMEOUT} seconds" + fi bashio::exit.nok fi
🧹 Nitpick comments (1)
tailscale/rootfs/usr/bin/healthcheck (1)
14-16: Consider making timeouts configurable.While the default timeout values are reasonable, consider making them configurable through environment variables or add-on options to allow users to adjust based on their network conditions.
Example implementation:
-readonly HEALTHCHECK_OFFLINE_TIMEOUT=300 # 5 minutes -readonly HEALTHCHECK_RESTART_TIMEOUT=3600 # 1 hour +readonly HEALTHCHECK_OFFLINE_TIMEOUT=${HEALTHCHECK_OFFLINE_TIMEOUT:-300} # 5 minutes +readonly HEALTHCHECK_RESTART_TIMEOUT=${HEALTHCHECK_RESTART_TIMEOUT:-3600} # 1 hour
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
tailscale/Dockerfile(1 hunks)tailscale/rootfs/etc/s6-overlay/s6-rc.d/post-tailscaled/run(1 hunks)tailscale/rootfs/usr/bin/healthcheck(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- tailscale/Dockerfile
- tailscale/rootfs/etc/s6-overlay/s6-rc.d/post-tailscaled/run
🔇 Additional comments (2)
tailscale/rootfs/usr/bin/healthcheck (2)
1-10: LGTM! Clear and comprehensive documentation.The script header provides excellent documentation of the health check conditions and rationale behind implementation choices.
11-12: LGTM! Proper log redirection for containerized environment.The output redirection ensures all health check logs are properly captured in the container logs.
| declare status_json | ||
| declare backend_state is_self_online | ||
| # STARTED_TIMESTAMP is in contenv at /var/run/s6/container_environment | ||
| # LAST_ONLINE_TIMESTAMP is in contenv at /var/run/s6/container_environment | ||
|
|
||
| if ! bashio::var.has_value "${STARTED_TIMESTAMP-}" | ||
| then | ||
| STARTED_TIMESTAMP=$(date +"%s") | ||
| printf "${STARTED_TIMESTAMP}" > /var/run/s6/container_environment/STARTED_TIMESTAMP | ||
| fi |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Fix the declare statement syntax.
The declare statement for multiple variables should have each variable quoted separately.
Apply this fix:
-declare backend_state is_self_online
+declare "backend_state" "is_self_online"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| declare status_json | |
| declare backend_state is_self_online | |
| # STARTED_TIMESTAMP is in contenv at /var/run/s6/container_environment | |
| # LAST_ONLINE_TIMESTAMP is in contenv at /var/run/s6/container_environment | |
| if ! bashio::var.has_value "${STARTED_TIMESTAMP-}" | |
| then | |
| STARTED_TIMESTAMP=$(date +"%s") | |
| printf "${STARTED_TIMESTAMP}" > /var/run/s6/container_environment/STARTED_TIMESTAMP | |
| fi | |
| declare status_json | |
| declare "backend_state" "is_self_online" | |
| # STARTED_TIMESTAMP is in contenv at /var/run/s6/container_environment | |
| # LAST_ONLINE_TIMESTAMP is in contenv at /var/run/s6/container_environment | |
| if ! bashio::var.has_value "${STARTED_TIMESTAMP-}" | |
| then | |
| STARTED_TIMESTAMP=$(date +"%s") | |
| printf "${STARTED_TIMESTAMP}" > /var/run/s6/container_environment/STARTED_TIMESTAMP | |
| fi |
| status_json=$(/opt/tailscale status --json --self=true --peers=false) | ||
| backend_state=$(jq -r '.BackendState' <<< "${status_json}") | ||
| is_self_online=$(jq -r '.Self.Online' <<< "${status_json}") | ||
|
|
||
| if bashio::var.equals "${backend_state}" "Running" && bashio::var.true "${is_self_online}" | ||
| then | ||
| LAST_ONLINE_TIMESTAMP=$(date +"%s") | ||
| printf "${LAST_ONLINE_TIMESTAMP}" > /var/run/s6/container_environment/LAST_ONLINE_TIMESTAMP | ||
| fi |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add error handling for Tailscale status and JSON parsing.
The status check could fail silently if the Tailscale command fails or returns malformed JSON.
Add error handling:
-status_json=$(/opt/tailscale status --json --self=true --peers=false)
+if ! status_json=$(/opt/tailscale status --json --self=true --peers=false); then
+ bashio::log.error "Failed to get Tailscale status"
+ bashio::exit.nok
+fi
+
+if ! backend_state=$(jq -r '.BackendState' <<< "${status_json}") || \
+ ! is_self_online=$(jq -r '.Self.Online' <<< "${status_json}"); then
+ bashio::log.error "Failed to parse Tailscale status JSON"
+ bashio::exit.nok
+fiCommittable suggestion skipped: line range outside the PR's diff.
Proposed Changes
This PR treats the following situations unhealthy:
In my case a plain WAN switch-over to backup WAN on the router was able to permanently make Tailscale offline. And it's not only me: tailscale/tailscale#12021 It seems to be an issue when TS is in a container. And similar bugs can happen in the future.
And after testing it in real life circumstances, TS was able to not go online even after a restart, if the transient in the WAN state was long enough. But after (re)start, we should restart the add-on again much slower, eg. to give time to authenticate on the very first time.
Note: I was able to put TS into Stopped state only manually, but maybe this can be helpful once, at least won't harm.
Note: This schema (below) would look nicer, but until a bug in the Supervisor/UI is not fixed (see #2640), we can't do this.
Related Issues
Summary by CodeRabbit