Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection issues with Uptime Kuma monitors after server restart #5443

Open
1 task done
homelab-alpha opened this issue Dec 16, 2024 · 4 comments
Open
1 task done
Labels
bug Something isn't working
Milestone

Comments

@homelab-alpha
Copy link
Contributor

homelab-alpha commented Dec 16, 2024

πŸ“‘ I have found these related issues/pull requests

I have checked, but couldn’t find anything similar.

πŸ›‘οΈ Security Policy

Description

After restarting the server (Ubuntu Server 24.04.1 LTS), Uptime Kuma monitors fail to reconnect to services like Docker and MariaDB. The logs indicate a PROTOCOL_CONNECTION_LOST error, pointing to connection instability with the database and Docker monitors.

However, as shown in the logs, Uptime Kuma starts successfully despite the trace error.

πŸ‘Ÿ Reproduction Steps

  1. Open the terminal and create the directory structure and files:

    This command creates the necessary directories and empty files (docker-compose.yml and my.cnf) for the Uptime Kuma setup.

    mkdir -p /docker/uptime-kuma-beta && touch /docker/uptime-kuma-beta/docker-compose.yml /docker/uptime-kuma-beta/my.cnf
  2. Edit the docker-compose.yml file.

    Go to the configuration file below, called here is the docker-compose.yml file (click to expand), copy the contents of the docker-compose.yml file, and paste it into the file using the nano editor.

    sudo nano /docker/uptime-kuma-beta/docker-compose.yml
  3. Edit the my.cnf file.

    Go to the configuration file below, called here is the my.cnf file (click to expand), copy the contents of the my.cnf file, and paste it into the file using the nano editor.

    sudo nano /docker/uptime-kuma-beta/my.cnf
  4. Bring the stack online with the following command:

    This command starts the Uptime Kuma stack using Docker Compose, runs it in the background (--detach), and removes orphaned containers (--remove-orphans).

    sudo docker compose --file /docker/uptime-kuma-beta/docker-compose.yml up --detach --remove-orphans
  5. Configure Uptime Kuma and create 5 monitors of different types (e.g., DNS, HTTPS, MySQL).

    This step involves setting up Uptime Kuma and creating the necessary monitors to check the uptime of various services. To access the Uptime Kuma web interface, open a browser and navigate to:

    http://server-ip-address:3001

    After completing the setup and creating the necessary monitors, you can proceed to Step 6.

  6. Restart the server (Ubuntu Server 24.04.1 LTS).

    Note: Ensure that any important work is saved before running this command, as it will reboot the server immediately.

    sudo shutdown --reboot now
  7. Observe the logs generated by Uptime Kuma and MariaDB before and after the server restart.

    NOTE: For an easier way to view logs from your containers, use Dozzle, a convenient web-based log viewer for Docker containers.

    You can monitor the logs of the Uptime Kuma container and MariaDB container to see how they behave before and after the server restart.

    sudo docker logs --follow --details uptime-kuma-beta
    sudo docker logs --follow --details uptime-kuma-beta_db

    You can view both logs in the CLI simultaneously, but it may be difficult to follow. To monitor both logs at the same time, run the following:

    sudo docker logs --follow --details uptime-kuma-beta & sudo docker logs --follow --details uptime-kuma-beta_db
  8. After testing, take the stack offline using the following command:

    This command will stop and remove the containers, cleaning up the Docker environment while keeping the configuration files intact.

    sudo docker compose --file /docker/uptime-kuma-beta/docker-compose.yml down
  9. Cleanup: After completing your testing and confirming that everything is done, you can clean up the directory.

    Remove the directory and all contents of the Uptime Kuma stack to finalize the cleanup process.

    Note: This command will delete the /docker/uptime-kuma-beta directory along with all files and subdirectories inside it, including the docker-compose.yml and my.cnf files. Ensure you don't lose any important data before executing this step.

    sudo rm -r /docker/uptime-kuma-beta

βš™οΈ Config files

Here is the docker-compose.yml file (click to expand)

---
networks:
  uptime-kuma-beta_net:
    attachable: false
    internal: false
    external: false
    name: uptime-kuma-beta
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.19.3.0/24
          ip_range: 172.19.3.0/24
          gateway: 172.19.3.1
    driver_opts:
      com.docker.network.bridge.default_bridge: "false"
      com.docker.network.bridge.enable_icc: "true"
      com.docker.network.bridge.enable_ip_masquerade: "true"
      com.docker.network.bridge.host_binding_ipv4: "0.0.0.0"
      com.docker.network.bridge.name: "uptime-kuma"
      com.docker.network.driver.mtu: "1500"
    labels:
      com.uptime-kuma-beta.network.description: "is an isolated bridge network."

services:
  uptime-kuma-beta_db:
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "1M"
        max-file: "2"
    stop_grace_period: 1m
    container_name: uptime-kuma-beta_db
    image: mariadb:latest
    pull_policy: if_not_present
    volumes:
      - /docker/uptime-kuma-beta/db:/var/lib/mysql
      - /docker/uptime-kuma-beta/my.cnf:/etc/my.cnf
    environment:
      PUID: "1000"
      PGID: "1000"
      TZ: Europe/Amsterdam
      MYSQL_ROOT_PASSWORD: "ThisIsADemoRootPassword"
      MYSQL_DATABASE: "uptime_kuma_db"
      MYSQL_USER: "uptime-kuma"
      MYSQL_PASSWORD: "ThisIsADemoPassword"
    hostname: uptime-kuma-beta_db
    networks:
      uptime-kuma-beta_net:
        ipv4_address: 172.19.3.2
    security_opt:
      - no-new-privileges:true
    labels:
      com.uptime-kuma-beta.db.description: "is an MySQL database."
    healthcheck:
      disable: false
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 10s
      start_interval: 5s

  uptime-kuma-beta_app:
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "1M"
        max-file: "2"
    stop_grace_period: 1m
    container_name: uptime-kuma-beta
    image: louislam/uptime-kuma:nightly2
    pull_policy: if_not_present
    depends_on:
      uptime-kuma-beta_db:
        condition: service_healthy
    links:
      - uptime-kuma-beta_db
    volumes:
      - /docker/uptime-kuma-beta/app:/app/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /usr/local/share/ca-certificates:/app/data/docker-tls
    environment:
      PUID: "1000"
      PGID: "1000"
      TZ: Europe/Amsterdam
      # This is a certificate authority for demonstration purposes.
      # NODE_EXTRA_CA_CERTS: /app/data/docker-tls/root_ca_demo_cert.pem
      MYSQL_HOST: "uptime-kuma-beta_db"
      MYSQL_PORT: 3306
      MYSQL_NAME: "uptime_kuma_db"
      MYSQL_USER: "uptime-kuma"
      MYSQL_PASSWORD: "ThisIsADemoPassword"
    domainname: status.local
    hostname: status
    networks:
      uptime-kuma-beta_net:
        ipv4_address: 172.19.3.3
    ports:
      - "3001:3001/tcp" # HTTP
      - "3001:3001/udp" # HTTP
    security_opt:
      - no-new-privileges:true
    labels:
      com.docker.compose.project: "uptime-kuma-beta"
      com.uptime-kuma-beta.description:
        "is an self-hosted monitoring tool that allows you to monitor uptime,
        status, and notifications for various services and domains."
    healthcheck:
      disable: false
      test: ["CMD", "extra/healthcheck"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 10s
      start_interval: 5s

Here is the my.cnf file (click to expand)

#
# The MariaDB/MySQL tools read configuration files in the following order:
# 0. "/etc/mysql/my.cnf" symlinks to this file, reason why all the rest is read.
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# If you are new to MariaDB, check out https://mariadb.com/kb/en/basic-mariadb-articles/

#
# This group is read both by the client and the server
# use it for options that affect everything
#
[client-server]
# Port or socket location where to connect
# port = 3306
socket = /run/mysqld/mysqld.sock

[mysqld]
# ============================================
# General Server Settings
# ============================================

# Unique server ID for replication. Each server in a replication setup
# must have a unique ID. Default is 0, which is invalid for replication.
server-id = 1

# Path to the database files. This is where all database data will be stored.
datadir = /var/lib/mysql/

# Location of the PID file. This file contains the process ID of the running MySQL server.
pid-file = /var/run/mysqld/mysqld.pid

# Disable DNS lookups when clients connect to the server, improving connection speed.
# Note: With this setting enabled, only IP addresses will be logged.
skip-name-resolve

# Set the default storage engine for new tables. InnoDB is generally recommended
# for its ACID compliance and support for transactions.
default-storage-engine = InnoDB

# Define the transaction isolation level. REPEATABLE-READ is a common default,
# ensuring consistent reads within a transaction.
transaction-isolation = REPEATABLE-READ

# Prevent the use of symbolic links to ensure data security.
skip-symbolic-links

# ============================================
# Performance Optimizations
# ============================================

# Enable native asynchronous I/O for improved performance in InnoDB.
innodb-use-native-aio = 0

# Set the size of the InnoDB buffer pool. This should be 70-80% of the available memory.
# Larger values allow more data to be cached in memory, improving performance.
innodb-buffer-pool-size = 2G

# Configure the size of the InnoDB redo log files. Larger log files can improve
# performance for write-heavy workloads but require more recovery time after a crash.
innodb-log-file-size = 1G

# Define the size of the InnoDB log buffer. This buffer holds transaction logs
# in memory before they are written to disk. Larger buffers reduce disk I/O.
innodb-log-buffer-size = 32M

# Flush logs to disk after each transaction. This ensures ACID compliance,
# but may reduce performance. Setting this to 2 can improve performance at the
# cost of potential data loss during crashes.
innodb-flush-log-at-trx-commit = 1

# Number of background threads for read and write operations in InnoDB.
# Increase these values for high I/O workloads.
innodb-read-io-threads = 8
innodb-write-io-threads = 8

# Configure the I/O capacity for background operations such as flushing.
# Set this based on your disk's capabilities.
innodb-io-capacity = 6000
innodb-io-capacity-max = 8000

# Disable query caching as it is not beneficial for most modern workloads.
# Consider enabling it only if your application benefits from repeated identical queries.
query-cache-type = 0

# Set the maximum size for temporary tables stored in memory.
tmp-table-size = 64M

# Configure the maximum size for internal temporary tables stored in memory.
max-heap-table-size = 64M

# Define the maximum number of simultaneous client connections. Adjust based
# on your application's concurrency requirements.
max-connections = 200

# Number of threads to cache for reuse. Higher values reduce the overhead of
# creating new threads for each connection.
thread-cache-size = 50

# Configure the cache size for table definitions and open tables. Larger caches
# improve performance for workloads with many tables.
table-definition-cache = 4000
table-open-cache = 4000

# Set the buffer size for the Aria storage engine, used in MariaDB-specific workloads.
aria-pagecache-buffer-size = 128M

# ============================================
# Security Settings
# ============================================

# Restrict file imports and exports to a secure directory to prevent unauthorized access.
secure-file-priv = /var/lib/mysql/

# Load the password validation plugin to enforce strong password policies.
# plugin-load-add = validate_password.so

# Enforce a strong password policy with specific requirements.
# validate-password-policy = STRONG

# Minimum password length for increased security.
# validate-password-length = 12

# Require at least one uppercase letter in passwords.
# validate-password-mixed-case-count = 1

# Require at least one numeric character in passwords.
# validate-password-number-count = 1

# Require at least one special character in passwords.
# validate-password-special-char-count = 1

# ============================================
# Binary Logging and Replication
# ============================================

# Enable binary logging for replication and point-in-time recovery.
# The log file will be named 'uptime-kuma-beta_binlog'. You can specify a different name if desired.
log-bin = uptime-kuma-beta_binlog

# Set the maximum size for each binary log file. Once the file size reaches this limit,
# a new binary log file is created. A value of 0 means there is no size limit for the binary logs.
max-binlog-size = 500M

# Define the number of days after which binary logs will be automatically purged.
# This helps prevent the binary log files from growing indefinitely.
expire-logs-days = 14

# Enable checksums for binary logs to ensure data integrity.
binlog-checksum = CRC32

# Configure the format for binary logging. ROW-based replication is recommended
# for ensuring data consistency.
binlog-format = ROW

# Enable compression for binary logs to reduce disk usage.
log-bin-compress = 1

# Disable binary log encryption for better performance. Enable it if data security
# requires encrypted logs.
encrypt-binlog = 0

# Automatically purge relay logs to save disk space.
relay-log-purge = 1

# Enable relay log recovery to ensure consistency during replication recovery.
relay-log-recovery = 1

# Configure whether replication slave connections are required before purging logs.
slave_connections_needed_for_purge = 0

# ============================================
# Character Set and Encoding
# ============================================

# Set the server's default character set to UTF-8 with full Unicode support.
character-set-server = utf8mb4

# Set the default collation for the server to match the UTF-8 character set.
collation-server = utf8mb4_general_ci

# Ensure proper encoding when clients connect to the server.
init-connect = 'SET NAMES utf8mb4'

# ============================================
# Monitoring and Debugging
# ============================================

# Enable the Performance Schema for detailed diagnostics.
performance-schema = 1

# Enable specific consumers in the Performance Schema for tracking events.
performance-schema-consumer-events-statements-history = 1
performance-schema-consumer-events-transactions-history = 1
performance-schema-consumer-events-waits-history = 1

# Enable detailed InnoDB status output for monitoring locks and transactions.
innodb-status-output = 0
innodb-status-output-locks = 0

# Disable the general query log by default to avoid excessive disk usage.
# Enable it only for debugging purposes.
general-log = 0

# ============================================
# Temporary Files
# ============================================

# Specify a directory for temporary files. Storing temporary files on an SSD
# can improve performance.
# tmpdir = /tmp

# ============================================
# SSL/TLS Security
# ============================================

# SSL/TLS-instellingen
# ssl-ca = /etc/mysql/ssl/ca-cert.pem
# ssl-cert = /etc/mysql/ssl/server-cert.pem
# ssl-key = /etc/mysql/ssl/server-key.pem
# require-secure-transport = 1


# ============================================
# Inclusion of Additional Configuration Files
# ============================================

# Import all .cnf files from the specified configuration directories.
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

πŸ‘€ Expected behavior

Monitors should reconnect to all configured services after a server restart.

πŸ˜“ Actual Behavior

Monitors are unable to connect to MariaDB and Docker services. Logs show
PROTOCOL_CONNECTION_LOST after auto retry, which seems to fix the issue.

🐻 Uptime-Kuma Version

2.0.0-beta.0

πŸ’» Operating System and Arch

Ubuntu Server 24.04.1 LTS (GNU/Linux 6.8.0-48-generic x86_64)

🌐 Browser

Brave Version 1.73.101 Chromium: 131.0.6778.139 (Official Build) (64-bit)

πŸ–₯️ Deployment Environment

  • Runtime:
    • Docker version 26.1.0, build 9714adc
    • Docker Compose version v2.26.1
    • Portainer Business Edition version 2.21.4
    • MariaDB 11.6.2
  • Database: MariaDB (external)
  • Filesystem used for the database: Linux/ext4 on an SSD
  • Number of monitors: 92

πŸ“ Relevant log output

Logs show errors related to connection loss. Examples:

Here is the logs from Dec 16, 2024 file (click to expand)

2024-12-16T06:00:00+01:00 [SERVER] INFO: Shutdown requested
2024-12-16T06:00:00+01:00 [SERVER] INFO: Called signal: SIGTERM
2024-12-16T06:00:00+01:00 [MONITOR] WARN: Monitor #20 'Dozzle Container': Pending:
            SELECT maintenance_id FROM monitor_maintenance
            WHERE monitor_id = 20
         - Connection lost: The server closed the connection. | Max retries: 3 | Retry: 1 | Retry Interval: 20 seconds | Type: docker
2024-12-16T06:00:00+01:00 [MONITOR] WARN: Monitor #86 'Uptime Kuma MariaDB': Pending:
            SELECT maintenance_id FROM monitor_maintenance
            WHERE monitor_id = 86
         - Connection lost: The server closed the connection. | Max retries: 3 | Retry: 1 | Retry Interval: 20 seconds | Type: mysql
2024-12-16T06:00:00+01:00 [MONITOR] WARN: Monitor #85 'Uptime Kuma DB Container': Pending:
            SELECT maintenance_id FROM monitor_maintenance
            WHERE monitor_id = 85
         - Connection lost: The server closed the connection. | Max retries: 3 | Retry: 1 | Retry Interval: 20 seconds | Type: docker
Trace: Error: insert into `heartbeat` (`down_count`, `end_time`, `important`, `monitor_id`, `msg`, `retries`, `status`, `time`) values (0, '2024-12-16 05:00:00.242', false, 53, 'connect ECONNREFUSED 192.168.**.**:443', 1, 2, '2024-12-16 05:00:00.189') - Connection lost: The server closed the connection.
    at Socket.<anonymous> (/app/node_modules/mysql2/lib/connection.js:117:31)
    at Socket.emit (node:events:519:28)
    at TCP.<anonymous> (node:net:339:12) {
  fatal: true,
  code: 'PROTOCOL_CONNECTION_LOST'
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:1170:17)
2024-12-16T06:00:00+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2024-12-16T06:00:00+01:00 [MONITOR] INFO: Try to restart the monitor
Trace: Error: insert into `stat_minutely` (`down`, `monitor_id`, `ping`, `ping_max`, `ping_min`, `timestamp`, `up`) values (1, 19, 0, 0, 0, 1734325200, 0) - Connection lost: The server closed the connection.
    at Socket.<anonymous> (/app/node_modules/mysql2/lib/connection.js:117:31)
    at Socket.emit (node:events:519:28)
    at TCP.<anonymous> (node:net:339:12) {
  fatal: true,
  code: 'PROTOCOL_CONNECTION_LOST'
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:1170:17)
2024-12-16T06:00:00+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2024-12-16T06:00:00+01:00 [MONITOR] INFO: Try to restart the monitor
Welcome to Uptime Kuma
Your Node.js version: 20.18.0
2024-12-16T06:00:45+01:00 [SERVER] INFO: Env: production
2024-12-16T06:00:50+01:00 [SERVER] INFO: Uptime Kuma Version: 2.0.0-beta.0
2024-12-16T06:00:50+01:00 [SERVER] INFO: Loading modules
2024-12-16T06:00:54+01:00 [SERVER] INFO: Creating express and socket.io instance
2024-12-16T06:00:54+01:00 [SERVER] INFO: Server Type: HTTP
2024-12-16T06:00:54+01:00 [SERVER] INFO: Data Dir: ./data/
2024-12-16T06:00:54+01:00 [DB] INFO: Database Type: mariadb
2024-12-16T06:00:54+01:00 [SERVER] INFO: Connected to the database
2024-12-16T06:00:55+01:00 [SERVICES] INFO: Starting nscd
2024-12-16T06:00:55+01:00 [SERVER] INFO: Listening on 3001

Here is the logs from Dec 17, 2024 file (click to expand)

Connection Error: Error: Connection lost: The server closed the connection.
Connection Error: Error: Connection lost: The server closed the connection.
2024-12-17T06:00:00+01:00 [MONITOR] WARN: Monitor #38 'kWh meter 1-fase MID': Pending: connect ECONNREFUSED 172.20.3.2:3306 | Max retries: 3 | Retry: 1 | Retry Interval: 20 seconds | Type: ping
Trace: Error: connect ECONNREFUSED 172.20.3.2:3306
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '172.20.3.2',
  port: 3306,
  fatal: true
}
    at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:1170:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-12-17T06:00:00+01:00 [MONITOR] INFO: Try to restart the monitor
2024-12-17T06:00:00+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2024-12-17T06:00:00+01:00 [SERVER] INFO: Shutdown requested
2024-12-17T06:00:00+01:00 [SERVER] INFO: Called signal: SIGTERM

@homelab-alpha homelab-alpha added the bug Something isn't working label Dec 16, 2024
@IV-R
Copy link

IV-R commented Dec 18, 2024

The issue you’ve referenced, #5443, describes connection problems with Uptime Kuma monitors following a server restart. To address this and ensure efficient resolution of similar issues, consider the following steps:

  1. Implement Robust Service Dependencies:
    β€’ Service Initialization Order: Ensure that dependent services (e.g., databases, Docker) are fully operational before Uptime Kuma starts. This can be managed by configuring service dependencies in your system’s init system (e.g., systemd) or by implementing startup scripts that verify the availability of required services before launching Uptime Kuma.
    β€’ Automatic Retry Mechanisms: Incorporate retry logic within Uptime Kuma to handle transient connection failures gracefully. This ensures that temporary unavailability of services during startup doesn’t lead to persistent monitoring issues.

  2. Enhance Logging and Monitoring:
    β€’ Detailed Logging: Enable verbose logging to capture detailed information about connection attempts and failures. This aids in diagnosing issues related to service dependencies and network stability.
    β€’ Health Checks: Implement health checks for critical services to monitor their status continuously. This allows for proactive detection and resolution of issues before they impact Uptime Kuma’s monitoring capabilities.

  3. Update and Patch Regularly:
    β€’ Software Updates: Keep Uptime Kuma and its dependencies up to date to benefit from the latest features, bug fixes, and security patches. Regular updates can resolve known issues that may affect service stability.
    β€’ Community Engagement: Monitor the Uptime Kuma GitHub repository for reported issues and participate in discussions. Engaging with the community can provide insights into common problems and their solutions.

  4. Optimize System Resources:
    β€’ Resource Allocation: Ensure that the server hosting Uptime Kuma has sufficient resources (CPU, memory, disk I/O) to handle the monitoring workload. Resource constraints can lead to service instability and connection issues.
    β€’ Performance Tuning: Optimize system and network configurations to reduce latency and improve the reliability of connections between Uptime Kuma and the services it monitors.

  5. Implement Redundancy and Failover Strategies:
    β€’ High Availability: Deploy Uptime Kuma in a high-availability configuration to prevent single points of failure. This may involve setting up multiple instances with load balancing and failover mechanisms.
    β€’ Backup and Recovery: Regularly back up Uptime Kuma configurations and data to facilitate quick recovery in case of failures. Implement automated backup routines and test recovery procedures periodically.

By following these steps, you can enhance the resilience of Uptime Kuma and mitigate connection issues that arise after server restarts, ensuring continuous and reliable monitoring of your services.

@homelab-alpha
Copy link
Contributor Author

@IV-R, Quick question: Have you looked at the Docker Compose file?

Health checks are applicable for both services. The issue occurs sometimes after a β€œShutdown requested” from Uptime Kuma and occasionally after the server restarts. Before Uptime Kuma can start, it must wait until the MariaDB container is in a service_healthy state. If this is not the case, the Uptime Kuma container will not start.

@homelab-alpha
Copy link
Contributor Author

@louislam, @CommanderStorm, I’ve updated the issue and I think I’ve found the problem. It seems to be related to how I set up the MariaDB configuration in the docker-compose file.

I’ve now added a complete configuration file for MariaDB using my.cnf. Hopefully, this was the issue. I’ll keep you updated.

@homelab-alpha
Copy link
Contributor Author

@louislam,

UPDATE: After upgrading the container to 2.0.0-beta.1 and adjusting the MariaDB configuration in the docker-compose.yml file, most of the issues I previously encountered have been resolved. However, I am now experiencing a Trace Error when rebooting the container.

This error occurs during the [DB] INFO: Closing the database step. Interestingly, it is not consistentβ€”sometimes it happens multiple times in a row after several reboots, while other times it does not occur at all.

Do you know what might be causing this Trace Error during the database closure process?

adjustments (click to expand)

From

command: [
    "--log-bin=uptime-kuma_binlog", # Enable binary logging with the name 'uptime-kuma_binlog'
    "--max-binlog-size=100M", # Set maximum binary log file size
    "--expire-logs-days=14", # Set binary log expiration to 14 days
    "--relay-log-purge=1", # Automatically purge relay logs
    "--relay-log-recovery=1", # Enable relay log recovery
    "--slave-connections-needed-for-purge=0", # Slave connection not required for purging logs
    "--transaction-isolation=REPEATABLE-READ", # Set default transaction isolation level
    "--binlog-checksum=1", # Enable binary log checksum
    "--binlog-format=ROW", # Enable row-based binary logging
    "--log-bin-compress=1", # Enable binary log compression
    "--encrypt-binlog=0", # Disable binary log encryption
    "--innodb-use-native-aio=0", # Disable native AIO support for InnoDB
    "--innodb-status-output=0", # Disable InnoDB status output
  ]

To

volumes:
  - /docker/uptime-kuma-beta/my.cnf:/etc/my.cnf

my.cnf (click to expand)

#
# The MariaDB/MySQL tools read configuration files in the following order:
# 0. "/etc/mysql/my.cnf" symlinks to this file, reason why all the rest is read.
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# If you are new to MariaDB, check out https://mariadb.com/kb/en/basic-mariadb-articles/

#
# This group is read both by the client and the server
# use it for options that affect everything
#
[client-server]
# Port or socket location where to connect
port = 3306
socket = /run/mysqld/mysqld.sock

[mysqld]
# ============================================
# General Server Settings
# ============================================

# Unique server ID for replication. Each server in a replication setup
# must have a unique ID. Default is 0, which is invalid for replication.
server-id = 1

# Path to the database files. This is where all database data will be stored.
datadir = /var/lib/mysql/

# Location of the PID file. This file contains the process ID of the running MySQL server.
pid-file = /var/run/mysqld/mysqld.pid

# Disable DNS lookups when clients connect to the server, improving connection speed.
# Note: With this setting enabled, only IP addresses will be logged.
skip-name-resolve

# Set the default storage engine for new tables. InnoDB is generally recommended
# for its ACID compliance and support for transactions.
default-storage-engine = InnoDB

# Define the transaction isolation level. REPEATABLE-READ is a common default,
# ensuring consistent reads within a transaction.
transaction-isolation = REPEATABLE-READ

# Prevent the use of symbolic links to ensure data security.
skip-symbolic-links

# ============================================
# Performance Optimizations
# ============================================

# Enable native asynchronous I/O for improved performance in InnoDB.
innodb-use-native-aio = 0

# Set the size of the InnoDB buffer pool. This should be 70-80% of the available memory.
# Larger values allow more data to be cached in memory, improving performance.
innodb-buffer-pool-size = 2G

# Configure the size of the InnoDB redo log files. Larger log files can improve
# performance for write-heavy workloads but require more recovery time after a crash.
innodb-log-file-size = 1G

# Define the size of the InnoDB log buffer. This buffer holds transaction logs
# in memory before they are written to disk. Larger buffers reduce disk I/O.
innodb-log-buffer-size = 32M

# Flush logs to disk after each transaction. This ensures ACID compliance,
# but may reduce performance. Setting this to 2 can improve performance at the
# cost of potential data loss during crashes.
innodb-flush-log-at-trx-commit = 1

# Number of background threads for read and write operations in InnoDB.
# Increase these values for high I/O workloads.
innodb-read-io-threads = 8
innodb-write-io-threads = 8

# Configure the I/O capacity for background operations such as flushing.
# Set this based on your disk's capabilities.
innodb-io-capacity = 6000
innodb-io-capacity-max = 8000

# Disable query caching as it is not beneficial for most modern workloads.
# Consider enabling it only if your application benefits from repeated identical queries.
query-cache-type = 0

# Set the maximum size for temporary tables stored in memory.
tmp-table-size = 64M

# Configure the maximum size for internal temporary tables stored in memory.
max-heap-table-size = 64M

# Define the maximum number of simultaneous client connections. Adjust based
# on your application's concurrency requirements.
max-connections = 200

# Number of threads to cache for reuse. Higher values reduce the overhead of
# creating new threads for each connection.
thread-cache-size = 50

# Configure the cache size for table definitions and open tables. Larger caches
# improve performance for workloads with many tables.
table-definition-cache = 4000
table-open-cache = 4000

# Set the buffer size for the Aria storage engine, used in MariaDB-specific workloads.
aria-pagecache-buffer-size = 128M

# ============================================
# Security Settings
# ============================================

# Restrict file imports and exports to a secure directory to prevent unauthorized access.
secure-file-priv = /var/lib/mysql/

# Load the password validation plugin to enforce strong password policies.
# plugin-load-add = validate_password.so

# Enforce a strong password policy with specific requirements.
# validate-password-policy = STRONG

# Minimum password length for increased security.
# validate-password-length = 12

# Require at least one uppercase letter in passwords.
# validate-password-mixed-case-count = 1

# Require at least one numeric character in passwords.
# validate-password-number-count = 1

# Require at least one special character in passwords.
# validate-password-special-char-count = 1

# ============================================
# Binary Logging and Replication
# ============================================

# Enable binary logging for replication and point-in-time recovery.
# The log file will be named 'uptime-kuma-beta_binlog'. You can specify a different name if desired.
log-bin = uptime-kuma-beta_binlog

# Set the maximum size for each binary log file. Once the file size reaches this limit,
# a new binary log file is created. A value of 0 means there is no size limit for the binary logs.
max-binlog-size = 500M

# Define the number of days after which binary logs will be automatically purged.
# This helps prevent the binary log files from growing indefinitely.
expire-logs-days = 14

# Enable checksums for binary logs to ensure data integrity.
binlog-checksum = CRC32

# Configure the format for binary logging. ROW-based replication is recommended
# for ensuring data consistency.
binlog-format = ROW

# Enable compression for binary logs to reduce disk usage.
log-bin-compress = 1

# Disable binary log encryption for better performance. Enable it if data security
# requires encrypted logs.
encrypt-binlog = 0

# Automatically purge relay logs to save disk space.
relay-log-purge = 1

# Enable relay log recovery to ensure consistency during replication recovery.
relay-log-recovery = 1

# Configure whether replication slave connections are required before purging logs.
slave_connections_needed_for_purge = 0

# ============================================
# Character Set and Encoding
# ============================================

# Set the server's default character set to UTF-8 with full Unicode support.
character-set-server = utf8mb4

# Set the default collation for the server to match the UTF-8 character set.
collation-server = utf8mb4_general_ci

# Ensure proper encoding when clients connect to the server.
init-connect = 'SET NAMES utf8mb4'

# ============================================
# Monitoring and Debugging
# ============================================

# Enable the Performance Schema for detailed diagnostics.
performance-schema = 1

# Enable specific consumers in the Performance Schema for tracking events.
performance-schema-consumer-events-statements-history = 1
performance-schema-consumer-events-transactions-history = 1
performance-schema-consumer-events-waits-history = 1

# Enable detailed InnoDB status output for monitoring locks and transactions.
innodb-status-output = 0
innodb-status-output-locks = 0

# Disable the general query log by default to avoid excessive disk usage.
# Enable it only for debugging purposes.
general-log = 0

# ============================================
# Temporary Files
# ============================================

# Specify a directory for temporary files. Storing temporary files on an SSD
# can improve performance.
# tmpdir = /tmp

# ============================================
# SSL/TLS Security
# ============================================

# SSL/TLS-instellingen
# ssl-ca = /etc/mysql/ssl/ca-cert.pem
# ssl-cert = /etc/mysql/ssl/server-cert.pem
# ssl-key = /etc/mysql/ssl/server-key.pem
# require-secure-transport = 1


# ============================================
# Inclusion of Additional Configuration Files
# ============================================

# Import all .cnf files from the specified configuration directories.
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

comand + log (click to expand)

Reboot command:

β”Œβ”€β”€(homelab-alpha@test-server)-[~]
└─$ sudo docker restart uptime-kuma-beta

Output log:

2024-12-21T12:07:45+01:00 [SERVER] INFO: Shutdown requested
2024-12-21T12:07:45+01:00 [SERVER] INFO: Called signal: SIGTERM
2024-12-21T12:07:45+01:00 [SERVICES] INFO: Stopping nscd
2024-12-21T12:07:45+01:00 [SERVER] INFO: Stopping all monitors
2024-12-21T12:07:47+01:00 [DB] INFO: Closing the database
2024-12-21T12:07:49+01:00 [DB] INFO: Database closed
2024-12-21T12:07:49+01:00 [CLOUDFLARED] INFO: Stop cloudflared
2024-12-21T12:07:49+01:00 [SERVER] INFO: Graceful shutdown successful!
Welcome to Uptime Kuma
Your Node.js version: 20.18.0
2024-12-21T12:16:49+01:00 [SERVER] INFO: Env: production
2024-12-21T12:16:50+01:00 [SERVER] INFO: Uptime Kuma Version: 2.0.0-beta.1
2024-12-21T12:16:50+01:00 [SERVER] INFO: Loading modules
2024-12-21T12:16:51+01:00 [SERVER] INFO: Creating express and socket.io instance
2024-12-21T12:16:51+01:00 [SERVER] INFO: Server Type: HTTP
2024-12-21T12:16:51+01:00 [SERVER] INFO: Data Dir: ./data/
2024-12-21T12:16:51+01:00 [DB] INFO: Database Type: mariadb
2024-12-21T12:16:51+01:00 [SERVER] INFO: Connected to the database
2024-12-21T12:16:51+01:00 [SERVICES] INFO: Starting nscd
2024-12-21T12:16:51+01:00 [SERVER] INFO: Listening on 3001
2024-12-21T12:17:25+01:00 [SOCKET] INFO: New polling connection, IP = 192.168.**.**

Reboot command:

β”Œβ”€β”€(homelab-alpha@test-server)-[~]
└─$ sudo docker restart uptime-kuma-beta

Output log:

2024-12-21T12:17:36+01:00 [SERVER] INFO: Shutdown requested
2024-12-21T12:17:36+01:00 [SERVER] INFO: Called signal: SIGTERM
2024-12-21T12:17:36+01:00 [SERVICES] INFO: Stopping nscd
2024-12-21T12:17:36+01:00 [SERVER] INFO: Stopping all monitors
2024-12-21T12:17:36+01:00 [MONITOR] INFO: [UniFi Switch 8 PoE (01)] isStop = true, no next check.
2024-12-21T12:17:37+01:00 [MONITOR] INFO: [UniFi Switch 8 PoE (01) SSH] isStop = true, no next check.
2024-12-21T12:17:37+01:00 [MONITOR] INFO: [UniFi Switch 8 PoE (02)] isStop = true, no next check.
2024-12-21T12:17:38+01:00 [DB] INFO: Closing the database
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:17:38+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:17:38+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:17:39+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:17:40+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2024-12-21T12:17:40+01:00 [DB] INFO: Database closed
2024-12-21T12:17:40+01:00 [CLOUDFLARED] INFO: Stop cloudflared
2024-12-21T12:17:40+01:00 [SERVER] INFO: Graceful shutdown successful!
Welcome to Uptime Kuma
Your Node.js version: 20.18.0
2024-12-21T12:17:41+01:00 [SERVER] INFO: Env: production
2024-12-21T12:17:42+01:00 [SERVER] INFO: Uptime Kuma Version: 2.0.0-beta.1
2024-12-21T12:17:42+01:00 [SERVER] INFO: Loading modules
2024-12-21T12:17:42+01:00 [SERVER] INFO: Creating express and socket.io instance
2024-12-21T12:17:42+01:00 [SERVER] INFO: Server Type: HTTP
2024-12-21T12:17:43+01:00 [SERVER] INFO: Data Dir: ./data/
2024-12-21T12:17:43+01:00 [DB] INFO: Database Type: mariadb
2024-12-21T12:17:43+01:00 [SERVER] INFO: Connected to the database
2024-12-21T12:17:43+01:00 [SERVICES] INFO: Starting nscd
2024-12-21T12:17:43+01:00 [SERVER] INFO: Listening on 3001
2024-12-21T12:17:49+01:00 [SOCKET] INFO: New polling connection, IP = 192.168.**.**
2024-12-21T12:17:51+01:00 [SOCKET] INFO: New polling connection, IP = 192.168.**.**

Reboot command:

β”Œβ”€β”€(homelab-alpha@test-server)-[~]
└─$ sudo docker restart uptime-kuma-beta

Output log:

2024-12-21T12:18:01+01:00 [SERVER] INFO: Shutdown requested
2024-12-21T12:18:01+01:00 [SERVER] INFO: Called signal: SIGTERM
2024-12-21T12:18:01+01:00 [SERVICES] INFO: Stopping nscd
2024-12-21T12:18:01+01:00 [SERVER] INFO: Stopping all monitors
2024-12-21T12:18:02+01:00 [MONITOR] INFO: [Home Assistant Portal] isStop = true, no next check.
2024-12-21T12:18:02+01:00 [MONITOR] INFO: [Home Assistant Server] isStop = true, no next check.
2024-12-21T12:18:03+01:00 [MONITOR] INFO: [Home Assistant Server SSH] isStop = true, no next check.
2024-12-21T12:18:03+01:00 [MONITOR] INFO: [homelab-alpha.nl] isStop = true, no next check.
2024-12-21T12:18:03+01:00 [DB] INFO: Closing the database
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:18:04+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:18:05+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:18:05+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2024-12-21T12:18:05+01:00 [DB] INFO: Database closed
2024-12-21T12:18:05+01:00 [CLOUDFLARED] INFO: Stop cloudflared
2024-12-21T12:18:05+01:00 [SERVER] INFO: Graceful shutdown successful!
Welcome to Uptime Kuma
Your Node.js version: 20.18.0
2024-12-21T12:18:06+01:00 [SERVER] INFO: Env: production
2024-12-21T12:18:07+01:00 [SERVER] INFO: Uptime Kuma Version: 2.0.0-beta.1
2024-12-21T12:18:07+01:00 [SERVER] INFO: Loading modules
2024-12-21T12:18:08+01:00 [SERVER] INFO: Creating express and socket.io instance
2024-12-21T12:18:08+01:00 [SERVER] INFO: Server Type: HTTP
2024-12-21T12:18:08+01:00 [SERVER] INFO: Data Dir: ./data/
2024-12-21T12:18:08+01:00 [DB] INFO: Database Type: mariadb
2024-12-21T12:18:08+01:00 [SERVER] INFO: Connected to the database
2024-12-21T12:18:08+01:00 [SERVICES] INFO: Starting nscd
2024-12-21T12:18:08+01:00 [SERVER] INFO: Listening on 3001
2024-12-21T12:18:13+01:00 [SOCKET] INFO: New polling connection, IP = 192.168.**.**

Reboot command:

β”Œβ”€β”€(homelab-alpha@test-server)-[~]
└─$ sudo docker restart uptime-kuma-beta

Output log:

2024-12-21T12:18:19+01:00 [SERVER] INFO: Shutdown requested
2024-12-21T12:18:19+01:00 [SERVER] INFO: Called signal: SIGTERM
2024-12-21T12:18:19+01:00 [SERVICES] INFO: Stopping nscd
2024-12-21T12:18:20+01:00 [SERVER] INFO: Stopping all monitors
2024-12-21T12:18:20+01:00 [MONITOR] INFO: [Dockge Portal] isStop = true, no next check.
2024-12-21T12:18:20+01:00 [MONITOR] INFO: [Dozzle Container] isStop = true, no next check.
2024-12-21T12:18:20+01:00 [MONITOR] INFO: [Dozzle Portal] isStop = true, no next check.
2024-12-21T12:18:21+01:00 [MONITOR] INFO: [Energy Display] isStop = true, no next check.
2024-12-21T12:18:22+01:00 [DB] INFO: Closing the database
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processTimers (node:internal/timers:516:9)
    at safeBeat (/app/server/model/monitor.js:1012:25)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processTimers (node:internal/timers:516:9)
2024-12-21T12:18:22+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:18:22+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: Unable to acquire a connection
    at Client_MySQL2.acquireConnection (/app/node_modules/knex/lib/client.js:311:13)
    at Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:305:46)
    at Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:30)
    at Target.then (/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)
    at safeBeat (/app/server/model/monitor.js:1012:25)
2024-12-21T12:18:23+01:00 [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues
2024-12-21T12:18:24+01:00 [DB] INFO: Database closed
2024-12-21T12:18:24+01:00 [CLOUDFLARED] INFO: Stop cloudflared
2024-12-21T12:18:24+01:00 [SERVER] INFO: Graceful shutdown successful!
Welcome to Uptime Kuma
Your Node.js version: 20.18.0
2024-12-21T12:18:24+01:00 [SERVER] INFO: Env: production
2024-12-21T12:18:25+01:00 [SERVER] INFO: Uptime Kuma Version: 2.0.0-beta.1
2024-12-21T12:18:25+01:00 [SERVER] INFO: Loading modules
2024-12-21T12:18:26+01:00 [SERVER] INFO: Creating express and socket.io instance
2024-12-21T12:18:26+01:00 [SERVER] INFO: Server Type: HTTP
2024-12-21T12:18:26+01:00 [SERVER] INFO: Data Dir: ./data/
2024-12-21T12:18:26+01:00 [DB] INFO: Database Type: mariadb
2024-12-21T12:18:26+01:00 [SERVER] INFO: Connected to the database
2024-12-21T12:18:26+01:00 [SERVICES] INFO: Starting nscd
2024-12-21T12:18:26+01:00 [SERVER] INFO: Listening on 3001
2024-12-21T12:18:34+01:00 [SOCKET] INFO: New polling connection, IP = 192.168.**.**

@louislam louislam added this to the 2.0.0-beta.2 milestone Dec 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants