* Several updates 2022_09_20. See full commit log.

* The biggest change is the addition of CrowdSec intrusion prevention system to my environment. It is mitigating well over 600 intrusion attempts per day. * Added some new middlewares chains in middlewares-chains.yml to support the addition of CrowdSec. * Renamed traefik bouncer middleware in middlewares.yml. * Modified Dockerfile-php7 to supress some errors in php logs. * Fixed minor mistakes in my rclone scripts. * Modified media-services.txt to reflect current setup. * Updated bash_aliases - major change is the addition of commands for CrowdSec. * Edited README. docker-compose-t2.yml * Fixed a typo chain-auth-basic to chain-basic-auth * Added CrowdSec (agent). Although an agent is installed here, my primary agent (the mothership; the decision maker) is the one in docker-compose-t2-web.yml. This instance just parses the logs but makes no decisions. * Obsoleted CrowdSec dashboard - was using too much resources. * Modified traefik bouncer to get decisions from the "mothership" crowdsec remote agent instead of the local agent. * Added Prometheus to harvest some CrowdSec stats. I can now visualize metrics on my Grafana. * Obsoleted YoutubeDL-Material - could never get this to work. It has worked for others. * Obsoleted Flaresolverr - Saw no real use for my situation. * Renamed airsonic service to airsonic-advanced. * Added Plex Meta Manager - will push my configs as example at a later point. * Re-Added Tautulli. docker-compose-t2-synology.yml * Added ZeroTier. Now all my key hosts are connected through ZeroTier and Plex, Arr apps, etc. can be accessed using ZeroTier IPs. * Re-Added Plex-Sync to sync watched status between two Plex servers. * VS Code server has been giving me issues recently and is currently non functional on Synology. * Minor changes to DDNS updater docker-compose-t2-web.yml * Biggest change is the addition of CrowdSec. CrowdSec on my Digital Ocean VPS is the primary agent. * Added wg-easy for easy Wireguard deployment. This will run on my Digital Ocean VPS, which will be always on and the most reliable of all my servers. * Modified logging for serveral apps (eg. Traefik, Nginx, etc.) to simplify CrowdSec implementation. These logs will be parsed by CrowdSec agent for malicious activities and decisions (ban) will be sent to the bouncers. * Added CrowdSec cloudflare bouncer to block malicious IPs using Cloudflare managed rules. Unfortunately CrowdSec limits blocklist to max 10,000 IPs. For the rest, see below. * Added CrowdSec Traefik bouncer to block malicious IPs. * In addition, installed CrowdSec Firewall bouncer on host machine to make Linux firewall block malicious IPs. * Replaced CF trusted IPs list with an environmental variable with same info. Looks cleaner.
bhulad · Sep 20, 2022 · a783d60 · a783d60
1 parent 7442900
commit a783d60
Show file tree

Hide file tree

Showing 13 changed files with 360 additions and 123 deletions.
diff --git a/README.md b/README.md
@@ -54,8 +54,8 @@ The following posts have been updated/replaced by the posts linked above:
 
 ## Docker, Docker Compose, and Traefik Versions (updated January 23, 2022)
 
-- Docker: 20.10.13
-- Docker Compose: v2.5.0
+- Docker: 20.10.17
+- Docker Compose: v2.6.0
 - Traefik: 2.7
 
 <strong>Update (September 13, 2021):</strong> I moved from TOML to YAML for Traefik 2 dynamic configurations. I have included example configuration files for both. However, since I do not use TOML anymore, there may be minor syntax errors or typos.
@@ -85,6 +85,8 @@ Almost any app/service from the docker-compose files listed above can be copy-pa
 
 I use Syncthing to keep certain key files synched between various systems.
 
+For security, I implemented CrowdSec multi-server setup recently. From the stats, it is blocking/mitigating well over 600 intrusion attempts per day on my servers. I will cover this in a separate guide later but you will find the docker-compose CrowdSec, Traefik Bouncer, and Cloudflare Bouncer Bouncers in my repo already.
+
 ## What apps are included in this stack?
 
 The apps I use are scattered around in several different docker-compose files. Some apps are used in more than one host and some on only one.

diff --git a/appdata/traefik2/rules/cloudserver/middlewares-chains.yml b/appdata/traefik2/rules/cloudserver/middlewares-chains.yml
@@ -3,7 +3,15 @@ http:
     chain-no-auth:
       chain:
         middlewares:
-          - middlewares-crowdsec-bouncer-traefik
+          - middlewares-traefik-bouncer # leave this out if you are not using CrowdSec
+          - middlewares-rate-limit
+          - middlewares-https-redirectscheme
+          - middlewares-secure-headers
+          - middlewares-compress
+
+    chain-no-auth-no-crowdsec:
+      chain:
+        middlewares:
           - middlewares-rate-limit
           - middlewares-https-redirectscheme
           - middlewares-secure-headers
@@ -12,7 +20,7 @@ http:
     chain-basic-auth:
       chain:
         middlewares:
-          - middlewares-crowdsec-bouncer-traefik
+          - middlewares-traefik-bouncer # leave this out if you are not using CrowdSec
           - middlewares-rate-limit
           - middlewares-https-redirectscheme
           - middlewares-secure-headers
@@ -22,7 +30,7 @@ http:
     chain-oauth:
       chain:
         middlewares:
-          - middlewares-crowdsec-bouncer-traefik
+          - middlewares-traefik-bouncer # leave this out if you are not using CrowdSec
           - middlewares-rate-limit
           - middlewares-https-redirectscheme
           - middlewares-secure-headers
@@ -32,7 +40,7 @@ http:
     chain-oauth-external:
       chain:
         middlewares:
-          - middlewares-crowdsec-bouncer-traefik
+          - middlewares-traefik-bouncer # leave this out if you are not using CrowdSec
           - middlewares-rate-limit
           - middlewares-https-redirectscheme
           - middlewares-secure-headers
@@ -42,7 +50,7 @@ http:
     chain-authelia:
       chain:
         middlewares:
-          - middlewares-crowdsec-bouncer-traefik
+          - middlewares-traefik-bouncer # leave this out if you are not using CrowdSec
           - middlewares-rate-limit
           - middlewares-https-redirectscheme
           - middlewares-secure-headers

diff --git a/appdata/traefik2/rules/cloudserver/middlewares.yml b/appdata/traefik2/rules/cloudserver/middlewares.yml
@@ -50,9 +50,9 @@ http:
         authResponseHeaders:
           - "X-Forwarded-User"
 
-    middlewares-crowdsec-bouncer-traefik:
+    middlewares-traefik-bouncer:
       forwardAuth:
-        address: "http://crowdsec-bouncer:8080/api/v1/forwardAuth" # Make sure you have the crowdsec-bouncer service in docker-compose.yml
+        address: "http://traefik-bouncer:8080/api/v1/forwardAuth" # Make sure you have the traefik-bouncer service in docker-compose.yml
         trustForwardHeader: true
 
     middlewares-oauth-external:
@@ -79,6 +79,5 @@ http:
         maxResponseBodyBytes: 2000000
         maxRequestBodyBytes: 10485760  
         memRequestBodyBytes: 2097152  
-        maxResponseBodyBytes: 10485760
         memResponseBodyBytes: 2097152
         retryExpression: "IsNetworkError() && Attempts() <= 2"
diff --git a/custom/Dockerfile-php7 b/custom/Dockerfile-php7
@@ -8,4 +8,10 @@ ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/do
 
 
 RUN chmod uga+x /usr/local/bin/install-php-extensions && sync && \
-    install-php-extensions gd mysqli pdo_mysql opcache imagick exif zip mcrypt pspell redis sockets ssh2 bcmath
+    install-php-extensions gd mysqli pdo_mysql opcache imagick exif zip mcrypt pspell redis sockets ssh2 bcmath intl
+
+# ADDED AUGUST 18, 2022 BECAUSE ERRORS IN PHP7 DOCKER LOGS TO INCREASE MAX_CHILDREN
+# https://serverfault.com/questions/884256/how-and-where-to-configure-pm-max-children-for-php-fpm-with-docker
+# SETUP PHP-FPM CONFIG SETTINGS (max_children / max_requests)
+RUN echo 'pm.max_children = 15' >> /usr/local/etc/php-fpm.d/zz-docker.conf && \
+    echo 'pm.max_requests = 500' >> /usr/local/etc/php-fpm.d/zz-docker.conf
diff --git a/docker-compose-t2-obsolete.yml b/docker-compose-t2-obsolete.yml
@@ -1904,3 +1904,94 @@ services:
       # Since cf-companion creates CNAMEs based on host rules, this a workaround for non-docker/external apps
       - "traefik.http.routers.cf-companion-rtr.rule=Host(`webmin.$DOMAINNAME2`) || Host(`shellh.$DOMAINNAME2`) || Host(`syno.$DOMAINNAME2`) || Host(`ibox.$DOMAINNAME2`) || Host(`synplex.$DOMAINNAME2`) || Host(`pirpi.$DOMAINNAME2`) || Host(`pidoc.$DOMAINNAME2` || Host(`stpc.$DOMAINNAME2`)"
 
+  # CrowdSec Dashboard
+  # Uses a lot of resources. May be crowdsec's cloud dashboard or Prometheus Grafana may be a better option.
+  csdash:
+    <<: *common-keys-apps # See EXTENSION FIELDS at the top
+    container_name: crowdsec-dashboard
+    build:
+      context: $DOCKERDIR/custom/
+      dockerfile: Dockerfile-csdash
+    depends_on:
+      - crowdsec
+    environment:
+      - MB_DB_FILE=/data/metabase.db
+      - MGID=1000
+    volumes:
+      - $DOCKERDIR/appdata/crowdsec/data:/metabase-data/
+    labels:
+      - "traefik.enable=true"
+      ## HTTP Routers
+      - "traefik.http.routers.csdash-rtr.entrypoints=https"
+      - "traefik.http.routers.csdash-rtr.rule=Host(`csdash.$DOMAINNAME_CLOUD_SERVER`)"
+      ## Middlewares
+      - "traefik.http.routers.csdash-rtr.middlewares=chain-oauth@file"
+      ## HTTP Services
+      - "traefik.http.routers.csdash-rtr.service=csdash-svc"
+      - "traefik.http.services.csdash-svc.loadbalancer.server.port=3000"
+
+  youtubedl-material:
+    <<: *common-keys-apps # See EXTENSION FIELDS at the top
+    image: tzahi12345/youtubedl-material:latest
+    container_name: youtubedl-material
+    # ports:
+    #   - "$YOUTUBEDLMATERIAL_PORT:17442"
+    volumes:
+      - $DOCKERDIR/appdata/youtubedl-material/config:/app/appdata
+      - $DATADIR/downloads/youtubedl-material/audio:/app/audio
+      - $DATADIR/downloads/youtubedl-material/video:/app/video
+      - $DATADIR/downloads/youtubedl-material/subscriptions:/app/subscriptions
+      - $DOCKERDIR/appdata/youtubedl-material/users:/app/users
+    environment:
+      UID: $PUID
+      GID: $PGID
+    labels:
+      - "traefik.enable=true"
+      ## HTTP Routers
+      - "traefik.http.routers.youtubedl-material-rtr.entrypoints=https"
+      - "traefik.http.routers.youtubedl-material-rtr.rule=Host(`ytdl.$DOMAINNAME_CLOUD_SERVER`)"
+      ## Middlewares
+      - "traefik.http.routers.youtubedl-material-rtr.middlewares=chain-oauth@file"
+      ## HTTP Services
+      - "traefik.http.routers.youtubedl-material-rtr.service=youtubedl-material-svc"
+      - "traefik.http.services.youtubedl-material-svc.loadbalancer.server.port=17442"
+
+  # FlareSolverr - Proxy server to bypass Cloudflare protection
+  # Not sure if I need this or not. Just testing.
+  flaresolverr:
+    <<: *common-keys-apps # See EXTENSION FIELDS at the top
+    image: ghcr.io/flaresolverr/flaresolverr:latest
+    container_name: flaresolverr
+    networks:
+      t2_proxy:
+        ipv4_address: 192.168.90.160
+    environment:
+      LOG_LEVEL: info
+      LOG_HTML: false
+      CAPTCHA_SOLVER: none
+      TZ: $TZ
+    #ports:
+    #  - "8191:8191"
+    restart: unless-stopped
+
+  # HeadScale - SDN
+  headscale:
+    <<: *common-keys-core # See EXTENSION FIELDS at the top
+    image: headscale/headscale:latest
+    container_name: headscale
+    volumes:
+      - $DOCKERDIR/appdata/headscale/config:/etc/headscale
+      - $DOCKERDIR/appdata/headscale/data:/var/lib/headscale
+    #ports:
+    #  - 27896:8080
+    command: headscale serve
+    labels:
+      - "traefik.enable=true"
+      ## HTTP Routers
+      - "traefik.http.routers.headscale-rtr.entrypoints=https"
+      - "traefik.http.routers.headscale-rtr.rule=Host(`headscale.$DOMAINNAME_SHB`)"
+      ## Middlewares
+      #- "traefik.http.routers.headscale-rtr.middlewares=chain-no-auth@file"
+      ## HTTP Services
+      - "traefik.http.routers.headscale-rtr.service=headscale-svc"
+      - "traefik.http.services.headscale-svc.loadbalancer.server.port=8080"
diff --git a/docker-compose-t2-synology.yml b/docker-compose-t2-synology.yml
@@ -392,6 +392,20 @@ services:
       - "traefik.http.routers.dashy-rtr.service=dashy-svc"
       - "traefik.http.services.dashy-svc.loadbalancer.server.port=80"
 
+  # ZeroTier-One - Private Network over Internet
+  zerotier:
+    image: zerotier/zerotier-synology:latest
+    container_name: zerotier
+    network_mode: host
+    restart: always
+    cap_add:
+      - NET_ADMIN
+      - SYS_ADMIN
+    devices:
+      - /dev/net/tun
+    volumes:
+      - $DOCKERDIR/appdata/zerotier:/var/lib/zerotier-one
+
   ############################# SMART HOME
 
   # Mosquitto - MQTT Broker
@@ -473,7 +487,7 @@ services:
       PLEX_CLAIM_FILE: /run/secrets/plex_claim
       PLEX_UID: $PUID
       PLEX_GID: $PGID
-      ADVERTISE_IP: http://$SERVER_IP1:32400/
+      ADVERTISE_IP: http://$ZEROTIER_IP_SYNOLOGY:32400/
     secrets:
       - plex_claim
     labels:
@@ -487,6 +501,19 @@ services:
       - "traefik.http.routers.plexms-rtr.service=plexms-svc"
       - "traefik.http.services.plexms-svc.loadbalancer.server.port=32400"
 
+  # Plex-Sync - For Syncing watched status between plex servers
+  plex-sync:
+    image: patsissons/plex-sync:develop
+    container_name: plex-sync
+    security_opt:
+      - no-new-privileges:true
+    environment:
+      INITIAL_RUN: "true"
+      DRY_RUN: 0
+      CRON_SCHEDULE: "*/30 * * * *"
+      SECTION_MAPS: $HOME_PLEX/$HOME_PLEX_MOVIESH $CLOUD_PLEX/$CLOUD_PLEX_MOVIESH | $HOME_PLEX/$HOME_PLEX_SHOWST $CLOUD_PLEX/$CLOUD_PLEX_SHOWST | $HOME_PLEX/$HOME_PLEX_MOVIESI $CLOUD_PLEX/$CLOUD_PLEX_MOVIESI | $HOME_PLEX/$HOME_PLEX_MOVIEST $CLOUD_PLEX/$CLOUD_PLEX_MOVIEST | $HOME_PLEX/$HOME_PLEX_MOVIESK $CLOUD_PLEX/$CLOUD_PLEX_MOVIESK | $HOME_PLEX/$HOME_PLEX_SHOWSK $CLOUD_PLEX/$CLOUD_PLEX_SHOWSK
+    restart: unless-stopped
+
   ############################# MEDIA FILE MANAGEMENT
 
   # Tdarr Server - Library Optimizer
@@ -751,9 +778,9 @@ services:
       - "traefik.http.services.syncthing-svc.loadbalancer.server.port=8384"
 
   # VSCode - VSCode Editing
-  vscode:
+  vscode: # NOT WORKING - Started seeing WebSocket Closed with Status Code 1006 Error somewhere in July
     <<: *common-keys-core # See EXTENSION FIELDS at the top
-    image: lscr.io/linuxserver/code-server:latest
+    image: lscr.io/linuxserver/code-server:latest 
     container_name: vscode
     ports:
       - "$VSCODE_PORT:8443"
@@ -769,7 +796,7 @@ services:
       # SUDO_PASSWORD: password #optional
       # SUDO_PASSWORD_HASH: #optional
       # PROXY_DOMAIN: code-server.my.domain #optional
-      DEFAULT_WORKSPACE: /config/data/User/Workspaces/AZ.code-workspace #optional
+      # DEFAULT_WORKSPACE: /config/data/User/Workspaces/AZ.code-workspace #optional
     labels:
       - "traefik.enable=true"
       ## HTTP Routers
@@ -852,14 +879,14 @@ services:
       - $DOCKERDIR/appdata/ddns-updater:/updater/data # Owned by UID 1000
     environment:
       <<: *default-tz-puid-pgid
-      PERIOD: 1h
+      PERIOD: 12h
       UPDATE_COOLDOWN_PERIOD: 5m
       PUBLICIP_DNS_TIMEOUT: 3s
       HTTP_TIMEOUT: 10s
       # Web UI
       LISTENING_PORT: 8000
       # Backup
-      BACKUP_PERIOD: 72h15m # 0 to disable
+      BACKUP_PERIOD: 96h # 0 to disable
       BACKUP_DIRECTORY: /updater/data/backups
       # Other
       LOG_LEVEL: info