Alert on management backup NAS mount and capacity #5

New issue

Closed

opened 2026-08-29 19:57:04 -04:00 by scruffydude · 1 comment

scruffydude commented

2026-08-29 19:57:04 -04:00

Owner

Monitor the NAS mount used by management-plane backups.

Acceptance criteria:

Alert when /mnt/nas/docker-backups is not a mounted CIFS filesystem.
Alert when the share is not writable.
Alert at defined free-space thresholds.
Report includes mount status, free space, and filesystem usage.

Migrated from gitea.darkcrest.net/scruffydude/homelab#6, originally opened 2026-06-13.

Monitor the NAS mount used by management-plane backups. Acceptance criteria: - Alert when `/mnt/nas/docker-backups` is not a mounted CIFS filesystem. - Alert when the share is not writable. - Alert at defined free-space thresholds. - Report includes mount status, free space, and filesystem usage. --- _Migrated from gitea.darkcrest.net/scruffydude/homelab#6, originally opened 2026-06-13._

scruffydude commented

2026-09-05 21:23:22 -04:00

Author

Owner

Closing: all four acceptance criteria are live and verified.

Not-mounted / wrong-fstype detection: cron check on broadcaster (ansible/roles/management_backup/templates/check-backup-nas-mount.sh.j2, every 10m) confirms mount + cifs fstype.
Writability: touch/rm probe against the mount.
Free-space threshold: alerts only when both free % (<10%) and free GB (<500GB) are breached, so the NAS's normal ~9-10% free (2TB+ headroom on a 21TB share) doesn't false-positive.
Report with mount status/free space/usage: pushed as the alert's error detail string on every run (success or failure).
Delivery: pushed into Gatus as a push-based external-endpoint, alerting to the existing Discord webhook (same path as every other check here).
Also added, while at it: Beszel now tracks the mount as an extra filesystem so there's a real historical capacity graph in its dashboard, not just point-in-time alerts.
Live-verified both directions: controlled umount -> two failed pushes -> Discord alert TRIGGERED; mount -a restore -> two successful pushes -> Discord alert RESOLVED.

Full writeup in docs/monitoring-alerting.md.

Closing: all four acceptance criteria are live and verified. - **Not-mounted / wrong-fstype detection**: cron check on `broadcaster` (`ansible/roles/management_backup/templates/check-backup-nas-mount.sh.j2`, every 10m) confirms mount + `cifs` fstype. - **Writability**: touch/rm probe against the mount. - **Free-space threshold**: alerts only when both free % (<10%) *and* free GB (<500GB) are breached, so the NAS's normal ~9-10% free (2TB+ headroom on a 21TB share) doesn't false-positive. - **Report with mount status/free space/usage**: pushed as the alert's `error` detail string on every run (success or failure). - Delivery: pushed into Gatus as a push-based external-endpoint, alerting to the existing Discord webhook (same path as every other check here). - Also added, while at it: Beszel now tracks the mount as an extra filesystem so there's a real historical capacity graph in its dashboard, not just point-in-time alerts. - Live-verified both directions: controlled `umount` -> two failed pushes -> Discord alert TRIGGERED; `mount -a` restore -> two successful pushes -> Discord alert RESOLVED. Full writeup in `docs/monitoring-alerting.md`.

scruffydude closed this issue

2026-09-05 21:23:23 -04:00

scruffydude referenced this issue from a commit

2026-09-05 21:27:40 -04:00

Close #5: NAS backup mount/capacity monitoring, and switch backups to weekly

scruffydude referenced this issue from a pull request that will close it,

2026-09-05 21:28:45 -04:00

NAS backup capacity monitoring, weekly backups, CloudCLI alert fixes #28