PLGT Alerting Guide

PLGT Alerting Guide

Last Updated: 2026-01-07

---

Overview

The PLGT stack implements a multi-channel alerting system that routes alerts to appropriate channels based on severity, category, and client labels.

graph LR
    subgraph "Alert Sources"
        PR[Prometheus Rules]
        LK[Loki Rules]
    end

    AM[Alertmanager]

    subgraph "Notification Channels"
        EM[Email<br>nhlanhla@isutech.co.za]
        SL[Slack<br>#alerts #critical]
        TG[Telegram<br>Mobile Notifications]
        MS[MS Teams<br>Enterprise]
    end

    PR --> AM
    LK --> AM
    AM --> EM
    AM --> SL
    AM --> TG
    AM -.-> MS

---

Alert Routing

Severity-Based Routing

Severity	Group Wait	Repeat	Channels
Critical	10 seconds	1 hour	Email + Slack (#critical) + Telegram
Security	10 seconds	30 min	Email + Slack (#critical) + Telegram
Warning	1 minute	4 hours	Email + Slack (#alerts)
Info	5 minutes	12 hours	Slack (#alerts) only

Client-Specific Routing

Client alerts are routed to dedicated receivers:

routes:
  - match:
      client: umdoni-municipality
    receiver: 'umdoni-alerts'
    routes:
      - match:
          severity: critical
        receiver: 'umdoni-critical-multichannel'

---

Notification Channels

Email Notifications

Primary notification channel with professional HTML templates.

Configuration:

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@isutech.co.za'
  smtp_auth_username: ''  # Via environment variable
  smtp_auth_password: ''  # Via environment variable
  smtp_require_tls: true

receivers:
  - name: 'isutech-team'
    email_configs:
      - to: 'nhlanhla@isutech.co.za'
        send_resolved: true
        headers:
          Subject: '{{ template "email.subject" . }}'
        html: '{{ template "email.html" . }}'

Template Features:

• Color-coded severity (red=critical, yellow=warning)
• Alert summary with instance details
• Start time and duration tracking
• Action buttons: View in Grafana, Explore Logs
• Professional iSuTech branding

---

Slack Notifications

Team collaboration channel with rich message formatting.

Setup Steps:

1. Create Slack workspace or use existing
2. Go to Apps → Incoming Webhooks
3. Create new webhook for your workspace
4. Copy the webhook URL

Configuration:

global:
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

receivers:
  - name: 'isutech-team'
    slack_configs:
      - channel: '#alerts'
        send_resolved: true
        username: 'PLGT Alertmanager'
        icon_emoji: ':warning:'
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'
        color: '{{ template "slack.color" . }}'

  - name: 'isutech-critical-multichannel'
    slack_configs:
      - channel: '#critical'
        username: 'PLGT CRITICAL'
        icon_emoji: ':rotating_light:'

Channel Structure:

Channel	Purpose
#alerts	Standard warnings, info notifications
#critical	Critical alerts, security events

---

Telegram Notifications

Mobile notifications for immediate awareness.

Setup Steps:

1. Create Bot:

   • Open Telegram, search for @BotFather
   • Send /newbot and follow prompts
   • Save the bot token (format: 123456789:ABC-DEF1234ghIkl-zyx57W2v1u123ew11)

2. Get Chat ID:

   • Send any message to your new bot
   • Visit: https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates
   • Find "chat":{"id":123456789} in the response

3. Configure Alertmanager:

receivers:
  - name: 'isutech-critical-multichannel'
    telegram_configs:
      - bot_token: 'YOUR_BOT_TOKEN'
        chat_id: 123456789
        parse_mode: 'HTML'
        message: '{{ template "telegram.message" . }}'
        send_resolved: true

Message Format:

<b>{{ .Status | toUpper }}</b>
{{ range .Alerts }}
<b>{{ .Labels.alertname }}</b>
Severity: {{ .Labels.severity }}
Instance: {{ .Labels.instance }}
{{ .Annotations.description }}
{{ end }}

---

Microsoft Teams (Optional)

Enterprise integration via webhooks.

Setup:

1. In Teams channel → Connectors → Incoming Webhook
2. Configure and copy webhook URL

Configuration:

receivers:
  - name: 'isutech-teams'
    webhook_configs:
      - url: 'https://outlook.office.com/webhook/YOUR/TEAMS/WEBHOOK'
        send_resolved: true

---

Alert Rules

Prometheus Alert Rules

Location: prometheus/rules/

Infrastructure Alerts (alerts.yml)

Alert	Trigger	Severity
InstanceDown	Target unreachable 1m	critical
HighCpuUsage	CPU > 80% for 5m	warning
CriticalCpuUsage	CPU > 95% for 2m	critical
HighMemoryUsage	Memory > 80% for 5m	warning
HighDiskUsage	Disk > 80% for 5m	warning
DiskWillFillIn24Hours	Predicted disk full	warning

Website Monitoring Alerts

Alert	Trigger	Severity
SiteDown	HTTP probe fails 2m	critical
SSLCertExpiringSoon	SSL < 30 days	warning
SSLCertExpiryCritical	SSL < 7 days	critical
HighResponseTime	Response > 3s	warning

Stack Self-Monitoring (self-monitoring.yml)

Alert	Trigger	Severity
PrometheusDown	Prometheus unreachable	critical
LokiDown	Loki unreachable	critical
GrafanaDown	Grafana unreachable	critical
AlertmanagerDown	Alertmanager unreachable	critical
PLGTServerDiskCritical	Disk > 85%	critical
NoMetricsReceived	No scrapes 5m	critical

---

Loki Alert Rules

Location: loki/rules/umdoni/

Log-based alerts using LogQL:

Activity Monitoring (activity-alerts.yml)

Alert	Trigger	Severity
UmdoniFailedLoginSpike	5+ failed logins in 10min	warning
UmdoniBruteForceDetected	10+ failed logins in 5min	critical
UmdoniAdminAfterHours	Admin login 18:00-07:00	warning
UmdoniBulkDelete	5+ deletes in 10min	warning
UmdoniUserManagement	User/permission changes	warning
UmdoniErrorSpike	5+ errors in 10min	warning
UmdoniNoActivity	No logs for 1 hour	warning

---

Inhibition Rules

Prevent duplicate alerts when parent issues exist:

inhibit_rules:
  # Critical suppresses warning for same alert
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

  # Site down suppresses SSL and response alerts
  - source_match:
      alertname: 'SiteDown'
    target_match_re:
      alertname: '(SSLCertExpiring|HighResponseTime)'
    equal: ['instance']

  # Umdoni site down suppresses all Umdoni alerts
  - source_match:
      alertname: 'UmdoniSiteDown'
    target_match_re:
      alertname: '(UmdoniDashboard.*|UmdoniMetrics.*)'
    equal: ['client']

---

Testing Alerts

Test Slack Integration

curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test alert from PLGT Stack"}' \
  'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

Test Telegram Integration

curl -X POST "https://api.telegram.org/bot<TOKEN>/sendMessage" \
  -d "chat_id=<CHAT_ID>" \
  -d "text=Test alert from PLGT Stack" \
  -d "parse_mode=HTML"

Trigger Test Alert in Prometheus

Temporarily add a test alert rule:

- alert: TestAlert
  expr: vector(1)
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: "Test alert"

Check Alertmanager Status

# View current alerts
curl http://localhost:9093/api/v2/alerts

# View silences
curl http://localhost:9093/api/v2/silences

---

Managing Alerts

Silencing Alerts

In Grafana: Alerting → Silences → New Silence

Or via API:

curl -X POST http://localhost:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [{"name": "alertname", "value": "HighCpuUsage"}],
    "startsAt": "2026-01-07T00:00:00Z",
    "endsAt": "2026-01-07T12:00:00Z",
    "createdBy": "admin",
    "comment": "Maintenance window"
  }'

Reloading Configuration

# Reload Alertmanager config
docker compose exec alertmanager kill -HUP 1

# Or via API (if enabled)
curl -X POST http://localhost:9093/-/reload

---

Troubleshooting

Alerts Not Firing

1. Check Prometheus targets: http://localhost:9090/targets
2. Verify alert rules: http://localhost:9090/rules
3. Check Alertmanager: http://localhost:9093/#/alerts

Notifications Not Sent

1. Check Alertmanager logs: docker compose logs alertmanager
2. Verify webhook URLs are correct
3. Test webhooks manually (see Testing section)
4. Check for network/firewall issues

Duplicate Alerts

1. Review inhibition rules
2. Check grouping configuration
3. Verify label matching in routes

---

Best Practices

1. Don’t over-alert - Only alert on actionable issues
2. Use severity appropriately - Critical = immediate action required
3. Include context - Alert descriptions should explain what to do
4. Test regularly - Verify notification channels work
5. Review and tune - Adjust thresholds based on actual incidents

---

Next: Operations Guide