Incident Runbooks

Incident Runbooks

Last Updated: 2026-01-07

---

Overview

Quick reference guides for responding to common alerts and incidents in the PLGT Observability Stack.

On-Call Priority

Critical alerts require immediate attention. Follow the escalation timeline in each runbook.

---

Runbook Index

Category	Alert	Severity
Website Down	SiteDown, UmdoniSiteDown	Critical
SSL Certificate Expiring	SSLCertExpiring*	Warning/Critical
High Response Time	HighResponseTime	Warning
Failed Login Spike	UmdoniFailedLoginSpike	Warning
Brute Force Detected	UmdoniBruteForceDetected	Critical
PLGT Stack Issues	*Down	Critical
Database Issues	Connection failures	Critical
Disk Space Critical	*DiskCritical	Critical

---

Website Down

Alerts: SiteDown, UmdoniSiteDown Severity: Critical Impact: Users cannot access the website

Immediate Actions

Step 1: Verify

# Test site accessibility
curl -sI https://umdoni.gov.za --max-time 10

# Check from external service
# https://downforeveryoneorjustme.com/umdoni.gov.za

Step 2: Diagnose

# DNS check
dig umdoni.gov.za

# SSL check
echo | openssl s_client -connect umdoni.gov.za:443 2>/dev/null | head -5

Step 3: Server Check

# On hosting server
systemctl status apache2
tail -100 /var/log/apache2/error.log
df -h

Troubleshooting Decision Tree

graph TD
    A[Site Down Alert] --> B{DNS Resolving?}
    B -->|No| C[Check DNS Provider]
    B -->|Yes| D{Server Responding?}
    D -->|No| E[Check Hosting Provider]
    D -->|Yes| F{Application Error?}
    F -->|Yes| G[Check Error Logs]
    F -->|No| H[Network/Firewall Issue]

Escalation Timeline

Time	Action
0-5 min	Verify and diagnose
5-15 min	Attempt restart/fix
15+ min	Contact hosting support
30+ min	Escalate to senior team

---

SSL Certificate Expiring

Alerts: SSLCertExpiringSoon, SSLCertExpiryCritical Severity: Warning (30 days), Critical (7 days) Impact: Browser security warnings, potential site blocking

Check Current Status

# Check expiry date
echo | openssl s_client -servername umdoni.gov.za -connect umdoni.gov.za:443 2>/dev/null \
  | openssl x509 -noout -dates

# Calculate days remaining
echo | openssl s_client -servername umdoni.gov.za -connect umdoni.gov.za:443 2>/dev/null \
  | openssl x509 -noout -enddate

Renewal Process

Let's Encrypt (cPanel)

1. Login to cPanel
2. Navigate to SSL/TLS Status
3. Click Run AutoSSL
4. Wait 5-10 minutes
5. Verify renewal

Manual Certificate

1. Generate CSR in cPanel
2. Purchase/obtain certificate from CA
3. Install via cPanel SSL Manager
4. Verify installation

Verification

curl -vI https://umdoni.gov.za 2>&1 | grep -E "expire|subject"

---

High Response Time

Alert: HighResponseTime Severity: Warning Impact: Poor user experience, potential timeouts

Diagnosis

# Measure response time
curl -o /dev/null -s -w "Total: %{time_total}s\nConnect: %{time_connect}s\nTTFB: %{time_starttransfer}s\n" \
  https://umdoni.gov.za

# Check server load (on server)
uptime
top -bn1 | head -20

Common Causes & Solutions

Cause	Diagnosis	Solution
High CPU	top shows >80%	Kill runaway processes
Slow database	Check slow query log	Optimize queries, add indexes
Large assets	Browser dev tools	Compress images, enable gzip
No caching	Check response headers	Enable browser caching

Quick Fixes

Enable Compression (.htaccess):

<IfModule mod_deflate.c>
  AddOutputFilterByType DEFLATE text/html text/css application/javascript
</IfModule>

Enable Caching (.htaccess):

<IfModule mod_expires.c>
  ExpiresActive On
  ExpiresByType image/jpg "access plus 1 month"
  ExpiresByType text/css "access plus 1 week"
</IfModule>

---

Failed Login Spike

Alert: UmdoniFailedLoginSpike Severity: Warning Impact: Potential security incident, user lockouts

Investigation

1. Check Grafana Dashboard

   • Navigate to: monitor.isutech.co.za/d/umdoni-activity
   • Review “Failed Login Attempts” panel

2. Query Loki

   {job="umdoni-activity"} |= "failed" |~ "login|Login"

3. Identify Pattern

Pattern	Likely Cause	Action
Same user, multiple attempts	Password issue	Contact user, reset password
Same IP, multiple users	Targeted attack	Block IP
Random users/IPs	Normal failures	Monitor only

Response Actions

User Password Issue:

• Contact user directly
• Reset password if needed
• Check for account lockout

Suspicious Activity:

• Block IP in firewall
• Enable CAPTCHA if available
• Continue monitoring

---

Brute Force Detected

Alert: UmdoniBruteForceDetected Severity: Critical Impact: Active attack, potential account compromise

Time-Critical Response

This is an active attack. Take immediate action to block the attacker.

Immediate Actions

1. Identify Attacker

# Query Loki for attacking IP
{job="umdoni-activity"} |= "failed" | json | ip != ""

2. Block IP

Via cPanel:

• Imunify360 → Block IP

Via .htaccess:

order allow,deny
deny from 123.456.789.0
allow from all

3. Check Compromise

# Check for successful logins from attacker IP
{job="umdoni-activity"} |= "success" | json | ip = "ATTACKER_IP"

Post-Incident Actions

1. Audit Accounts

   • Check for unusual activity
   • Force password reset if compromise suspected

2. Security Improvements

   • Enable 2FA if available
   • Implement rate limiting
   • Add CAPTCHA to login

3. Document Incident

   • Time of attack
   • Attacking IPs
   • Actions taken
   • Accounts affected

---

PLGT Stack Issues

Alerts: PrometheusDown, LokiDown, GrafanaDown, AlertmanagerDown Severity: Critical Impact: Monitoring capabilities degraded or unavailable

Access PLGT Server

ssh root@46.224.40.5
cd /opt/plg-unified

Quick Diagnostics

# Check all services
docker compose ps

# Check specific service logs
docker compose logs <service> --tail 100

# Check resource usage
docker stats --no-stream

Common Issues

Container Down

docker compose restart <service>
docker compose logs <service> --tail 50

Out of Memory

free -m
# Increase limits in docker-compose.yml
# Restart stack
docker compose down && docker compose up -d

Disk Full

df -h
./scripts/disk-cleanup.sh
docker system prune -a --volumes

Config Error

# Validate Prometheus config
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml

# Validate Alertmanager config
docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml

Full Stack Restart

docker compose down
docker compose up -d
docker compose ps

---

Database Issues

Alert: Database connection failures Severity: Critical Impact: Application not functioning

Quick Diagnostics

# Test connection
mysql -h hostname -u user -p -e "SELECT 1"

# Check MySQL status
systemctl status mysql

# Check disk space
df -h /var/lib/mysql

Common Fixes

Too Many Connections

SHOW PROCESSLIST;
-- Kill idle connections
KILL <process_id>;

Table Crash

mysqlcheck -u root -p --repair database_name

Slow Queries

SHOW FULL PROCESSLIST;
-- Identify and kill long-running queries
KILL <process_id>;

---

Disk Space Critical

Alerts: HighDiskUsage, PLGTServerDiskCritical Severity: Warning/Critical Impact: Services may crash, data loss possible

Assessment

# Check usage
df -h

# Find large directories
du -sh /* | sort -rh | head -20

# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null

Cleanup Actions

PLGT Server

# Run cleanup script
/opt/plg-unified/scripts/disk-cleanup.sh

# Clean Docker
docker system prune -a --volumes

# Clean journal logs
journalctl --vacuum-size=500M

Web Server

# Clean old logs
find /var/log -name "*.gz" -mtime +30 -delete

# Clean temp files
rm -rf /tmp/*

# Clean old backups
find /backups -mtime +30 -delete

---

Quick Reference

Essential Commands

# Check site
curl -sI https://umdoni.gov.za | head -5

# Check SSL
echo | openssl s_client -connect umdoni.gov.za:443 2>/dev/null \
  | openssl x509 -noout -dates

# Check DNS
dig +short umdoni.gov.za

# PLGT services
ssh root@46.224.40.5 "cd /opt/plg-unified && docker compose ps"

Dashboard Links

Dashboard	URL
Main	monitor.isutech.co.za
Umdoni Activity	/d/umdoni-activity
SLA Tracking	/d/sla-tracking
Alertmanager	/alertmanager

Contacts

Role	Contact
Primary On-Call	Nhlanhla Mnyandu
Email	nhlanhla@isutech.co.za
Afrihost Support	support@afrihost.com

---

Next: Client Onboarding