PLGT Security Enhancement Plan

PLGT Security Enhancement Plan

Status: DEFERRED - Awaiting memory upgrade Created: 2026-02-08 Server: Hetzner (46.224.40.5) Current RAM: ~3.7GB (insufficient for full PLGT stack)

---

Background

On 2026-02-08, the server was compromised by a cryptocurrency miner. Investigation revealed:

1. PLGT stack was mostly down (OOM killed)
2. Auth logs were deleted by attacker
3. No active blocking mechanism was in place

Immediate fix: Installed fail2ban for SSH protection.

---

Current Protection (Active)

Tool	Status	Function
fail2ban	Active	SSH brute-force blocking (24h ban)
recidive jail	Active	Repeat offender blocking (1 week ban)
iptables	Active	Firewall rules blocking mining pool

---

PLGT Current State (Mostly Down)

Component	Status	Issue
Grafana	Running	OK
Loki	Running	OK
Alloy	Down	OOM killed (exit 137)
Prometheus	Down	Stopped
Alertmanager	Down	Stopped
Node-exporter	Down	Stopped
Tempo	Down	OOM killed (exit 137)

---

Priority Tasks (When Memory Available)

Priority 1: Restart PLGT Core Components

# Restart essential components only (skip Tempo for now)
cd /opt/plg-unified
docker-compose up -d prometheus alertmanager node-exporter alloy loki grafana

Memory optimization options:

• Remove Tempo (tracing) - saves ~500MB
• Reduce Prometheus retention
• Limit Loki ingestion rate

Priority 2: Add Security Alerting to PLGT

Create Loki-based alerts for security events:

/opt/plg-unified/loki/rules/security-alerts.yml

groups:
  - name: security
    rules:
      - alert: SSHBruteForce
        expr: |
          count_over_time({job="auth"} |~ "Failed password" [5m]) > 10
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "SSH brute force attempt detected"

      - alert: fail2banBan
        expr: |
          count_over_time({job="syslog"} |~ "fail2ban.*Ban" [1m]) > 0
        labels:
          severity: info
        annotations:
          summary: "fail2ban banned an IP"

      - alert: SuspiciousProcess
        expr: |
          count_over_time({job="syslog"} |~ "/tmp/.*executed" [5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Suspicious process executed from /tmp"

Priority 3: Create Security Dashboard in Grafana

Dashboard panels to add:

1. fail2ban Bans Over Time - Graph of bans per hour/day
2. Currently Banned IPs - Table with IP, jail, ban time
3. SSH Login Attempts - Success vs failure ratio
4. Top Attacking IPs - Leaderboard of blocked IPs
5. Auth Log Stream - Live view of auth.log

Priority 4: Integrate fail2ban Metrics

Option A: Use fail2ban-exporter for Prometheus

# Install fail2ban-exporter
pip install fail2ban-prometheus-exporter

# Add to Prometheus scrape config
scrape_configs:
  - job_name: 'fail2ban'
    static_configs:
      - targets: ['localhost:9191']

Option B: Parse fail2ban.log via Alloy/Loki (already configured)

Priority 5: Consider CrowdSec (Future)

CrowdSec advantages over fail2ban:

• Community threat intelligence sharing
• Built-in Prometheus metrics
• Pre-built Grafana dashboards
• API-driven architecture
• Supports more attack types (web, API, etc.)

Migration path:

# Install CrowdSec
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | bash
apt install crowdsec crowdsec-firewall-bouncer-iptables

# Keep fail2ban running during transition
# Test CrowdSec in detection-only mode first

---

Memory Requirements

Configuration	Estimated RAM
Current (Grafana + Loki only)	~800MB
Minimal PLGT (no Tempo)	~2GB
Full PLGT stack	~3GB
PLGT + CrowdSec	~3.5GB

Recommendation: Upgrade to 8GB RAM for comfortable operation.

---

Quick Reference Commands

Check PLGT Status

cd /opt/plg-unified && docker-compose ps

Restart PLGT (when ready)

cd /opt/plg-unified && docker-compose down && docker-compose up -d

Check Memory

free -h

Monitor fail2ban

fail2ban-client status sshd
tail -f /var/log/fail2ban.log

---

Decision Log

Date	Decision	Reason
2026-02-08	Installed fail2ban	Immediate protection needed
2026-02-08	Deferred PLGT restart	Insufficient memory
2026-02-08	Documented plan	Future reference

---

Next review: When server RAM is upgraded Contact: Nhlanhla Mnyandu (nhlanhla@isutech.co.za)