PLGT Unified Observability Stack

PLGT Unified Observability Stack

Service: PLGT Stack (Prometheus, Loki, Grafana, Tempo) Server: Hetzner (46.224.40.5) Access: monitor.isutech.co.za Last Updated: 2026-01-07

---

Overview

The PLGT Stack is iSu Technologies’ unified monitoring and observability solution, providing complete visibility across metrics, logs, and traces:

• Metrics Collection - Real-time performance data (Prometheus)
• Log Aggregation - Centralized log management (Loki)
• Distributed Tracing - Request flow analysis (Tempo)
• Unified Dashboards - Single pane of glass (Grafana)
• Multi-Channel Alerting - Slack, Telegram, Email notifications
• SLA Tracking - Uptime and compliance monitoring
• Automated Reports - Weekly status reports and analytics

The Four Pillars of Observability

PLGT implements all four pillars: Metrics (Prometheus), Logs (Loki), Traces (Tempo), and Visualization (Grafana).

---

Architecture

graph TB
    subgraph "Visualization Layer"
        GF[Grafana<br>:3004]
    end

    subgraph "Storage & Processing"
        PR[Prometheus<br>:9090]
        LK[Loki<br>:3100]
        TM[Tempo<br>:3200]
        AM[Alertmanager<br>:9093]
    end

    subgraph "Notification Channels"
        SL[Slack<br>#alerts #critical]
        TG[Telegram<br>Bot Notifications]
        EM[Email<br>SMTP]
    end

    subgraph "Data Collection"
        NE[Node Exporter<br>:9100]
        CA[cAdvisor<br>:8081]
        BB[Blackbox Exporter<br>:9115]
        AL[Alloy<br>:12345]
    end

    subgraph "External Targets"
        SITES[iSuTech Sites]
        CLIENTS[Client Sites]
    end

    NE --> PR
    CA --> PR
    BB --> PR
    AL --> LK
    AL --> TM
    PR --> AM
    PR --> GF
    LK --> GF
    TM --> GF
    AM --> GF
    SITES --> BB
    CLIENTS --> BB
    CLIENTS -.->|Metrics| PR
    CLIENTS -.->|Logs| LK
    AM --> SL
    AM --> TG
    AM --> EM

---

Component Summary

Core Components

Component	Purpose	Port	Access
Grafana	Visualization & Dashboards	3004	monitor.isutech.co.za
Prometheus	Metrics Collection & Storage	9090	Internal only
Loki	Log Aggregation & Indexing	3100	Internal / Proxy
Tempo	Distributed Tracing	3200	Internal only
Alertmanager	Alert Routing & Notifications	9093	Internal only

Data Collectors

Component	Purpose	Port	Access
Alloy	Unified Collector (Logs + Traces)	12345	Internal only
Node Exporter	Host System Metrics	9100	Internal only
cAdvisor	Container Metrics	8081	Internal only
Blackbox Exporter	HTTP/TCP/SSL Probes	9115	Internal only

---

Quick Access

View Dashboards

1. Open monitor.isutech.co.za
2. Login with your credentials
3. Navigate to Dashboards → iSu Technologies

Check Alerts

1. Open Grafana
2. Go to Alerting → Alert rules
3. View firing and pending alerts

Search Logs

1. Open Grafana → Explore
2. Select Loki datasource
3. Enter your LogQL query

Explore Traces

1. Open Grafana → Explore
2. Select Tempo datasource
3. Search by trace ID or service

---

Pre-configured Dashboards

Core Dashboards

Dashboard	UID	Purpose
iSuTech Services	isutech-services	Main overview - websites, APIs, SSL status
Infrastructure Overview	infrastructure	CPU, Memory, Disk, Network
Container Metrics	container-metrics	Docker container resources
Logs Explorer	logs-explorer	Search logs via Loki
Tempo - Distributed Tracing	tempo-traces	Trace search & analysis
Alloy - Unified Collector	alloy-metrics	Collector health & metrics

Operations Dashboards

Dashboard	UID	Purpose
SLA Tracking	sla-tracking	Uptime percentages, SLA compliance, response times
Cost & Resource Tracking	plgt-cost-tracking	Infrastructure costs, storage usage, monthly estimates

Client Dashboards

Dashboard	UID	Purpose
Umdoni Municipality	umdoni-municipality	Client system & database metrics
Umdoni User Activity	umdoni-activity	Login activity, session events, user actions
Umdoni Security	umdoni-security	Failed logins, brute force detection, security events

---

Monitored Targets

iSuTech Production Sites

Site	Type	Monitoring
monitor.isutech.co.za	Grafana	HTTP probe
docs.isutech.co.za	Documentation	HTTP probe (auth-protected)
api.isutech.co.za	API Gateway	HTTP probe
sace.isutech.co.za	SACE Portal	HTTP probe
sansa.isutech.co.za	SANSA Portal	HTTP probe
tumi.isutech.co.za	Tumi App	HTTP probe
thrivesend.isutech.co.za	ThriveSend	HTTP probe
prospectiq.isutech.co.za	ProspectIQ	HTTP probe

Client Sites

Client	Site	Monitoring
Umdoni Municipality	umdoni.gov.za	HTTP probe, metrics.php, activity logs to Loki

---

Alert Categories

Category	Examples	Channels
Critical	Instance down, CPU >95%, Disk >95%	Email + Slack (#critical) + Telegram
Warning	High CPU (>80%), High memory, SSL expiry <30d	Email + Slack (#alerts)
Security	Failed login spike, Brute force detection	Email + Slack (#critical) + Telegram
Info	Container restarts, Configuration reloads	Slack (#alerts) only

Notification Channels

Channel	Purpose	Configuration
Email	Primary notifications	SMTP via Gmail
Slack	Team collaboration	#alerts (standard), #critical (urgent)
Telegram	Mobile notifications	Bot API with direct messaging
Microsoft Teams	Enterprise integration	Webhook (optional)

Alert Routing

Alerts are routed based on severity and client labels. Critical alerts trigger all channels immediately, while info alerts only go to Slack to avoid email spam.

---

Client Integration

The PLGT stack supports monitoring external client sites through:

1. HTTP Probes - Website availability via Blackbox Exporter
2. Metrics Endpoint - PHP metrics exporter for shared hosting
3. Log Shipping - Activity logs shipped to Loki via PHP cron

See the Client Integration section below for implementation details.

---

Documentation

Document	Description
Installation Guide	Full deployment instructions
Configuration Reference	All configuration options
Alerting Guide	Multi-channel notification setup
Operations Guide	Daily operations and maintenance
Troubleshooting	Common issues and solutions
Query Reference	PromQL, LogQL, and TraceQL examples
Runbooks	Incident response procedures
Client Onboarding	Adding new clients to monitoring

---

Quick Commands

# Start stack
docker compose up -d

# View status
docker compose ps

# View logs
docker compose logs -f [service]

# Restart service
docker compose restart [service]

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check Loki ready
curl http://localhost:3100/ready

# Check Tempo ready
curl http://localhost:3200/ready

# Reload Alertmanager config
docker compose exec alertmanager kill -HUP 1

# Test Slack notification
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test alert from PLGT"}' \
  $SLACK_WEBHOOK_URL

---

Related Documentation

• Hetzner Server Management

---

Document Version: 2.2 Created: 2025-12-31 Updated: 2026-01-07 Author: iSu Technologies