PLGT Stack Configuration Reference

PLGT Stack Configuration Reference

Last Updated: 2026-01-07

---

Directory Structure

plg-unified/
├── docker-compose.yml          # Service definitions
├── .env                        # Environment variables
├── prometheus/
│   ├── prometheus.yml          # Prometheus config
│   └── rules/
│       ├── alerts.yml          # Infrastructure alerts
│       └── umdoni-alerts.yml   # Client-specific alerts
├── alertmanager/
│   ├── alertmanager.yml        # Alertmanager config
│   └── templates/              # Notification templates
│       ├── email.tmpl          # HTML email templates
│       ├── slack.tmpl          # Slack message templates
│       └── telegram.tmpl       # Telegram message templates
├── loki/
│   ├── loki-config.yml         # Loki config
│   ├── runtime-config.yml      # Per-stream retention (optional)
│   └── rules/
│       └── umdoni/             # Loki alert rules
│           └── activity-alerts.yml
├── alloy/
│   └── config.alloy            # Unified collector config
├── tempo/
│   └── tempo.yaml              # Distributed tracing config
├── blackbox/
│   └── blackbox.yml            # HTTP/TCP probe modules
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/        # Auto-provisioned sources
│   │   ├── dashboards/         # Dashboard provisioning
│   │   └── access-control/     # Client access policies
│   └── dashboards/             # Dashboard JSON files
│       ├── isutech-services.json
│       ├── umdoni-activity.json
│       ├── sla-tracking.json
│       └── cost-tracking.json
├── clients/
│   └── umdoni/                 # Client integration templates
│       ├── metrics.php
│       └── loki-shipper.php
├── scripts/
│   ├── deploy.sh               # Deployment script
│   ├── health-check.sh         # Health check script
│   ├── disk-cleanup.sh         # Automated cleanup
│   └── reports/
│       └── weekly-report.py    # Automated weekly reports
├── prometheus/rules/
│   ├── alerts.yml              # Infrastructure alerts
│   ├── umdoni-alerts.yml       # Client-specific alerts
│   └── self-monitoring.yml     # Stack health alerts
└── templates/
    └── client-onboarding/      # New client setup templates

---

Environment Variables

Edit .env to configure the stack:

Variable	Description	Default
GRAFANA_ADMIN_USER	Admin username	admin
GRAFANA_ADMIN_PASSWORD	Admin password	isutech2025
GRAFANA_ROOT_URL	External URL	http://localhost:3000
SMTP_ENABLED	Enable email alerts	false
SMTP_HOST	SMTP server	smtp.gmail.com:587
SMTP_USER	SMTP username	-
SMTP_PASSWORD	SMTP password	-
SMTP_FROM	From address	noreply@isutech.co.za

---

Prometheus Configuration

Main Configuration

File: prometheus/prometheus.yml

global:
  scrape_interval: 15s          # How often to scrape
  evaluation_interval: 15s       # How often to evaluate rules

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Adding Scrape Targets

To monitor additional services:

scrape_configs:
  # Your application
  - job_name: 'my-app'
    static_configs:
      - targets: ['app-server:8080']
        labels:
          app: 'my-application'
          environment: 'production'

Storage Settings

In docker-compose.yml:

command:
  - '--storage.tsdb.retention.time=30d'   # Keep 30 days
  - '--storage.tsdb.retention.size=10GB'  # Max 10GB

---

Alert Rules Configuration

File: prometheus/rules/alerts.yml

Alert Structure

groups:
  - name: group-name
    rules:
      - alert: AlertName
        expr: <PromQL expression>
        for: <duration>
        labels:
          severity: critical|warning|info
        annotations:
          summary: "Short description"
          description: "Detailed description"

Default Alert Thresholds

Alert	Condition	Severity
InstanceDown	Service unreachable for 1m	critical
HighCpuUsage	CPU > 80% for 5m	warning
CriticalCpuUsage	CPU > 95% for 2m	critical
HighMemoryUsage	Memory > 80% for 5m	warning
HighDiskUsage	Disk > 80% for 5m	warning
DiskWillFillIn24Hours	Predicted full	warning

Custom Alert Example

- alert: ApplicationErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High error rate on {{ $labels.instance }}"
    description: "Error rate is {{ $value | printf \"%.1f\" }}%"

---

Alertmanager Configuration

File: alertmanager/alertmanager.yml

Notification Routing

route:
  receiver: 'isutech-team'
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: 'isutech-critical'
      repeat_interval: 1h

Email Configuration

receivers:
  - name: 'isutech-team'
    email_configs:
      - to: 'team@isutech.co.za'
        send_resolved: true
        headers:
          Subject: '[iSuTech Alert] {{ .GroupLabels.alertname }}'

Slack Configuration

global:
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

receivers:
  - name: 'isutech-team'
    slack_configs:
      - channel: '#alerts'
        send_resolved: true
        username: 'PLGT Alertmanager'
        icon_emoji: ':warning:'
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'
        color: '{{ template "slack.color" . }}'

  - name: 'isutech-critical-multichannel'
    slack_configs:
      - channel: '#critical'
        send_resolved: true
        username: 'PLGT CRITICAL'
        icon_emoji: ':rotating_light:'

Telegram Configuration

receivers:
  - name: 'isutech-critical-multichannel'
    telegram_configs:
      - bot_token: 'YOUR_BOT_TOKEN'  # From @BotFather
        chat_id: 123456789           # Your chat/group ID
        parse_mode: 'HTML'
        message: '{{ template "telegram.message" . }}'
        send_resolved: true

Getting Telegram Chat ID

1. Create bot via @BotFather → get token
2. Send message to your bot
3. Visit: https://api.telegram.org/bot<TOKEN>/getUpdates
4. Find your chat_id in the response

Custom Alert Templates

Templates are stored in alertmanager/templates/:

email.tmpl

Professional HTML email with severity colors, alert details, and action buttons.

slack.tmpl

Rich Slack messages with color-coded severity and structured fields.

telegram.tmpl

HTML-formatted Telegram messages with emoji indicators.

---

Loki Configuration

File: loki/loki-config.yml

Retention Settings

limits_config:
  retention_period: 720h              # 30 days log retention
  reject_old_samples: true
  reject_old_samples_max_age: 1680h   # Accept logs up to 70 days old
  unordered_writes: true              # Allow out-of-order log ingestion

Query Limits

limits_config:
  max_entries_limit_per_query: 5000
  max_query_series: 500
  max_query_parallelism: 32

Ruler Configuration (Log-based Alerts)

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /tmp/loki/rules
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Per-Stream Retention (Optional)

File: loki/runtime-config.yml

Different retention periods can be configured per log stream:

overrides:
  isutech:
    retention_period: 720h  # 30 days default
    retention_stream:
      # Security logs - keep 90 days
      - selector: '{level="error"}'
        priority: 1
        period: 2160h
      # Activity logs - keep 90 days
      - selector: '{action_type=~"login|logout|failed_login"}'
        priority: 2
        period: 2160h
      # Application logs - keep 30 days
      - selector: '{level="info"}'
        priority: 3
        period: 720h

Enable in loki-config.yml:

runtime_config:
  file: /etc/loki/runtime-config.yml
  period: 10m

---

Alloy Configuration

Alloy is the unified collector replacing Promtail, handling both logs and traces.

File: alloy/config.alloy

Docker Container Logs

// Discover Docker containers
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

// Collect container logs
loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.containers.targets
  forward_to = [loki.write.default.receiver]
}

// Send logs to Loki
loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

System Logs

loki.source.file "syslog" {
  targets = [
    {__path__ = "/var/log/syslog", job = "syslog"},
    {__path__ = "/var/log/auth.log", job = "auth"},
  ]
  forward_to = [loki.write.default.receiver]
}

OTLP Trace Collection

// Receive OTLP traces
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }

  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

// Forward traces to Tempo
otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls { insecure = true }
  }
}

---

Tempo Configuration

File: tempo/tempo.yaml

Basic Configuration

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

compactor:
  compaction:
    block_retention: 720h  # 30 days

---

Blackbox Exporter Configuration

File: blackbox/blackbox.yml

HTTP Probe Modules

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200, 201, 202, 204]
      follow_redirects: true
      preferred_ip_protocol: ip4

  http_2xx_ssl:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 201, 202, 204]
      fail_if_ssl: false
      fail_if_not_ssl: true

  http_auth_protected:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 401, 403]

  tcp_connect:
    prober: tcp
    timeout: 5s

Adding HTTP Probes to Prometheus

scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://monitor.isutech.co.za
          - https://umdoni.gov.za
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

---

Grafana Configuration

Data Sources

Auto-provisioned from grafana/provisioning/datasources/:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

  - name: Loki
    type: loki
    url: http://loki:3100

  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogs:
        datasourceUid: loki
        tags: ['job', 'container']

Dashboard Provisioning

File: grafana/provisioning/dashboards/dashboards.yml

apiVersion: 1
providers:
  - name: 'iSuTech Dashboards'
    folder: 'iSu Technologies'
    type: file
    options:
      path: /var/lib/grafana/dashboards

---

Applying Changes

Reload Without Restart

Prometheus

curl -X POST http://localhost:9090/-/reload

Alertmanager

curl -X POST http://localhost:9093/-/reload

Full Restart

docker compose restart

Specific Service

docker compose restart prometheus
docker compose restart alloy

---

Client Integration

PHP Metrics Exporter

For shared hosting clients, deploy a PHP metrics endpoint:

<?php
// metrics.php - Prometheus metrics for PHP applications
define('METRICS_TOKEN', 'your_secret_token');

header('Content-Type: text/plain; charset=utf-8');
if (($_GET['token'] ?? '') !== METRICS_TOKEN) {
    http_response_code(403);
    die("# Access denied\n");
}

// Database connectivity
try {
    $pdo = new PDO($dsn, $user, $pass);
    echo "app_database_up 1\n";
} catch (Exception $e) {
    echo "app_database_up 0\n";
}

// Disk usage
$free = disk_free_space('/');
$total = disk_total_space('/');
echo "app_disk_usage_percent " . round((($total - $free) / $total) * 100, 2) . "\n";

Activity Log Shipper

For shipping application logs to Loki via cron:

<?php
// loki-shipper.php - Ship MySQL logs to Loki
define('LOKI_URL', 'https://monitor.isutech.co.za/loki/api/v1/push');
define('LOKI_TENANT', 'isutech');

// Fetch new logs and build Loki payload
$logs = $pdo->query("SELECT * FROM logs WHERE id > $lastId");
$payload = ['streams' => [['stream' => $labels, 'values' => $entries]]];

// Send to Loki with tenant header
$ch = curl_init(LOKI_URL);
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => json_encode($payload),
    CURLOPT_HTTPHEADER => [
        'Content-Type: application/json',
        'X-Scope-OrgID: ' . LOKI_TENANT
    ]
]);

See clients/umdoni/ for complete implementation examples.

---

Next: Operations Guide