PLGT Stack Troubleshooting Guide

Last Updated: 2026-01-05

Quick Diagnostics

# Check all services
docker compose ps

# Health check
./scripts/health-check.sh

# View recent logs
docker compose logs --tail=50

Common Issues

Service Won’t Start

Diagnosis:

# Check container logs
docker compose logs <service-name>

# Check exit code
docker inspect <container-name> --format='{{.State.ExitCode}}'

Exit Codes:

Code	Meaning	Solution
0	Normal exit	Check if intended
1	Application error	Check logs for details
137	Out of memory	Increase memory limit
139	Segmentation fault	Check configuration

Prometheus Not Scraping Targets

Diagnosis:

Go to http://localhost:9090/targets
Check error message for each target

Solutions:

Network Issue
Config Issue

# Verify target is reachable
docker exec prometheus ping target-host

# Test metrics endpoint
curl http://target-host:port/metrics

# Verify configuration
docker exec prometheus cat /etc/prometheus/prometheus.yml

# Validate config
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

Loki Not Receiving Logs

Diagnosis:

# Check Alloy status
docker compose logs alloy

# Verify Loki is ready
curl http://localhost:3100/ready

# Check Alloy UI for targets
# Open http://localhost:12345 in browser

Common Causes:

# Check log file permissions
ls -la /var/log/

# Alloy needs read access to log files

# In config.alloy, verify __path__
loki.source.file "syslog" {
  targets = [
    {__path__ = "/var/log/syslog", job = "syslog"},
  ]
}

# Check if Alloy can access Docker socket
docker exec alloy ls -la /var/run/docker.sock

Tempo Not Receiving Traces

Diagnosis:

# Check Tempo status
docker compose logs tempo

# Verify Tempo is ready
curl http://localhost:3200/ready

# Check if OTLP receiver is listening
curl http://localhost:3200/status

Common Causes:

Your application needs OpenTelemetry instrumentation to send traces.

# Verify your app is sending to correct endpoint
# OTLP gRPC: tempo:4317
# OTLP HTTP: tempo:4318

# Check Alloy logs for trace forwarding
docker compose logs alloy | grep -i trace

Grafana Dashboard Shows “No Data”

Diagnosis:

Go to Configuration → Data sources
Click Test on each data source

Solutions:

# Verify Prometheus is accessible
curl http://localhost:9090/api/v1/query?query=up

# Verify Loki is accessible
curl http://localhost:3100/loki/api/v1/labels

docker compose logs grafana

Alerts Not Firing

Diagnosis:

# Check alert rules in Prometheus
curl http://localhost:9090/api/v1/rules

# Check Alertmanager status
curl http://localhost:9093/api/v2/status

Solutions:

# Validate rule file
docker exec prometheus promtool check rules /etc/prometheus/rules/alerts.yml

docker compose logs alertmanager

High Memory Usage

Diagnosis:

# Check memory usage
docker stats --no-stream

# Check Prometheus storage
curl http://localhost:9090/api/v1/status/tsdb

Solutions:

Reduce Retention
Add Memory Limits

command:
  - '--storage.tsdb.retention.time=15d'

services:
  prometheus:
    deploy:
      resources:
        limits:
          memory: 4G

Disk Space Issues

Diagnosis:

# Check disk usage
df -h

# Check Docker disk usage
docker system df

Solutions:

# Clean up Docker
docker system prune -f

# Reduce retention (edit configs)
# Then restart
docker compose restart

Configuration Errors

Validation Commands:

docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

docker exec prometheus promtool check rules /etc/prometheus/rules/alerts.yml

docker exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml

Network Issues

Container Cannot Reach Other Containers

# Check network
docker network inspect plg-network

# Test connectivity
docker exec prometheus ping loki
docker exec grafana ping prometheus

Port Conflicts

# Check if port is in use
sudo lsof -i :3000
sudo lsof -i :9090

# Solution: Stop conflicting service or change port

Recovery Procedures

Full Stack Recovery

# Stop everything
docker compose down

# Redeploy
./scripts/deploy.sh deploy

Restore from Backup

# Stop services
docker compose down

# Restore configuration
cp -r backups/YYYYMMDD_HHMMSS/* .

# Restart
docker compose up -d

Reset Grafana Admin Password

docker exec grafana grafana-cli admin reset-admin-password newpassword

Debug Information

Collecting Logs for Support

mkdir -p debug-$(date +%Y%m%d)
docker compose ps > debug-$(date +%Y%m%d)/ps.txt
docker compose logs > debug-$(date +%Y%m%d)/logs.txt
docker stats --no-stream > debug-$(date +%Y%m%d)/stats.txt
./scripts/health-check.sh > debug-$(date +%Y%m%d)/health.txt

Log Locations

Service	Command
Prometheus	`docker compose logs prometheus`
Alertmanager	`docker compose logs alertmanager`
Loki	`docker compose logs loki`
Tempo	`docker compose logs tempo`
Alloy	`docker compose logs alloy`
Grafana	`docker compose logs grafana`
Blackbox Exporter	`docker compose logs blackbox-exporter`

Client Integration Issues

Client Logs Not Appearing in Loki

Diagnosis:

# Check if logs are being pushed from client
# On monitoring server, check Loki ingestion
curl -s http://localhost:3100/loki/api/v1/query \
  -G --data-urlencode 'query={job="umdoni-activity"}' | jq

Common Causes:

# In loki-config.yml, increase max age
limits_config:
  reject_old_samples_max_age: 1680h  # 70 days
  unordered_writes: true

# Test from client server
curl -v https://monitor.isutech.co.za/loki/api/v1/push

// Ensure X-Scope-OrgID header is set
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Content-Type: application/json',
    'X-Scope-OrgID: isutech'
]);

Client Metrics Not Scraped

Diagnosis:

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="umdoni-metrics")'

Common Causes:

Metrics Endpoint Down
SSL/TLS Issues

# Test from monitoring server
curl -v "https://umdoni.gov.za/api/metrics.php?token=xxx"

# In prometheus.yml, add TLS config
- job_name: 'client-metrics'
  scheme: https
  tls_config:
    insecure_skip_verify: true  # If needed

Next: Query Reference