PLGT Stack Query Reference

Last Updated: 2026-01-05

PromQL (Prometheus Query Language)

Basic Syntax

metric_name{label="value"}

Label Matchers

Operator	Description	Example
`=`	Equal	`{job="prometheus"}`
`!=`	Not equal	`{job!="test"}`
`=~`	Regex match	`{job=~"prom.*"}`
`!~`	Regex not match	`{job!~"test.*"}`

Aggregation Functions

Function	Description
`sum()`	Sum all values
`avg()`	Calculate average
`min()`	Find minimum
`max()`	Find maximum
`count()`	Count series
`topk(k, ...)`	Top K values
`bottomk(k, ...)`	Bottom K values

Rate Functions

Function	Use Case
`rate()`	Per-second rate over time range
`irate()`	Instant rate (last two points)
`increase()`	Total increase over range
`delta()`	Difference for gauges

System Metrics Queries

# CPU usage percentage
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# CPU by mode
sum by(mode) (irate(node_cpu_seconds_total[5m])) * 100

# Load averages
node_load1
node_load5
node_load15

# Memory usage percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Memory used (bytes)
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

# Memory available
node_memory_MemAvailable_bytes

# Disk usage percentage
(1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes)) * 100

# Disk free space
node_filesystem_avail_bytes{mountpoint="/"}

# Disk I/O rate
rate(node_disk_read_bytes_total[5m])
rate(node_disk_written_bytes_total[5m])

# Network receive rate
rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m])

# Network transmit rate
rate(node_network_transmit_bytes_total{device!~"lo|veth.*"}[5m])

# Network errors
rate(node_network_receive_errs_total[5m])

Container Metrics Queries

# Container CPU usage (%)
sum(rate(container_cpu_usage_seconds_total{name!=""}[5m])) by (name) * 100

# Container memory usage
container_memory_usage_bytes{name!=""}

# Container memory percentage
(container_memory_usage_bytes{name!=""} / container_spec_memory_limit_bytes{name!=""}) * 100

# Container network receive
rate(container_network_receive_bytes_total{name!=""}[5m])

# Container restart count
increase(container_restart_count{name!=""}[1h])

# Running containers
count(container_last_seen{name!=""})

Alerting Queries

# High CPU (>80% for 5 minutes)
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80

# Low disk space (<20% free)
(node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 20

# High memory (>90%)
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90

# Predict disk full in 24h
predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0

# Service down
up == 0

LogQL (Loki Query Language)

Basic Syntax

{label="value"} |= "search term"

Stream Selectors

Operator	Description	Example
`=`	Equal	`{job="nginx"}`
`!=`	Not equal	`{job!="test"}`
`=~`	Regex match	`{job=~"nginx.*"}`
`!~`	Regex not match	`{job!~".test."}`

Line Filters

Operator	Description	Example
`\|=`	Contains	`{job="syslog"} \|= "error"`
`!=`	Not contains	`{job="syslog"} != "debug"`
`\|~`	Regex match	`{job="syslog"} \|~ "fail(ed\|ure)"`
`!~`	Regex not match	`{job="syslog"} !~ "health"`

Log Queries

# All logs from a job
{job="containerlogs"}

# Logs containing "error"
{job="syslog"} |= "error"

# Exclude pattern
{job="nginx"} != "health"

# Multiple conditions
{job="syslog"} |= "error" != "expected"

# Case insensitive
{job="syslog"} |~ "(?i)error"

# Parse JSON
{job="app"} | json

# Filter by parsed field
{job="app"} | json | level="error"

# Format output
{job="app"} | json | line_format "{{.level}}: {{.message}}"

# Parse with regex
{job="nginx"} | regexp `(?P<ip>\d+\.\d+\.\d+\.\d+)`

# Parse key-value
{job="app"} | logfmt

Log Metrics

# Count log lines per minute
count_over_time({job="syslog"}[1m])

# Count errors per minute
count_over_time({job="syslog"} |= "error" [1m])

# Rate of errors per second
rate({job="syslog"} |= "error" [5m])

# Bytes per minute
sum(bytes_over_time({job="nginx"}[1m]))

# Top 5 error sources
topk(5, sum by(source) (count_over_time({job="syslog"} |= "error" [1h])))

# Aggregation by label
sum by(level) (count_over_time({job="app"} | json [1h]))

Blackbox Exporter Queries

Website Availability

# Probe success (1 = up, 0 = down)
probe_success{job="blackbox-http"}

# All sites that are DOWN
probe_success{job="blackbox-http"} == 0

# Response time (seconds)
probe_http_duration_seconds{job="blackbox-http"}

# HTTP status code
probe_http_status_code{job="blackbox-http"}

SSL Certificate Monitoring

# Days until SSL expiry
(probe_ssl_earliest_cert_expiry - time()) / 86400

# Certificates expiring within 30 days
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 30

# SSL certificate issuer
probe_ssl_last_chain_info{job="blackbox-http"}

Performance Metrics

# DNS lookup time
probe_dns_lookup_time_seconds

# TCP connection time
probe_connect_duration_seconds

# TLS handshake time
probe_tls_handshake_seconds

# Total request duration
probe_http_duration_seconds

TraceQL (Tempo Query Language)

Basic Syntax

{ <attribute> <operator> <value> }

Common Operators

Operator	Description	Example
`=`	Equal	`{ resource.service.name = "api" }`
`!=`	Not equal	`{ status != ok }`
`=~`	Regex match	`{ name =~ "HTTP.*" }`
`>`	Greater than	`{ duration > 1s }`
`<`	Less than	`{ duration < 100ms }`

Trace Queries

# Find all traces from a service
{ resource.service.name = "my-api" }

# Multiple services
{ resource.service.name =~ "api|frontend" }

# Slow traces (>1 second)
{ duration > 1s }

# Very slow traces (>5 seconds)
{ duration > 5s }

# Fast traces
{ duration < 100ms }

# Error traces
{ status = error }

# Successful traces
{ status = ok }

# By HTTP method
{ span.http.method = "POST" }

# By HTTP status code
{ span.http.status_code >= 400 }

# By URL
{ span.http.url =~ ".*api/users.*" }

Complex Trace Queries

# Slow POST requests with errors
{ span.http.method = "POST" && duration > 1s && status = error }

# Database queries taking too long
{ span.db.system = "mysql" && duration > 500ms }

# Traces from specific client
{ resource.client.name = "umdoni-municipality" }

Client-Specific Queries

Umdoni Municipality

# All Umdoni activity
{job="umdoni-activity"}

# Failed logins
{job="umdoni-activity"} | json | action="Login Failed"

# Successful logins
{job="umdoni-activity"} | json | action="Login"

# Activity by user
{job="umdoni-activity"} | json | username="admin"

# Count failed logins per hour
sum(count_over_time({job="umdoni-activity"} | json | action="Login Failed" [1h]))

# Umdoni site status
probe_success{instance="https://umdoni.gov.za"}

# SSL certificate expiry
(probe_ssl_earliest_cert_expiry{instance="https://umdoni.gov.za"} - time()) / 86400

# Response time
probe_http_duration_seconds{instance="https://umdoni.gov.za", phase="transfer"}

# Database connectivity
app_database_up{client="umdoni-municipality"}

# Disk usage
app_disk_usage_percent{client="umdoni-municipality"}

# Active sessions
app_active_sessions{client="umdoni-municipality"}

Quick Reference Card

PromQL Essentials

metric_name                           # Basic metric
metric_name{label="value"}            # With label filter
rate(metric_name[5m])                 # Rate for counters
sum by(label) (metric_name)           # Aggregation
metric_name > 80                      # Comparison

LogQL Essentials

{job="name"}                          # Stream selector
{job="name"} |= "search"              # Contains filter
{job="name"} | json                   # Parse JSON
{job="name"} | json | level="error"   # Filter parsed
count_over_time({job="name"}[5m])     # Count logs

TraceQL Essentials

{ resource.service.name = "api" }     # By service
{ duration > 1s }                     # By duration
{ status = error }                    # By status
{ span.http.method = "POST" }         # By HTTP method

Blackbox Essentials

probe_success                         # Site up/down
probe_http_duration_seconds           # Response time
(probe_ssl_earliest_cert_expiry - time()) / 86400  # SSL days left

Dashboard Panel Examples

CPU Gauge

100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Memory Time Series

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Error Rate Graph

sum(rate({job=~".+"} |= "error" [1m]))

Container Status Table

count by (name) (container_last_seen{name!=""})

Previous: Troubleshooting Guide