Server Monitoring Quick Reference

Server Monitoring Quick Reference

Your comprehensive go-to reference for monitoring the iSu Technologies production server. Commands organized for quick access during daily operations and emergencies.

Quick Navigation

Use Ctrl+F (or Cmd+F on Mac) to quickly find specific commands or services.

---

🎯 Quick Reference Card

The most frequently used commands for daily server monitoring:

Task	Command	Use When
Memory overview	free -h	Checking available memory
Top processes	htop	Finding resource-hungry processes
Disk space	df -h	Checking storage usage
System load	uptime	Checking server health
Service status	systemctl status <service>	Verifying service is running
Service logs	journalctl -u <service> -n 50	Troubleshooting errors
Restart service	systemctl restart <service>	After config changes
DB connections	sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"	Checking database load

---

💾 Memory Monitoring

Quick Memory Overview

Check current memory usage:

free -h

Example Output:

              total        used        free      shared  buff/cache   available
Mem:           15Gi       8.2Gi       1.1Gi       234Mi       6.1Gi       6.5Gi
Swap:         2.0Gi       512Mi       1.5Gi

Understanding the Output

• used: Actual memory used by applications
• available: Memory available for new applications (includes reclaimable cache)
• buff/cache: Memory used for disk caching (can be freed if needed)

---

Detailed Memory Statistics

Get comprehensive memory information:

cat /proc/meminfo

Show memory in megabytes (easier to read):

free -m

Watch memory usage in real-time (updates every 2 seconds):

watch -n 2 free -h

When to Use

Use watch when monitoring a deployment or investigating a memory leak. Press Ctrl+C to exit.

---

Virtual Memory Statistics

Monitor memory, swap, and I/O (5 samples, 1 second apart):

vmstat 1 5

Example Output:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 524288 1048576 512000 6291456    0    0     1     5    1    2  5  2 93  0  0

Key Columns

• r: Processes waiting for CPU
• free: Free memory (KB)
• si/so: Swap in/out (high values = memory pressure)
• us/sy/id: CPU usage (user/system/idle)

---

🔍 Process Monitoring

Interactive Process Viewer

Launch interactive process monitor:

htop

htop Navigation

• F5: Tree view (shows parent/child processes)
• F6: Sort by different columns
• F9: Kill a process
• F10 or q: Quit

If htop not installed:

sudo apt install htop

---

Top Memory-Consuming Processes

Show top 20 processes by memory usage:

ps aux --sort=-%mem | head -20

Example Output:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres  1234  2.1 15.3 2456780 2401234 ?     Ss   Jan30  45:12 postgres: main
node      5678  1.8 12.4 1834560 1945678 ?     Ssl  Jan30  32:45 node /var/www/sansa-frontend

Column Meanings

• %MEM: Percentage of physical memory used
• VSZ: Virtual memory size (KB)
• RSS: Resident set size - actual physical memory used (KB)

---

Top CPU-Consuming Processes

Show top 20 processes by CPU usage:

ps aux --sort=-%cpu | head -20

Filter for specific application type:

# Python processes
ps aux | grep python --sort=-%mem | head -10

# Node.js processes
ps aux | grep node --sort=-%cpu | head -10

---

System Load Average

Check server load and uptime:

uptime

Example Output:

11:45:23 up 45 days,  3:12,  2 users,  load average: 0.52, 0.48, 0.45

Load Average Interpretation

On a 4-core CPU:

• < 4.0: Healthy
• 4.0 - 8.0: Moderate load
• > 8.0: High load, investigate!

---

💿 Disk & Swap Monitoring

Disk Usage Overview

Check disk space across all mounted filesystems:

df -h

Example Output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        80G   53G   24G  69% /
/dev/sdb1       200G  142G   48G  75% /var

Disk Space Alerts

• > 80% usage: Start planning cleanup
• > 90% usage: Take immediate action
• > 95% usage: Critical - services may fail

Check specific directory sizes:

du -sh /var/www/*
du -sh /var/log/*

---

Swap Usage

Check swap usage:

swapon --show

Example Output:

NAME      TYPE SIZE USED PRIO
/swapfile file   2G 512M   -2

Swap Usage Guidelines

• 0-25% used: Healthy
• 25-75% used: Monitor memory usage
• > 75% used: Memory pressure - consider adding RAM

---

Disk I/O Statistics

Monitor disk I/O (5 samples, 1 second apart):

iostat -x 1 5

Example Output:

Device  r/s  w/s  rkB/s  wkB/s  await  %util
sda    12.3 45.6  234.5  1234.5  5.23   8.45

Key Metrics

• r/s / w/s: Reads/writes per second
• await: Average wait time (ms) - high values indicate bottleneck
• %util: Percentage of time device was busy

---

🗄️ PostgreSQL Monitoring

Active Queries

Show all active queries (excluding idle connections):

sudo -u postgres psql -c "SELECT pid, state, query FROM pg_stat_activity WHERE state != 'idle';"

Example Output:

  pid  |        state        |           query
-------+---------------------+----------------------------
 12345 | active              | SELECT * FROM providers...
 12346 | idle in transaction | BEGIN;

Long-Running Queries

If you see queries running for > 30 seconds, investigate performance issues.

---

Connection Statistics

Count total active connections:

sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"

Show connections per database:

sudo -u postgres psql -c "SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;"

Example Output:

    datname     | count
----------------+-------
 sansa_production  |    12
 sace_production   |     8
 tumi_production   |     5

---

Configuration Settings

Check maximum connections allowed:

sudo -u postgres psql -c "SHOW max_connections;"

Check shared buffers (memory for caching):

sudo -u postgres psql -c "SHOW shared_buffers;"

Typical Values

• max_connections: 100-200 for production
• shared_buffers: 25% of system RAM

---

Database Size

Check specific database size:

sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('sansa_production'));"

Check all database sizes:

sudo -u postgres psql -c "SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database ORDER BY pg_database_size(datname) DESC;"

Example Output:

      datname       | pg_size_pretty
--------------------+----------------
 sansa_production   | 1247 MB
 sace_production    | 856 MB
 tumi_production    | 234 MB

---

Table Sizes

Check table sizes in a database:

sudo -u postgres psql -d sansa_production -c "SELECT tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC LIMIT 10;"

---

⚙️ Service Management

Check Service Status

Individual services:

systemctl status autoslip
systemctl status sansa-backend
systemctl status sace-backend
systemctl status tumi-backend
systemctl status prospectiq-api
systemctl status postgresql
systemctl status redis
systemctl status nginx

Check multiple services at once:

systemctl status sansa-backend tumi-backend sace-backend postgresql redis nginx

Check all failed services:

systemctl --failed

---

View Service Logs

Last 50 lines:

journalctl -u autoslip -n 50 --no-pager
journalctl -u sansa-backend -n 50 --no-pager

Follow logs in real-time:

journalctl -u <service> -f

Logs from specific time period:

# Today's logs
journalctl -u sansa-backend --since today

# Last hour
journalctl -u sansa-backend --since "1 hour ago"

# Specific date range
journalctl -u sansa-backend --since "2026-02-01" --until "2026-02-02"

Filter by log level:

# Errors only
journalctl -u sansa-backend -p err

# Warnings and errors
journalctl -u sansa-backend -p warning

---

PM2 Process Management

Check all PM2 processes:

pm2 status

View specific application logs:

pm2 logs sansa-frontend
pm2 logs thrivesend
pm2 logs isutech-backend

Monitor PM2 dashboard:

pm2 monit

---

🌐 Network Monitoring

Active Connections

Show all listening ports and established connections:

ss -tuln

Example Output:

Netid State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
tcp   LISTEN 0      128    0.0.0.0:80           0.0.0.0:*
tcp   LISTEN 0      128    0.0.0.0:443          0.0.0.0:*
tcp   LISTEN 0      128    0.0.0.0:22           0.0.0.0:*
tcp   LISTEN 0      128    127.0.0.1:8001       0.0.0.0:*

Port Reference

• 22: SSH
• 80: HTTP
• 443: HTTPS
• 8001: SACE Backend
• 8003: SANSA Backend
• 8004: Tumi Backend

---

Connection Statistics Summary

Show connection count by state:

ss -s

Example Output:

Total: 1234
TCP:   567 (estab 234, closed 123, orphaned 0, timewait 45)

---

Check Specific Port

See what’s listening on a specific port:

sudo ss -tulpn | grep :8001
sudo ss -tulpn | grep :5432  # PostgreSQL
sudo ss -tulpn | grep :6379  # Redis

---

🚨 Emergency Procedures

High Memory Usage Response

Emergency Protocol

Follow these steps when memory usage exceeds 90%

Step 1: Identify memory hogs

ps aux --sort=-%mem | head -10

Step 2: Check for memory leaks

# Monitor memory over time
watch -n 5 'ps aux --sort=-%mem | head -10'

Step 3: Clear page cache (SAFE - won’t affect running processes)

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

What This Does

• Flushes filesystem buffers to disk (sync)
• Drops dentries, inodes, and page cache
• Safe to run - doesn’t kill processes or remove data
• Temporary fix - memory will fill again with cache

Step 4: Check for zombie processes

ps aux | grep -w Z

Step 5: Restart problematic service

# If a specific service is using too much memory
sudo systemctl restart <service-name>

---

High CPU Usage Response

Step 1: Identify CPU hogs

ps aux --sort=-%cpu | head -10

Step 2: Check system load

uptime

Step 3: Monitor in real-time

htop
# Press F6 to sort by CPU%

Step 4: Investigate specific process

# Get detailed info about a process
ps -p <PID> -o pid,cmd,%mem,%cpu,etime

# Check what files the process has open
sudo lsof -p <PID>

---

Disk Space Emergency

Step 1: Find what’s using space

df -h
du -sh /var/www/* | sort -h
du -sh /var/log/* | sort -h

Step 2: Clear old logs

# Clear systemd journal logs older than 7 days
sudo journalctl --vacuum-time=7d

# Clear old nginx logs
sudo find /var/log/nginx -name "*.log.*" -mtime +30 -delete

Step 3: Find large files

sudo find /var -type f -size +100M -exec ls -lh {} \; | sort -k5 -h

Step 4: Check for deleted files still held open

sudo lsof | grep deleted | grep -v "/tmp"

---

Database Connection Pool Exhausted

Step 1: Check active connections

sudo -u postgres psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

Step 2: Find long-running queries

sudo -u postgres psql -c "SELECT pid, now() - query_start AS duration, state, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC;"

Step 3: Kill specific connection (if needed)

# Be VERY careful with this command
sudo -u postgres psql -c "SELECT pg_terminate_backend(<PID>);"

Step 4: Restart application to release connections

sudo systemctl restart sansa-backend
# or
pm2 restart sansa-frontend

---

📊 Health Check Commands

Full System Health Check

Run this comprehensive check when investigating issues:

#!/bin/bash
# Quick Health Check Script

echo "=== SYSTEM HEALTH CHECK ==="
echo ""
echo "--- Uptime & Load ---"
uptime
echo ""
echo "--- Memory ---"
free -h
echo ""
echo "--- Disk Space ---"
df -h | grep -E '(Filesystem|/dev/)'
echo ""
echo "--- Top 5 Memory Processes ---"
ps aux --sort=-%mem | head -6
echo ""
echo "--- Top 5 CPU Processes ---"
ps aux --sort=-%cpu | head -6
echo ""
echo "--- Service Status ---"
systemctl status nginx --no-pager | grep Active
systemctl status postgresql --no-pager | grep Active
systemctl status redis --no-pager | grep Active
echo ""
echo "--- PostgreSQL Connections ---"
sudo -u postgres psql -c "SELECT count(*) as total_connections FROM pg_stat_activity;" 2>/dev/null
echo ""
echo "--- Failed Services ---"
systemctl --failed --no-pager

Save and run:

nano ~/health-check.sh
chmod +x ~/health-check.sh
~/health-check.sh

---

🔔 Alert Thresholds

Use these thresholds to determine when to take action:

Memory

Metric	Normal	Warning	Critical	Action
Available Memory	> 20%	10-20%	< 10%	Restart services, investigate leaks
Swap Usage	< 25%	25-75%	> 75%	Add RAM, optimize applications

CPU

Metric	Normal	Warning	Critical	Action
Load Average (4 cores)	< 4.0	4.0-8.0	> 8.0	Scale up, optimize code
CPU Wait (wa)	< 5%	5-15%	> 15%	Check disk I/O, optimize queries

Disk

Metric	Normal	Warning	Critical	Action
Disk Usage	< 80%	80-90%	> 90%	Clean logs, archive data
Disk I/O Await	< 10ms	10-50ms	> 50ms	Check slow queries, optimize

Database

Metric	Normal	Warning	Critical	Action
Active Connections	< 50% max	50-80% max	> 80% max	Scale connections, check leaks
Long Queries	< 5s	5-30s	> 30s	Optimize queries, add indexes

---

📚 Related Documentation

• Hetzner Server Guide - Complete operations guide
• Development Workflow - Deployment procedures
• Hetzner Server Management - Infrastructure details

---

💡 Tips & Best Practices

Daily Monitoring Routine

Run these commands every morning:

free -h              # Check memory
df -h                # Check disk
uptime               # Check load
systemctl --failed   # Check for failures

Before Deployments

Create a baseline snapshot:

~/health-check.sh > ~/pre-deployment-$(date +%Y%m%d).txt

Save Frequent Commands as Aliases

Add to ~/.bashrc:

alias memcheck='free -h && ps aux --sort=-%mem | head -10'
alias diskcheck='df -h && du -sh /var/www/* | sort -h'
alias services='systemctl status nginx postgresql redis --no-pager'

---

Last Updated: 02/01/2026 | Document Owner: Operations Team | Version: 1.0