Hetzner Server Operations Guide

Hetzner Server Operations Guide

Complete guide to managing the iSu Technologies Hetzner VPS server.

---

🖥️ Server Information

Server Details

• Provider: Hetzner
• IP Address: 46.224.40.5
• Operating System: Ubuntu 22.04 LTS
• SSH Access: ssh root@46.224.40.5
• Server Name: iSu Technologies Production Server

Hosted Applications

Total: 14 Applications

1. sansa.isutech.co.za - SANSA Educator Platform (Vision 2030) - Frontend (PM2) + Backend (systemd)
2. tumi.isutech.co.za - Tumi Mentorship Platform - Frontend (nginx) + Backend (systemd)
3. sace.isutech.co.za - SACE Provider Intelligence System - Frontend (systemd) + Backend (systemd)
4. prospectiq - ProspectIQ Lead Management - 5 systemd services (frontend, api, celery, celery-beat, worker)
5. thrivesend - ThriveSend B2B2G Platform - Full-stack Next.js (PM2)
6. umdoni.gov.za - Umdoni Municipality Website - PHP/nginx
7. docs.isutech.co.za - Documentation Hub - MkDocs Material (nginx static)
8. isutech.co.za - iSuTech Corporate Site - Frontend + Backend + Web (nginx + PM2)
9. nhlanhla.isutech.co.za - Portfolio Website - Static site (nginx)
10. monitor.isutech.co.za - iSu Monitor - Monitoring dashboard
11. AutoSlip - Receipt Automation Service - Systemd service
12. MatchMind - Football Prediction Platform - Development (stopped)

---

🔐 Access & Authentication

SSH Access

# Connect to server
ssh root@46.224.40.5

# First time connection
# You'll be prompted to accept the server's fingerprint
# Type "yes" to continue

Run your work inside tmux

SSH sessions on remote servers should run inside tmux so they persist across network drops and support split panes. See the tmux Tutorial for a full walkthrough.

User Management

# Current user
whoami

# List all users
cat /etc/passwd

# Add new user (if needed)
sudo useradd -m -s /bin/bash username

# Set password for user
sudo passwd username

# Add user to sudo group
sudo usermod -aG sudo username

---

🔑 SSH Key Setup for New Team Members

This section guides new team members through setting up SSH key access to the Hetzner server.

Step 1: Check for Existing SSH Key

On macOS/Linux:

ls -la ~/.ssh/

On Windows (PowerShell):

ls ~/.ssh/

• If you see id_ed25519 or id_rsa files, skip to Step 3
• If you get “No such file” or no key files, continue to Step 2

Step 2: Generate SSH Key (if needed)

On macOS/Linux:

ssh-keygen -t ed25519 -C "your.email@isutech.co.za"

On Windows (PowerShell):

ssh-keygen -t ed25519 -C "your.email@isutech.co.za"

• Press Enter to accept default location (~/.ssh/id_ed25519)
• Press Enter twice for no passphrase (or set one for extra security)

Step 3: Display Your Public Key

On macOS/Linux:

cat ~/.ssh/id_ed25519.pub

On Windows (PowerShell):

Get-Content ~/.ssh/id_ed25519.pub

Copy the entire output - it starts with ssh-ed25519...

Step 4: Add Key to Server

Option A: Ask a team member with access Send your public key to a team member who already has server access. They will run:

ssh root@46.224.40.5
echo 'YOUR_PUBLIC_KEY_HERE' >> ~/.ssh/authorized_keys
exit

Option B: Use root password (if you have it)

ssh root@46.224.40.5
# Enter password when prompted
echo 'YOUR_PUBLIC_KEY_HERE' >> ~/.ssh/authorized_keys
exit

Step 5: Test Connection

ssh root@46.224.40.5

You should connect without being asked for a password.

Resetting Root Password (Admin Only)

If you need the root password and don’t have SSH key access:

1. Log into Hetzner Cloud Console
2. Select the server (autoslip-prod-01 / 46.224.40.5)
3. Go to Rescue tab
4. Click “Reset Root Password” (under ROOT PASSWORD section)
5. The new password will be displayed on screen
6. Use this password to SSH in and add your SSH key
7. Consider resetting the password again after setup for security

Managing Authorized Keys

View current authorized keys:

cat ~/.ssh/authorized_keys

Add a new key:

echo 'ssh-ed25519 AAAAC3... user@email.com' >> ~/.ssh/authorized_keys

Remove a key:

# Edit the file and remove the line containing the key
nano ~/.ssh/authorized_keys

Troubleshooting SSH Issues

Problem	Solution
”Permission denied (publickey)“	Your public key isn’t on the server - ask a team member to add it
”Connection refused”	Server may be down or SSH service not running
”Connection timed out”	Network issue or firewall blocking port 22
”Host key verification failed”	Server was reinstalled - remove old key with ssh-keygen -R 46.224.40.5

---

📂 Directory Structure

Application Directories

/var/www/
├── sansa-frontend/                # SANSA Frontend (Next.js on PM2)
│   ├── src/                       # Source code
│   ├── public/                    # Static assets
│   ├── .env.production            # Production config
│   ├── package.json               # Node dependencies
│   └── .next/                     # Built application (712MB)
│
├── sansa-backend/                 # SANSA Backend (FastAPI on systemd)
│   ├── app/                       # Application code
│   │   ├── api/                   # API routes
│   │   ├── core/                  # Core config
│   │   ├── models/                # Database models
│   │   └── main.py                # Entry point
│   ├── venv/                      # Python virtual environment
│   ├── .env                       # Environment variables
│   ├── alembic/                   # Database migrations
│   └── requirements.txt           # Python dependencies (590MB)
│
├── tumi-frontend/                 # Tumi Frontend (Static site)
│   ├── build/                     # Built static files
│   ├── assets/                    # Static assets
│   └── index.html                 # Entry point (145MB)
│
├── tumi-backend/                  # Tumi Backend (FastAPI on systemd)
│   ├── app/                       # Application code
│   ├── venv/                      # Python virtual environment
│   ├── .env                       # Environment variables
│   └── requirements.txt           # Python dependencies (94MB)
│
├── sace-frontend/                 # SACE Frontend (systemd)
│   ├── src/                       # Source code
│   ├── public/                    # Static assets
│   ├── .env.production            # Production config
│   ├── package.json               # Node dependencies
│   └── .next/                     # Built application
│
├── sace-backend/                  # SACE Backend (FastAPI on systemd)
│   ├── app/                       # Application code
│   │   ├── api/                   # API routes
│   │   ├── core/                  # Core config
│   │   ├── models/                # Database models
│   │   └── main.py                # Entry point
│   ├── venv/                      # Python virtual environment
│   ├── .env                       # Environment variables
│   ├── alembic/                   # Database migrations
│   └── requirements.txt           # Python dependencies
│
├── thrivesend/                    # ThriveSend (Next.js on PM2)
│   ├── src/                       # Source code
│   ├── public/                    # Static assets
│   ├── .env.production            # Production config
│   ├── package.json               # Node dependencies
│   └── .next/                     # Built application (1.2GB)
│
├── prospect-iq/                   # ProspectIQ (5 systemd services)
│   ├── frontend/                  # Frontend application
│   ├── backend/                   # Backend API
│   ├── celery/                    # Celery task queue
│   └── worker/                    # Background workers
│
├── umdoni/                        # Umdoni Municipality Website
│   ├── public/                    # PHP application
│   ├── assets/                    # Static assets
│   └── index.php                  # Entry point (284MB)
│
├── nhlanhla-portfolio/            # Portfolio Website
│   ├── build/                     # Built static files
│   └── assets/                    # Static assets (30MB)
│
├── isutech-frontend/              # iSuTech Corporate Frontend
│   └── [application files]
│
├── isutech-backend/               # iSuTech Corporate Backend (PM2)
│   └── [application files]
│
├── isutech-web/                   # iSuTech Corporate Web
│   └── [application files]
│
├── matchmind/                     # MatchMind (Development - stopped)
│   └── [application files]
│
└── docs/                          # Documentation Hub
    ├── repo/                      # Git repository
    │   ├── docs/                  # Markdown content
    │   ├── site/                  # Built static site
    │   └── mkdocs.yml             # MkDocs configuration
    └── scripts/
        └── update-docs.sh         # Auto-update script

Development Directory

/projects/active/
├── thrive-send-b2b2g/            # ThriveSend development (2.0GB)
├── isutech-prospect-iq/          # ProspectIQ development (696MB)
├── umdoni-website/               # Umdoni development (271MB)
├── mnyandu-portfolio_from_root/  # Portfolio development (147MB)
├── matchmind/                    # MatchMind development (19MB)
├── iSuMonitor/                   # Monitor development (7.6MB)
├── autoslip/                     # AutoSlip development (1.7MB)
└── MyProfile/                    # Profile docs (56KB)

Note: Development happens at /projects/active/ and is deployed to /var/www/ for production.

System Configuration Directories

/etc/
├── nginx/
│   ├── sites-available/          # Available site configs
│   │   ├── api.isutech.co.za
│   │   ├── autoslip
│   │   ├── docs.isutech.co.za
│   │   ├── isutech.co.za
│   │   ├── monitor.isutech.co.za
│   │   ├── nhlanhla.isutech.co.za
│   │   ├── prospectiq
│   │   ├── sace.isutech.co.za
│   │   ├── sansa.isutech.co.za
│   │   ├── thrivesend
│   │   ├── tumi.isutech.co.za
│   │   ├── umdoni.gov.za
│   │   └── (matchmind - not active)
│   └── sites-enabled/            # Active site configs (13 symlinks)
│
├── systemd/system/               # Systemd service files
│   ├── sansa-backend.service
│   ├── tumi-backend.service
│   ├── sace-backend.service
│   ├── sace-frontend.service
│   ├── autoslip.service
│   ├── prospectiq-api.service
│   ├── prospectiq-frontend.service
│   ├── prospectiq-celery.service
│   ├── prospectiq-celery-beat.service
│   ├── prospectiq-worker.service
│   └── docs-update.timer        # Auto-update scheduler
│
└── letsencrypt/                  # SSL certificates
    ├── live/
    │   ├── api.isutech.co.za/
    │   ├── docs.isutech.co.za/
    │   ├── isutech.co.za/
    │   ├── monitor.isutech.co.za/
    │   ├── nhlanhla.isutech.co.za/
    │   ├── sace.isutech.co.za/
    │   ├── sansa.isutech.co.za/
    │   ├── tumi.isutech.co.za/
    │   └── umdoni.gov.za/
    └── renewal/                  # Auto-renewal configs

---

🚀 Application Management

SACE Backend (FastAPI)

Service Control:

# Check status
sudo systemctl status sace-backend

# Start service
sudo systemctl start sace-backend

# Stop service
sudo systemctl stop sace-backend

# Restart service
sudo systemctl restart sace-backend

# Enable auto-start on boot
sudo systemctl enable sace-backend

# View logs
sudo journalctl -u sace-backend -f

# View last 100 lines
sudo journalctl -u sace-backend -n 100

Manual Operations:

# Navigate to directory
cd /var/www/sace-backend

# Activate virtual environment
source venv/bin/activate

# Check environment variables
cat .env

# Run migrations
alembic upgrade head

# Test API directly
uvicorn app.main:app --host 0.0.0.0 --port 8001

# Run Python scripts
python seed_demo_data.py
python seed_activities.py

Configuration Files:

• /var/www/sace-backend/.env - Environment variables
• /etc/systemd/system/sace-backend.service - Systemd service

---

SACE Frontend (Next.js)

Service Control:

# Check status
sudo systemctl status sace-frontend

# Start service
sudo systemctl start sace-frontend

# Stop service
sudo systemctl stop sace-frontend

# Restart service
sudo systemctl restart sace-frontend

# Enable auto-start on boot
sudo systemctl enable sace-frontend

# View logs
sudo journalctl -u sace-frontend -f

Manual Operations:

# Navigate to directory
cd /var/www/sace-frontend

# Check environment variables
cat .env.production

# Install dependencies
npm install

# Build application
npm run build

# Start manually (for testing)
npm start

# Development mode (not for production)
npm run dev

Configuration Files:

• /var/www/sace-frontend/.env.production - Environment variables
• /etc/systemd/system/sace-frontend.service - Systemd service

---

SANSA Backend (FastAPI)

Service Control:

# Check status
sudo systemctl status sansa-backend

# Start service
sudo systemctl start sansa-backend

# Stop service
sudo systemctl stop sansa-backend

# Restart service
sudo systemctl restart sansa-backend

# Enable auto-start on boot
sudo systemctl enable sansa-backend

# View logs
sudo journalctl -u sansa-backend -f

# View last 100 lines
sudo journalctl -u sansa-backend -n 100

Manual Operations:

# Navigate to directory
cd /var/www/sansa-backend

# Activate virtual environment
source venv/bin/activate

# Check environment variables
cat .env

# Run migrations
alembic upgrade head

# Test API directly
uvicorn app.main:app --host 0.0.0.0 --port 8003

# Run database seeding
python scripts/seed_database.py

Configuration Files:

• /var/www/sansa-backend/.env - Environment variables
• /etc/systemd/system/sansa-backend.service - Systemd service

API Endpoints:

• Base URL: https://sansa.isutech.co.za/api/v1
• API Docs: https://sansa.isutech.co.za/api/docs
• Health Check: https://sansa.isutech.co.za/api/v1/health

---

SANSA Frontend (Next.js/PM2)

Service Control:

# Check status
pm2 status sansa-frontend

# Start service
pm2 start sansa-frontend

# Stop service
pm2 stop sansa-frontend

# Restart service
pm2 restart sansa-frontend

# View logs
pm2 logs sansa-frontend

# View last 100 lines
pm2 logs sansa-frontend --lines 100

Manual Operations:

# Navigate to directory
cd /var/www/sansa-frontend

# Check environment variables
cat .env.production

# Install dependencies
npm install

# Build application
npm run build

# Start manually (for testing)
npm start

Configuration Files:

• /var/www/sansa-frontend/.env.production - Environment variables
• PM2 managed (no systemd service)

Test Users:

Email	Password	Role
admin@sansa.gov.za	Admin@2024	Admin (Superuser)
dan.matsapola@sansa.gov.za	Sansa@2024	Admin
facilitator@sansa.gov.za	Sansa@2024	Facilitator
viewer@sansa.gov.za	Sansa@2024	Viewer

---

Tumi Backend (FastAPI)

Service Control:

# Check status
sudo systemctl status tumi-backend

# Start service
sudo systemctl start tumi-backend

# Stop service
sudo systemctl stop tumi-backend

# Restart service
sudo systemctl restart tumi-backend

# View logs
sudo journalctl -u tumi-backend -f

# View last 100 lines
sudo journalctl -u tumi-backend -n 100

Manual Operations:

# Navigate to directory
cd /var/www/tumi-backend

# Activate virtual environment
source venv/bin/activate

# Check environment variables
cat .env

# Run migrations
alembic upgrade head

# Test API directly
uvicorn app.main:app --host 0.0.0.0 --port 8004

Configuration Files:

• /var/www/tumi-backend/.env - Environment variables
• /etc/systemd/system/tumi-backend.service - Systemd service

API Endpoints:

• Base URL: https://tumi.isutech.co.za/api
• Health Check: https://tumi.isutech.co.za/api/health

---

Tumi Frontend (Static Site)

Deployment: The Tumi frontend is a static site served directly by nginx from /var/www/tumi-frontend/.

Manual Operations:

# Navigate to directory
cd /var/www/tumi-frontend

# View files
ls -lah

# Check nginx config
cat /etc/nginx/sites-available/tumi.isutech.co.za

No service management needed - nginx serves static files directly.

---

ThriveSend (Next.js/PM2)

Service Control:

# Check status
pm2 status thrivesend

# Start service
pm2 start thrivesend

# Stop service
pm2 stop thrivesend

# Restart service
pm2 restart thrivesend

# View logs
pm2 logs thrivesend

# View last 100 lines
pm2 logs thrivesend --lines 100

Manual Operations:

# Navigate to directory
cd /var/www/thrivesend

# Check environment variables
cat .env.production

# Install dependencies
npm install

# Build application
npm run build

# Start manually
npm start

Configuration Files:

• /var/www/thrivesend/.env.production - Environment variables
• PM2 managed (no systemd service)

---

ProspectIQ (5 Systemd Services)

ProspectIQ runs with 5 separate systemd services for scalability:

Service Control:

# Check all ProspectIQ services
sudo systemctl status prospectiq-api prospectiq-frontend prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Restart all services
sudo systemctl restart prospectiq-api prospectiq-frontend prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Start individual services
sudo systemctl start prospectiq-api
sudo systemctl start prospectiq-frontend
sudo systemctl start prospectiq-celery
sudo systemctl start prospectiq-celery-beat
sudo systemctl start prospectiq-worker

# View logs for specific service
sudo journalctl -u prospectiq-api -f
sudo journalctl -u prospectiq-celery -f

Service Descriptions:

• prospectiq-api - Backend API service (FastAPI)
• prospectiq-frontend - Frontend application (Next.js)
• prospectiq-celery - Celery worker for async tasks
• prospectiq-celery-beat - Celery beat scheduler
• prospectiq-worker - Background worker service

Manual Operations:

# Navigate to directory
cd /var/www/prospect-iq

# View service configurations
ls -lah /etc/systemd/system/prospectiq-*

# Check nginx config
cat /etc/nginx/sites-available/prospectiq

Configuration Files:

• /var/www/prospect-iq/.env - Environment variables
• /etc/systemd/system/prospectiq-*.service - Systemd services (5 files)
• /etc/nginx/sites-available/prospectiq - Nginx configuration

---

Umdoni Municipality Website

Deployment: The Umdoni website is a PHP-based application served by nginx from /var/www/umdoni/.

Manual Operations:

# Navigate to directory
cd /var/www/umdoni

# View files
ls -lah

# Check nginx config
cat /etc/nginx/sites-available/umdoni.gov.za

# Check PHP-FPM status (if applicable)
sudo systemctl status php*-fpm

Configuration Files:

• /etc/nginx/sites-available/umdoni.gov.za - Nginx configuration
• PHP files served directly by nginx

---

Nhlanhla Portfolio Website

Deployment: Portfolio website is a static site served by nginx from /var/www/nhlanhla-portfolio/.

Manual Operations:

# Navigate to directory
cd /var/www/nhlanhla-portfolio

# View files
ls -lah

# Check nginx config
cat /etc/nginx/sites-available/nhlanhla.isutech.co.za

No service management needed - nginx serves static files directly.

---

iSuTech Corporate Site

Service Control:

# Check PM2 backend status
pm2 status isutech-backend

# Restart backend
pm2 restart isutech-backend

# View logs
pm2 logs isutech-backend

Manual Operations:

# Navigate to directories
cd /var/www/isutech-frontend
cd /var/www/isutech-backend
cd /var/www/isutech-web

# Check nginx configs
cat /etc/nginx/sites-available/isutech.co.za
cat /etc/nginx/sites-available/api.isutech.co.za

---

AutoSlip (Receipt Automation)

Service Control:

# Check status
sudo systemctl status autoslip

# Start service
sudo systemctl start autoslip

# Stop service
sudo systemctl stop autoslip

# Restart service
sudo systemctl restart autoslip

# View logs
sudo journalctl -u autoslip -f

Manual Operations:

# Development code location
cd /projects/active/autoslip

# Check service configuration
cat /etc/systemd/system/autoslip.service

# Check nginx config
cat /etc/nginx/sites-available/autoslip

Note: AutoSlip runs as a systemd service. Development code is at /projects/active/autoslip/.

---

Documentation Hub (MkDocs)

Auto-Update System:

# Check update timer status
sudo systemctl status docs-update.timer

# Check update service status
sudo systemctl status docs-update.service

# View update logs
sudo journalctl -u docs-update.service -n 50

# Manual update
sudo /var/www/docs/scripts/update-docs.sh

# View recent update log
tail -n 30 /var/log/docs-update.log

Manual Operations:

# Navigate to repository
cd /var/www/docs/repo

# Pull latest changes
git pull origin main

# Build documentation
mkdocs build

# Serve locally (for testing)
mkdocs serve

# Deploy to production
mkdocs build
# Site built to: /var/www/docs/repo/site

Configuration Files:

• /var/www/docs/repo/mkdocs.yml - MkDocs configuration
• /var/www/docs/scripts/update-docs.sh - Update script

---

🌐 Nginx Configuration

Viewing Configurations

# List all site configs
ls -la /etc/nginx/sites-available/

# View docs config
cat /etc/nginx/sites-available/docs.isutech.co.za

# View SACE config
cat /etc/nginx/sites-available/sace.isutech.co.za

# Check enabled sites
ls -la /etc/nginx/sites-enabled/

Managing Nginx

# Test configuration syntax
sudo nginx -t

# Reload configuration (no downtime)
sudo systemctl reload nginx

# Restart Nginx
sudo systemctl restart nginx

# Check status
sudo systemctl status nginx

# View error logs
sudo tail -f /var/log/nginx/error.log

# View access logs
sudo tail -f /var/log/nginx/access.log

Adding New Site

# 1. Create configuration file
sudo nano /etc/nginx/sites-available/newsite.isutech.co.za

# 2. Enable site (create symlink)
sudo ln -s /etc/nginx/sites-available/newsite.isutech.co.za /etc/nginx/sites-enabled/

# 3. Test configuration
sudo nginx -t

# 4. Reload Nginx
sudo systemctl reload nginx

# 5. Install SSL certificate
sudo certbot --nginx -d newsite.isutech.co.za

---

🗄️ Database Management

PostgreSQL

Service Control:

# Check status
sudo systemctl status postgresql

# Start service
sudo systemctl start postgresql

# Stop service
sudo systemctl stop postgresql

# Restart service
sudo systemctl restart postgresql

Database Operations:

# Connect as postgres user
sudo -u postgres psql

# List databases
sudo -u postgres psql -c "\l"

# Connect to SACE database
sudo -u postgres psql -d sace_production

# Using SACE user credentials
PGPASSWORD="SaceProd2025Secure" psql -U sace_user -d sace_production -h localhost

# Backup database
sudo -u postgres pg_dump sace_production > /tmp/sace_backup_$(date +%Y%m%d).sql

# Restore database
sudo -u postgres psql sace_production < /tmp/sace_backup_20251111.sql

# Check database size
sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('sace_production'));"

Common Queries:

-- Inside psql
\l                              -- List databases
\du                             -- List users
\dt                             -- List tables
\d table_name                   -- Describe table
\q                              -- Quit

-- Sample queries
SELECT COUNT(*) FROM providers;
SELECT COUNT(*) FROM educators;
SELECT COUNT(*) FROM activities;
SELECT COUNT(*) FROM enrollments;

---

Redis

Service Control:

# Check status
sudo systemctl status redis

# Start service
sudo systemctl start redis

# Stop service
sudo systemctl stop redis

# Restart service
sudo systemctl restart redis

Redis Operations:

# Connect to Redis CLI
redis-cli

# Inside redis-cli:
# ping                          # Test connection
# keys *                        # List all keys
# flushall                      # Clear all data (DANGER!)
# info                          # Server info
# quit                          # Exit

---

🔒 SSL Certificates

Certbot Management

Check Certificate Status:

# List all certificates
sudo certbot certificates

# Check specific certificate
sudo certbot certificates --cert-name docs.isutech.co.za

Renew Certificates:

# Test renewal (dry run)
sudo certbot renew --dry-run

# Manually renew
sudo certbot renew

# Renew specific certificate
sudo certbot renew --cert-name docs.isutech.co.za

Install New Certificate:

# For Nginx
sudo certbot --nginx -d newdomain.isutech.co.za

# Manual mode (no auto-config)
sudo certbot certonly --nginx -d newdomain.isutech.co.za

Auto-Renewal:

# Certbot sets up automatic renewal via systemd timer
# Check timer status
sudo systemctl status certbot.timer

# View renewal logs
sudo journalctl -u certbot.service

---

📊 Monitoring & Logs

Quick Reference Available

For a comprehensive monitoring command reference including PostgreSQL monitoring, emergency procedures, and health check scripts, see the Server Monitoring Quick Reference.

System Health

Disk Space:

# Check disk usage
df -h

# Check directory sizes
du -sh /var/www/*

# Find large files
sudo find / -type f -size +100M -exec ls -lh {} \;

Memory & CPU:

# Memory usage
free -h

# CPU usage
top

# Process list
htop  # If installed

# Running processes
ps aux | grep python
ps aux | grep node

Network:

# Check open ports
sudo netstat -tulpn

# Check listening services
sudo ss -tulpn

# Test port connectivity
curl http://localhost:8001/api/health
curl http://localhost:3000

Application Logs

SACE Backend:

# Live logs
sudo journalctl -u sace-backend -f

# Last 100 lines
sudo journalctl -u sace-backend -n 100

# Logs from today
sudo journalctl -u sace-backend --since today

# Logs with error level
sudo journalctl -u sace-backend -p err

SACE Frontend:

# Live logs
sudo journalctl -u sace-frontend -f

# Last 100 lines
sudo journalctl -u sace-frontend -n 100

Nginx:

# Access logs
sudo tail -f /var/log/nginx/access.log

# Error logs
sudo tail -f /var/log/nginx/error.log

# Search for errors
sudo grep "error" /var/log/nginx/error.log

# Site-specific logs (if configured)
sudo tail -f /var/log/nginx/sace-access.log
sudo tail -f /var/log/nginx/docs-access.log

---

🔧 Maintenance Tasks

Regular Maintenance (Weekly)

# 1. Update system packages
sudo apt update
sudo apt upgrade -y

# 2. Check disk space
df -h

# 3. Clean old logs
sudo journalctl --vacuum-time=7d

# 4. Check service status
sudo systemctl status sansa-backend tumi-backend sace-backend sace-frontend \
  autoslip prospectiq-api prospectiq-frontend prospectiq-celery \
  prospectiq-celery-beat prospectiq-worker nginx postgresql redis
pm2 status

# 5. Verify SSL certificate expiry
sudo certbot certificates

# 6. Backup database
sudo -u postgres pg_dump sace_production > ~/backups/sace_backup_$(date +%Y%m%d).sql

Monthly Tasks

# 1. Review and rotate logs
sudo logrotate -f /etc/logrotate.conf

# 2. Check for security updates
sudo apt list --upgradable | grep -i security

# 3. Review disk usage trends
du -sh /var/www/* /var/log/*

# 4. Test backup restoration
# Restore to test database

# 5. Review failed systemd services
sudo systemctl --failed

---

🚨 Troubleshooting

Service Won’t Start

# 1. Check service status
sudo systemctl status service-name

# 2. View detailed logs
sudo journalctl -u service-name -n 50

# 3. Check configuration syntax
# For Nginx:
sudo nginx -t

# For Python/FastAPI:
cd /var/www/sace-backend
source venv/bin/activate
python -m app.main

# 4. Check port conflicts
sudo netstat -tulpn | grep :8001

Application Not Accessible

# 1. Check Nginx status
sudo systemctl status nginx

# 2. Check DNS resolution
nslookup sace.isutech.co.za

# 3. Check firewall rules
sudo ufw status

# 4. Test local connectivity
curl http://localhost:8001/api/health
curl http://localhost:3000

# 5. Check SSL certificate
sudo certbot certificates

# 6. Review Nginx error logs
sudo tail -f /var/log/nginx/error.log

Database Connection Issues

# 1. Check PostgreSQL status
sudo systemctl status postgresql

# 2. Test connection
sudo -u postgres psql -c "SELECT 1;"

# 3. Check user permissions
sudo -u postgres psql -c "\du"

# 4. Test SACE user connection
PGPASSWORD="SaceProd2025Secure" psql -U sace_user -d sace_production -h localhost -c "SELECT 1;"

# 5. Check PostgreSQL logs
sudo tail -f /var/log/postgresql/postgresql-14-main.log

High Memory/CPU Usage

# 1. Identify resource-hungry processes
top
htop

# 2. Check specific service resources
systemctl status sace-backend
systemctl status sace-frontend

# 3. Restart problematic service
sudo systemctl restart service-name

# 4. Check for memory leaks
# Review application logs for errors

---

🔒 Security Best Practices

User Access

• ✅ Use SSH key authentication (disable password auth)
• ✅ Create separate users for different team members
• ✅ Use sudo instead of logging in as root
• ✅ Regularly review user accounts and permissions

Firewall

# Check UFW status
sudo ufw status

# Allow specific ports only
sudo ufw allow 80/tcp    # HTTP
sudo ufw allow 443/tcp   # HTTPS
sudo ufw allow 22/tcp    # SSH

# Enable firewall
sudo ufw enable

Application Security

• ✅ Keep all software updated
• ✅ Use environment variables for secrets (never commit)
• ✅ Enable HTTPS for all applications
• ✅ Use strong passwords and rotate regularly
• ✅ Implement rate limiting on API endpoints
• ✅ Regular security audits

Database Security

• ✅ Use strong passwords
• ✅ Restrict database access to localhost
• ✅ Regular backups with tested restoration
• ✅ Limit user permissions (principle of least privilege)

---

📦 Deployment Procedures

Deploying SACE Updates

Backend Updates:

# 1. SSH to server
ssh root@46.224.40.5

# 2. Navigate to backend
cd /var/www/sace-backend

# 3. Backup current code
cp -r app app.backup

# 4. Pull latest changes (if using Git)
git pull origin main

# 5. Activate virtual environment
source venv/bin/activate

# 6. Install new dependencies
pip install -r requirements.txt

# 7. Run migrations
alembic upgrade head

# 8. Restart service
sudo systemctl restart sace-backend

# 9. Verify
curl http://localhost:8001/api/health
sudo journalctl -u sace-backend -n 20

Frontend Updates:

# 1. Navigate to frontend
cd /var/www/sace-frontend

# 2. Backup current build
cp -r .next .next.backup

# 3. Pull latest changes
git pull origin main

# 4. Install dependencies
npm install

# 5. Build application
npm run build

# 6. Restart service
sudo systemctl restart sace-frontend

# 7. Verify
curl http://localhost:3000
sudo journalctl -u sace-frontend -n 20

Deploying Documentation Updates

Automatic Process:

• Documentation auto-updates every 10 minutes via systemd timer
• Pulls from GitHub and rebuilds automatically

Manual Update:

# Run update script manually
sudo /var/www/docs/scripts/update-docs.sh

# View logs
tail -n 30 /var/log/docs-update.log

---

📝 Quick Reference Commands

Most Used Commands

# Check all systemd service statuses
sudo systemctl status sansa-backend tumi-backend sace-backend sace-frontend \
  autoslip prospectiq-api prospectiq-frontend prospectiq-celery \
  prospectiq-celery-beat prospectiq-worker nginx postgresql redis

# Check all PM2 processes
pm2 status

# Restart all backend services
sudo systemctl restart sansa-backend tumi-backend sace-backend \
  prospectiq-api prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Restart all frontend services (systemd)
sudo systemctl restart sace-frontend prospectiq-frontend

# Restart all PM2 applications
pm2 restart all

# View all logs in real-time
sudo journalctl -f

# Check disk space
df -h

# Check memory
free -h

# Test API health endpoints
curl https://sansa.isutech.co.za/api/v1/health
curl https://tumi.isutech.co.za/api/health
curl https://sace.isutech.co.za/api/health

# Update documentation manually
sudo /var/www/docs/scripts/update-docs.sh

# Backup all databases
sudo -u postgres pg_dump sace_production > ~/backups/sace_$(date +%Y%m%d).sql
sudo -u postgres pg_dump sansa_production > ~/backups/sansa_$(date +%Y%m%d).sql
sudo -u postgres pg_dump tumi_production > ~/backups/tumi_$(date +%Y%m%d).sql

# Check Nginx config and reload
sudo nginx -t && sudo systemctl reload nginx

# View all nginx sites
ls -la /etc/nginx/sites-enabled/

# Check all SSL certificates
sudo certbot certificates

---

📞 Support & Escalation

When to Restart Services

• Application not responding
• High memory usage
• After configuration changes
• After code deployment

When to Contact Support

• Persistent service failures
• Database corruption
• Security incidents
• SSL certificate issues
• Hardware problems

Emergency Procedures

1. Service Down: Restart service → Check logs → Verify config
2. Database Issue: Stop writes → Backup → Restore from backup
3. Security Breach: Isolate server → Review logs → Reset credentials
4. Disk Full: Clear old logs → Remove temp files → Expand disk

---

📚 Additional Resources

Documentation Links

• Nginx Documentation
• FastAPI Documentation
• Next.js Documentation
• PostgreSQL Documentation
• MkDocs Material

Internal Documentation

• SACE System Documentation

---

Last Updated: 02/01/2026 | Document Owner: Operations Team | Version: 2.0

---

📋 Application Inventory Summary

#	Application	Location	Domain	Service Type	Status
1	SANSA Frontend	/var/www/sansa-frontend/	sansa.isutech.co.za	PM2 (port 3003)	✅ Running
2	SANSA Backend	/var/www/sansa-backend/	sansa.isutech.co.za/api	systemd (port 8003)	✅ Running
3	Tumi Frontend	/var/www/tumi-frontend/	tumi.isutech.co.za	nginx static	✅ Serving
4	Tumi Backend	/var/www/tumi-backend/	tumi.isutech.co.za/api	systemd (port 8004)	✅ Running
5	SACE Frontend	/var/www/sace-frontend/	sace.isutech.co.za	systemd	✅ Running
6	SACE Backend	/var/www/sace-backend/	sace.isutech.co.za/api	systemd (port 8001)	✅ Running
7	ThriveSend	/var/www/thrivesend/	thrivesend	PM2	✅ Running
8	ProspectIQ API	/var/www/prospect-iq/	prospectiq	systemd	✅ Running
9	ProspectIQ Frontend	/var/www/prospect-iq/	prospectiq	systemd	✅ Running
10	ProspectIQ Celery	/var/www/prospect-iq/	-	systemd	✅ Running
11	ProspectIQ Beat	/var/www/prospect-iq/	-	systemd	✅ Running
12	ProspectIQ Worker	/var/www/prospect-iq/	-	systemd	✅ Running
13	DocsHub	/var/www/docs/	docs.isutech.co.za	nginx static	✅ Serving
14	Umdoni	/var/www/umdoni/	umdoni.gov.za	nginx + PHP	✅ Serving
15	iSuTech Corporate	/var/www/isutech-*/	isutech.co.za	nginx + PM2	⚠️ Partial
16	Portfolio	/var/www/nhlanhla-portfolio/	nhlanhla.isutech.co.za	nginx static	⚠️ Unknown
17	AutoSlip	Service only	autoslip	systemd	✅ Running
18	MatchMind	/var/www/matchmind/	-	-	🔴 Stopped

Total: 18 services across 14 applications Running: 14 services Partially Running: 1 (iSuTech) Unknown: 1 (Portfolio) Stopped: 2 (MatchMind services)