Hetzner Server Management Guide

Hetzner Server Management Guide

Server: production-server-01 Provider: Hetzner Last Updated: 2026-02-01

---

Server Overview

Property	Value
Hostname	production-server-01
IP Address	46.224.40.5
OS	Ubuntu 22.04 LTS
Web Server	Nginx
Database	PostgreSQL 14
Cache	Redis
Process Manager	systemd + PM2
Container Runtime	Docker + Containerd

---

Project Inventory

Active Projects (14 Applications)

#	Project	Location	Domain	Service Type	Status
1	SANSA Frontend	/var/www/sansa-frontend/	sansa.isutech.co.za	PM2 (port 3003)	✅ Running
2	SANSA Backend	/var/www/sansa-backend/	sansa.isutech.co.za/api	systemd (port 8003)	✅ Running
3	Tumi Frontend	/var/www/tumi-frontend/	tumi.isutech.co.za	nginx static	✅ Serving
4	Tumi Backend	/var/www/tumi-backend/	tumi.isutech.co.za/api	systemd (port 8004)	✅ Running
5	SACE Frontend	/var/www/sace-frontend/	sace.isutech.co.za	systemd	✅ Running
6	SACE Backend	/var/www/sace-backend/	sace.isutech.co.za/api	systemd	✅ Running
7	ThriveSend	/var/www/thrivesend/	thrivesend (nginx)	PM2	✅ Running
8	ProspectIQ	/var/www/prospect-iq/	prospectiq (nginx)	systemd (5 services)	✅ Running
9	DocsHub	/var/www/docs/repo/	docs.isutech.co.za	nginx static	✅ Serving
10	Umdoni	/var/www/umdoni/	umdoni.gov.za	nginx + PHP	✅ Serving
11	iSu Tech Frontend	/var/www/isutech-frontend/	isutech.co.za	nginx static	✅ Serving
12	iSu Tech Backend	/var/www/isutech-backend/	api.isutech.co.za	PM2	✅ Running
13	Nhlanhla Portfolio	/var/www/nhlanhla-portfolio/	nhlanhla.isutech.co.za	nginx static	⚠️ Configured
14	MatchMind	/var/www/matchmind/	-	-	🔴 Stopped

Development Projects: /projects/active/ (see Development Workflow)

---

Project Details

1. SANSA (Vision 2030 Educator Platform)

Frontend:

• Path: /var/www/sansa-frontend/
• Size: 712MB
• Type: Next.js application
• Port: 3003
• Process Manager: PM2
• Domain: https://sansa.isutech.co.za

Commands:

# Status
pm2 status sansa-frontend

# Restart
pm2 restart sansa-frontend

# Logs
pm2 logs sansa-frontend

# Manual start
cd /var/www/sansa-frontend
npm run build
pm2 start npm --name sansa-frontend -- start

Backend:

• Path: /var/www/sansa-backend/
• Size: 590MB
• Type: FastAPI (Python)
• Port: 8003
• Service: sansa-backend.service
• Domain: https://sansa.isutech.co.za/api

Commands:

# Status
systemctl status sansa-backend

# Restart
systemctl restart sansa-backend

# Logs
journalctl -u sansa-backend -f

# Manual operation
cd /var/www/sansa-backend
source venv/bin/activate
uvicorn app.main:app --host 0.0.0.0 --port 8003

---

2. Tumi Platform

Frontend:

• Path: /var/www/tumi-frontend/
• Size: 145MB
• Type: Static site
• Domain: https://tumi.isutech.co.za

Backend:

• Path: /var/www/tumi-backend/
• Size: 94MB
• Type: FastAPI (Python)
• Port: 8004
• Service: tumi-backend.service
• Domain: https://tumi.isutech.co.za/api

Commands:

# Backend status
systemctl status tumi-backend

# Backend restart
systemctl restart tumi-backend

# Backend logs
journalctl -u tumi-backend -f

# Frontend (static - no restart needed)
# Files served directly by nginx

---

3. SACE (Provider Intelligence)

Frontend:

• Path: /var/www/sace-frontend/
• Size: 564MB
• Type: Next.js application
• Service: sace-frontend.service
• Domain: https://sace.isutech.co.za

Backend:

• Path: /var/www/sace-backend/
• Size: 552MB
• Type: FastAPI (Python)
• Service: sace-backend.service
• Domain: https://sace.isutech.co.za/api

Commands:

# Status
systemctl status sace-backend sace-frontend

# Restart both
systemctl restart sace-backend sace-frontend

# Logs
journalctl -u sace-backend -f
journalctl -u sace-frontend -f

---

4. ThriveSend (B2B2G Platform)

• Path: /var/www/thrivesend/
• Size: 1.2GB
• Type: Next.js full-stack application
• Process Manager: PM2
• Nginx Config: /etc/nginx/sites-enabled/thrivesend

Commands:

# Status
pm2 status thrivesend

# Restart
pm2 restart thrivesend

# Logs
pm2 logs thrivesend

# View all ThriveSend processes
pm2 list | grep thrive

---

5. ProspectIQ (Lead Intelligence Platform)

• Path: /var/www/prospect-iq/
• Size: 3.6MB (code) + node_modules
• Stack: Python FastAPI + Next.js + Celery + PostgreSQL + Redis
• Nginx Config: /etc/nginx/sites-enabled/prospectiq

Services (5 total):

1. prospectiq-api.service - FastAPI Backend
2. prospectiq-frontend.service - Next.js Frontend
3. prospectiq-celery.service - Celery Worker
4. prospectiq-celery-beat.service - Celery Beat Scheduler
5. prospectiq-worker.service - Async Task Processor

Commands:

# Status all services
systemctl status prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Restart all
systemctl restart prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Logs
journalctl -u prospectiq-api -f
journalctl -u prospectiq-frontend -f
journalctl -u prospectiq-celery -f

---

6. DocsHub (Knowledge Hub)

• Path: /var/www/docs/repo/
• Size: 237MB
• Domain: https://docs.isutech.co.za 🔒 (Password Protected)
• Type: MkDocs static site (Material theme)
• Git Remote: git@github.com:gedeza/isutech-knowledge-hub.git

Commands:

# Update documentation
cd /var/www/docs/repo
git pull origin main
mkdocs build

# Preview locally
mkdocs serve  # http://localhost:8000

# Manual deployment
cd /var/www/docs/repo
git pull && mkdocs build

Auto-Update:

• Documentation auto-updates via systemd timer
• Check: systemctl status docs-update.timer

---

7. Umdoni Municipality Website

• Path: /var/www/umdoni/
• Size: 284MB
• Domain: https://umdoni.gov.za
• Type: PHP-based application
• Status: ✅ Serving

Commands:

# View nginx configuration
cat /etc/nginx/sites-available/umdoni.gov.za

# Check PHP-FPM status
systemctl status php*-fpm

# View logs
tail -f /var/log/nginx/umdoni-access.log

---

8. iSu Tech Corporate Website

Frontend:

• Path: /var/www/isutech-frontend/
• Size: 3.2MB
• Domain: https://isutech.co.za
• Type: Static site

Backend:

• Path: /var/www/isutech-backend/
• Size: 371MB
• Domain: https://api.isutech.co.za
• Process Manager: PM2
• PM2 Names: isutech-backend, isu-api

Commands:

# Backend status
pm2 status isutech-backend

# Backend restart
pm2 restart isutech-backend

# Backend logs
pm2 logs isutech-backend

---

9. AutoSlip (Receipt Automation)

• Production: Running as systemd service
• Development Code: /projects/active/autoslip/
• Service: autoslip.service
• Purpose: WhatsApp-based receipt processing

Commands:

# Status
systemctl status autoslip

# Restart
systemctl restart autoslip

# Logs
journalctl -u autoslip -f

---

10. Nhlanhla Portfolio

• Path: /var/www/nhlanhla-portfolio/
• Size: 30MB
• Domain: https://nhlanhla.isutech.co.za
• Type: Static site
• Status: Configured, serving status unknown

---

11. MatchMind (Sports Prediction)

• Path: /var/www/matchmind/
• Size: 568MB
• Status: 🔴 Stopped (code exists, no service running)
• Development: /projects/active/matchmind/
• Primary Deployment: Railway

---

Development Workflow

Development Directory: /projects/active/ (3.1GB)

Active Development Projects:

1. thrive-send-b2b2g (2.0GB)
2. isutech-prospect-iq (696MB)
3. umdoni-website (271MB)
4. mnyandu-portfolio_from_root (147MB)
5. matchmind (19MB)
6. iSuMonitor (7.6MB)
7. autoslip (1.7MB)
8. MyProfile (56KB)

Workflow:

Development → Testing → Production
/projects/active/[project] → test → /var/www/[project]

See: Development Workflow Guide

---

Infrastructure Services

Nginx (Web Server / Reverse Proxy)

Enabled Sites (13):

• api.isutech.co.za
• autoslip
• docs.isutech.co.za
• isutech.co.za
• monitor.isutech.co.za
• nhlanhla.isutech.co.za
• prospectiq
• sace.isutech.co.za
• sansa.isutech.co.za
• thrivesend
• tumi.isutech.co.za
• umdoni.gov.za
• (matchmind - configured but not serving)

Commands:

# Status
systemctl status nginx

# Test configuration
nginx -t

# Reload configuration (no downtime)
systemctl reload nginx

# Restart nginx
systemctl restart nginx

# View enabled sites
ls -la /etc/nginx/sites-enabled/

# View specific config
cat /etc/nginx/sites-available/docs.isutech.co.za

---

PostgreSQL 14

Status: ✅ Running

Commands:

# Status
systemctl status postgresql@14-main

# Connect to database
sudo -u postgres psql

# List databases
sudo -u postgres psql -c "\l"

# Backup database
sudo -u postgres pg_dump dbname > backup_$(date +%Y%m%d).sql

# Restore database
sudo -u postgres psql dbname < backup_20260201.sql

---

Redis

Status: ✅ Running

Commands:

# Status
systemctl status redis-server

# Connect to Redis CLI
redis-cli

# Check info
redis-cli INFO

# Inside redis-cli:
# ping              # Test connection
# keys *            # List all keys
# info              # Server info
# quit              # Exit

---

Docker & Containerd

Status: ✅ Running

Commands:

# Docker status
systemctl status docker

# Containerd status
systemctl status containerd

# List containers
docker ps

# List images
docker images

---

Directory Structure

/var/www/
├── docs/                        # DocsHub (237MB)
│   └── repo/                    # Git repository + built site
├── sansa-frontend/              # SANSA Frontend (712MB)
├── sansa-backend/               # SANSA Backend (590MB)
├── tumi-frontend/               # Tumi Frontend (145MB)
├── tumi-backend/                # Tumi Backend (94MB)
├── sace-frontend/               # SACE Frontend (564MB)
├── sace-backend/                # SACE Backend (552MB)
├── thrivesend/                  # ThriveSend (1.2GB)
├── prospect-iq/                 # ProspectIQ (3.6MB + modules)
├── umdoni/                      # Umdoni Municipality (284MB)
├── isutech-frontend/            # iSu Tech Frontend (3.2MB)
├── isutech-backend/             # iSu Tech Backend (371MB)
├── isutech-web/                 # iSu Tech Web (6.5MB)
├── nhlanhla-portfolio/          # Portfolio (30MB)
├── matchmind/                   # MatchMind (568MB - stopped)
└── html/                        # Default nginx (unused)

/projects/active/                # Development (3.1GB)
├── thrive-send-b2b2g/           # ThriveSend dev
├── isutech-prospect-iq/         # ProspectIQ dev
├── umdoni-website/              # Umdoni dev
├── mnyandu-portfolio_from_root/ # Portfolio dev
├── matchmind/                   # MatchMind dev
├── autoslip/                    # AutoSlip dev
├── iSuMonitor/                  # Monitoring assets
└── MyProfile/                   # CV and profiles

/etc/nginx/sites-enabled/
├── api.isutech.co.za
├── autoslip
├── docs.isutech.co.za
├── isutech.co.za
├── monitor.isutech.co.za
├── nhlanhla.isutech.co.za
├── prospectiq
├── sace.isutech.co.za
├── sansa.isutech.co.za
├── thrivesend
├── tumi.isutech.co.za
└── umdoni.gov.za

---

Service Management Commands

Quick Reference

# View all running systemd services
systemctl list-units --type=service --state=running

# View all PM2 processes
pm2 list

# View nginx enabled sites
ls /etc/nginx/sites-enabled/

# Check what's listening on ports
ss -tlnp | grep LISTEN

# Check disk usage
df -h
du -sh /var/www/*

# Check memory usage
free -h

---

Restart All Application Services

# All systemd backend services
systemctl restart sansa-backend tumi-backend sace-backend sace-frontend \
  prospectiq-api prospectiq-frontend prospectiq-celery \
  prospectiq-celery-beat prospectiq-worker autoslip

# All PM2 frontend services
pm2 restart all

# Nginx
systemctl reload nginx

---

Restart by Application

# SANSA
systemctl restart sansa-backend
pm2 restart sansa-frontend

# Tumi
systemctl restart tumi-backend
# (Frontend is static, no restart needed)

# SACE
systemctl restart sace-backend sace-frontend

# ThriveSend
pm2 restart thrivesend

# ProspectIQ
systemctl restart prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

# AutoSlip
systemctl restart autoslip

# iSu Tech
pm2 restart isutech-backend
# (Frontend is static, no restart needed)

---

Deployment Workflows

General Deployment Pattern

# 1. Backup current version
cp -r /var/www/[project] /var/www/[project].backup.$(date +%Y%m%d)

# 2. Pull latest code (if using Git)
cd /var/www/[project]
git pull origin main

# 3. Install dependencies
# For Python:
source venv/bin/activate && pip install -r requirements.txt
# For Node:
npm install

# 4. Build if needed
# For Next.js:
npm run build
# For MkDocs:
mkdocs build

# 5. Restart service
# For systemd:
systemctl restart [service-name]
# For PM2:
pm2 restart [app-name]

# 6. Verify
curl http://localhost:[port]/health
journalctl -u [service] -n 20  # or pm2 logs [app]

---

Specific Application Deployments

SANSA:

# Backend
cd /var/www/sansa-backend
source venv/bin/activate
git pull && pip install -r requirements.txt
alembic upgrade head
systemctl restart sansa-backend

# Frontend
cd /var/www/sansa-frontend
git pull && npm install && npm run build
pm2 restart sansa-frontend

DocsHub:

cd /var/www/docs/repo
git pull origin main
mkdocs build
# No restart needed - nginx serves static files

ProspectIQ:

cd /var/www/prospect-iq
git pull origin main
pip install -r requirements.txt
npm install && npm run build
systemctl restart prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

---

Monitoring & Logs

View Service Logs

Systemd Services:

# Live logs (follow)
journalctl -u SERVICE_NAME -f

# Last 100 lines
journalctl -u SERVICE_NAME -n 100

# Today's logs
journalctl -u SERVICE_NAME --since today

# Error level only
journalctl -u SERVICE_NAME -p err

# Multiple services
journalctl -u sansa-backend -u tumi-backend -f

PM2 Applications:

# Live logs
pm2 logs APP_NAME

# Last 100 lines
pm2 logs APP_NAME --lines 100

# All PM2 apps
pm2 logs

# PM2 monitoring dashboard
pm2 monit

Nginx:

# Access logs
tail -f /var/log/nginx/access.log

# Error logs
tail -f /var/log/nginx/error.log

# Site-specific logs (if configured)
tail -f /var/log/nginx/sansa-access.log
tail -f /var/log/nginx/docs-access.log

---

Check Resource Usage

# Overall system
htop  # or top

# Disk usage
df -h
du -sh /var/www/*

# Memory usage
free -h

# CPU and memory per process
ps aux | grep python
ps aux | grep node

# Network connections
netstat -tulpn
ss -tulpn

# PM2 resource monitoring
pm2 monit

---

Known Issues & Notes

Issue	Project	Status	Notes
Multiple restarts	isutech-backend (PM2)	⚠️ Monitor	Currently 23 restarts
Not running	MatchMind	🔴 Stopped	Code exists, no active service
Not running	isu-api (PM2)	🔴 Stopped	Check if needed
Development on prod	All projects	⚠️ Notice	Development happens directly on production server (Mac broken)

---

Security Notes

• SSH Access: Key-based authentication only
• Firewall: UFW enabled (ports 80, 443, 22)
• SSL Certificates: Let’s Encrypt (auto-renewal via certbot.timer)
• Password Protection: DocsHub uses HTTP Basic Auth
• Git Authentication: SSH keys or Personal Access Tokens
• Database: PostgreSQL restricted to localhost
• Redis: Restricted to localhost

Check Security:

# Firewall status
ufw status

# SSL certificate expiry
certbot certificates

# Failed login attempts
journalctl _SYSTEMD_UNIT=ssh.service | grep "Failed"

# Open ports
ss -tlnp

---

Contacts & Resources

• Server Provider: Hetzner Cloud
• Server IP: 46.224.40.5
• SSL Provider: Let’s Encrypt / Certbot
• Git Repositories: github.com/gedeza
• Documentation: https://docs.isutech.co.za

---

Quick Emergency Commands

# Service down - restart
systemctl restart [service-name]

# Check why service failed
systemctl status [service-name]
journalctl -u [service-name] -n 50

# High memory usage
free -h
ps aux --sort=-%mem | head -10
systemctl restart [problematic-service]

# Disk full
df -h
du -sh /var/www/* /var/log/*
journalctl --vacuum-time=7d  # Clear old logs

# Nginx issues
nginx -t  # Test config
systemctl restart nginx
tail -f /var/log/nginx/error.log

# Database issues
systemctl status postgresql@14-main
sudo -u postgres psql -c "SELECT 1;"

---

Document Version: 2.0 Last Updated: 2026-02-01 Previous Update: 2025-12-09 Author: iSu Technologies Operations Team