Hetzner Server Operations Guide (legacy — Dube TradePort is now primary)

Hetzner Server Operations Guide

Migration status — 2026-08-09

The primary server flipped. Most applications now run live on isutech-svr-01 (Dube TradePort/iConnect, 41.78.233.110), not Hetzner. For current app-by-app location, ports, and status, see the Server Management Guide — that page is kept current through the migration; this one is the deeper process/config reference and hasn’t been fully re-audited against Dube yet. Where this guide’s specifics (paths, service names, hardening details) conflict with the management guide, trust the management guide and treat this page’s Hetzner-specific content as historical until it’s reviewed. Hetzner itself is being emptied app-by-app, not decommissioned in one step — see the internal migration-plan tracker for what’s still outstanding (PoleloDMS, the isutech.co.za flagship site, DocsHub’s own hosting, SANSA/SACE/AutoSlip, and a handful of never-confirmed live-vs-retired apps).

Complete guide to managing iSu Technologies’ production servers.

---

🖥️ Server Information

Migration status — 2026-08-09, corrects the note below

The note directly below (“all applications listed below are still live on Hetzner”) is stale — it described the pre-commissioning state from 2026-06/07. As of 2026-08-09 16 apps have cut over to Dube TradePort (isutech-svr-01) and are DNS-flipped, production traffic. Only a handful remain on Hetzner: isutech.co.za flagship, DocsHub itself (this site), SANSA, SACE, AutoSlip, plus a few whose live-vs-retired status was never confirmed (Umdoni, MatchMind). For the current, per-app source of truth, use Dube Box — Where Everything Lives (what’s on Dube) and the Server Management Guide (both boxes side by side). The “Hosted Applications” list further down on this page is the original pre-migration inventory — treat it as historical, not current.

Migration in progress — historical, 2026-06/07

iSu Technologies is migrating its full application estate from Hetzner to a new Dube TradePort/iConnect server. All applications listed below are still live on Hetzner — the new server is commissioned, hardened, and empty, waiting for the app-by-app migration. Box-level facts on this page (specs, access method, security posture) are kept current for both servers; the Hosted Applications list and everything after it still describes Hetzner only, until each app actually cuts over.

Server Details — Hetzner (legacy, being emptied — no longer “current production”)

• Provider: Hetzner
• IP Address: 46.224.40.5
• Operating System: Ubuntu 22.04 LTS · ARM64
• SSH Access: ssh root@46.224.40.5
• Server Name: iSu Technologies Production Server (production-server-01)
• Still hosts (2026-08-09): isutech.co.za flagship, DocsHub, SANSA, SACE, AutoSlip — see the migration-status note above for the full current picture.

Server Details — Dube TradePort (primary/production since 2026-08-09)

• Provider: Dube TradePort / iConnect
• Public IP: 41.78.233.110 (internal 10.2.38.10)
• Operating System: Ubuntu 24.04.4 LTS · x86_64
• SSH Access: ssh -p 33822 isutech-admin@41.78.233.110 — key-only auth, no root login (see SSH Access below — the access model differs meaningfully from Hetzner’s, not just the connection string)
• Server Name: isutech-svr-01
• Spec: 16 vCPU · 24 GB RAM · 200 GB OS + 800 GB data drive
• Hardening: UFW (default-deny) + CrowdSec with firewall-bouncer + auditd + AIDE (file-integrity monitoring). Hetzner also now runs CrowdSec (fail2ban was retired there too — see the Server Management Guide); Dube’s remaining edge is auditd + AIDE, which Hetzner still lacks.
• What’s live here: 16 apps, DNS-flipped — full list at Dube Box — Where Everything Lives

Hosted Applications — original pre-migration inventory (historical, do not use to find where an app lives today)

Total: 14 Applications (as inventoried 2026-06/07, before any app had moved)

1. sansa.isutech.co.za - SANSA Educator Platform (Vision 2030) - Frontend (PM2) + Backend (systemd)
2. tumi.isutech.co.za - Tumi Mentorship Platform - Frontend (nginx) + Backend (systemd)
3. sace.isutech.co.za - SACE Provider Intelligence System - Frontend (systemd) + Backend (systemd)
4. prospectiq - ProspectIQ Lead Management - 5 systemd services (frontend, api, celery, celery-beat, worker)
5. thrivesend - ThriveSend B2B2G Platform - Full-stack Next.js (PM2)
6. umdoni.gov.za - Umdoni Municipality Website - PHP/nginx
7. docs.isutech.co.za - Documentation Hub - MkDocs Material (nginx static)
8. isutech.co.za - iSuTech Corporate Site - Frontend + Backend + Web (nginx + PM2)
9. nhlanhla.isutech.co.za - Portfolio Website - Static site (nginx)
10. monitor.isutech.co.za - iSu Monitor - Monitoring dashboard
11. AutoSlip - Receipt Automation Service - Systemd service
12. MatchMind - Football Prediction Platform - Development (stopped)

---

🔐 Access & Authentication

Two different access models — don’t mix them up

Hetzner (production-server-01) uses shared root SSH + a root password reset via Hetzner’s console — this is legacy and being phased out specifically because of the risks a shared root password carries. isutech-svr-01 (Dube TradePort) uses a materially safer model: key-only auth, a non-root sudo user (isutech-admin), no root SSH login at all, no password-based access of any kind. This is the standard going forward for any new server iSu commissions, not just this one — see the Dube TradePort standard below.

SSH Access — production-server-01 (Hetzner) — legacy, current for now

# Connect to server
ssh root@46.224.40.5

# First time connection
# You'll be prompted to accept the server's fingerprint
# Type "yes" to continue

Run your work inside tmux

SSH sessions on remote servers should run inside tmux so they persist across network drops and support split panes. See the tmux Tutorial for a full walkthrough.

User Management

# Current user
whoami

# List all users
cat /etc/passwd

# Add new user (if needed)
sudo useradd -m -s /bin/bash username

# Set password for user
sudo passwd username

# Add user to sudo group
sudo usermod -aG sudo username

---

🔑 SSH Key Setup for New Team Members — Hetzner (legacy)

This section guides new team members through setting up SSH key access to the Hetzner server. This workflow is specific to Hetzner’s shared-root-password model — see the Dube TradePort section further down for the new standard.

Step 1: Check for Existing SSH Key

On macOS/Linux:

ls -la ~/.ssh/

On Windows (PowerShell):

ls ~/.ssh/

• If you see id_ed25519 or id_rsa files, skip to Step 3
• If you get “No such file” or no key files, continue to Step 2

Step 2: Generate SSH Key (if needed)

On macOS/Linux:

ssh-keygen -t ed25519 -C "your.email@isutech.co.za"

On Windows (PowerShell):

ssh-keygen -t ed25519 -C "your.email@isutech.co.za"

• Press Enter to accept default location (~/.ssh/id_ed25519)
• Press Enter twice for no passphrase (or set one for extra security)

Step 3: Display Your Public Key

On macOS/Linux:

cat ~/.ssh/id_ed25519.pub

On Windows (PowerShell):

Get-Content ~/.ssh/id_ed25519.pub

Copy the entire output - it starts with ssh-ed25519...

Step 4: Add Key to Server

Option A: Ask a team member with access Send your public key to a team member who already has server access. They will run:

ssh root@46.224.40.5
echo 'YOUR_PUBLIC_KEY_HERE' >> ~/.ssh/authorized_keys
exit

Option B: Use root password (if you have it)

ssh root@46.224.40.5
# Enter password when prompted
echo 'YOUR_PUBLIC_KEY_HERE' >> ~/.ssh/authorized_keys
exit

Step 5: Test Connection

ssh root@46.224.40.5

You should connect without being asked for a password.

Resetting Root Password (Admin Only)

If you need the root password and don’t have SSH key access:

1. Log into Hetzner Cloud Console
2. Select the server (autoslip-prod-01 / 46.224.40.5)
3. Go to Rescue tab
4. Click “Reset Root Password” (under ROOT PASSWORD section)
5. The new password will be displayed on screen
6. Use this password to SSH in and add your SSH key
7. Consider resetting the password again after setup for security

Managing Authorized Keys

View current authorized keys:

cat ~/.ssh/authorized_keys

Add a new key:

echo 'ssh-ed25519 AAAAC3... user@email.com' >> ~/.ssh/authorized_keys

Remove a key:

# Edit the file and remove the line containing the key
nano ~/.ssh/authorized_keys

Troubleshooting SSH Issues

Problem	Solution
”Permission denied (publickey)“	Your public key isn’t on the server - ask a team member to add it
”Connection refused”	Server may be down or SSH service not running
”Connection timed out”	Network issue or firewall blocking port 22
”Host key verification failed”	Server was reinstalled - remove old key with ssh-keygen -R 46.224.40.5

---

🔑 SSH Access — isutech-svr-01 (Dube TradePort) — current standard

No shared root password, no console-based password reset, no root SSH login at all. Access is granted by adding your public key to the existing isutech-admin sudo account — done once by whoever already has access, not self-service via a shared credential.

Connecting

ssh -p 33822 isutech-admin@41.78.233.110

Note the non-default SSH port (33822) — the server’s internal SSH daemon listens on the standard port 22, but Dube TradePort’s NAT maps external port 33822 to it. Every command against this box needs -p 33822; forgetting it is the most common mistake.

Set up an SSH config alias

Add this to your local ~/.ssh/config so you can just type ssh dube-tradeport instead of the full command every time:

Host dube-tradeport
    HostName 41.78.233.110
    Port 33822
    User isutech-admin
    IdentityFile ~/.ssh/dube_tradeport_isutech-admin

Requesting access

1. Generate your own dedicated keypair (don’t reuse a key from another server or your personal machine):

   ssh-keygen -t ed25519 -C "your.email@isutech.co.za (dube-tradeport)" -f ~/.ssh/dube_tradeport_yourname

   Set a real passphrase — store it in a password manager (e.g. Bitwarden), not a plaintext note.
2. Send your public key (the .pub file’s contents — never the private key) to whoever already has isutech-admin access.
3. They add it to ~/.ssh/authorized_keys on the server (ssh-copy-id or a manual append, run once by them, on your behalf).
4. Test: ssh -i ~/.ssh/dube_tradeport_yourname -p 33822 isutech-admin@41.78.233.110 — should log in without ever asking for a password.

Why this model, not Hetzner’s

The one-time-handover pattern — take initial access once (root or an initial sudo account), lock it down to key-based auth immediately, then manage all further account/key changes entirely ourselves — is the general standard to insist on with any managed-hosting provider, not just Dube TradePort/iConnect. It removes the provider from routine account administration and eliminates the shared-secret risk that Hetzner’s root-password model carries (anyone who’s ever had the root password, or seen it in a chat/email, retains access until it’s rotated — which rarely happens in practice).

Hardening in place on this server

• PasswordAuthentication no, PermitRootLogin no, key-only auth enforced at the sshd level (not just “we don’t tell people the password” — password auth is structurally disabled).
• UFW, default-deny, only 22 (SSH, internally)/80/443 open.
• CrowdSec + crowdsec-firewall-bouncer — bans IPs on SSH brute-force patterns, same as Hetzner’s existing setup (this part is consistent between the two boxes — not fail2ban, which was retired in favour of CrowdSec).
• auditd + AIDE (file-integrity monitoring) — watches system binaries, cron, sudoers, and SSH keys for tampering, and catches both successful and failed attempts. This closes a gap Hetzner still has open.

---

📂 Directory Structure

Application Directories

/var/www/
├── sansa-frontend/                # SANSA Frontend (Next.js on PM2)
│   ├── src/                       # Source code
│   ├── public/                    # Static assets
│   ├── .env.production            # Production config
│   ├── package.json               # Node dependencies
│   └── .next/                     # Built application (712MB)
│
├── sansa-backend/                 # SANSA Backend (FastAPI on systemd)
│   ├── app/                       # Application code
│   │   ├── api/                   # API routes
│   │   ├── core/                  # Core config
│   │   ├── models/                # Database models
│   │   └── main.py                # Entry point
│   ├── venv/                      # Python virtual environment
│   ├── .env                       # Environment variables
│   ├── alembic/                   # Database migrations
│   └── requirements.txt           # Python dependencies (590MB)
│
├── tumi-frontend/                 # Tumi Frontend (Static site)
│   ├── build/                     # Built static files
│   ├── assets/                    # Static assets
│   └── index.html                 # Entry point (145MB)
│
├── tumi-backend/                  # Tumi Backend (FastAPI on systemd)
│   ├── app/                       # Application code
│   ├── venv/                      # Python virtual environment
│   ├── .env                       # Environment variables
│   └── requirements.txt           # Python dependencies (94MB)
│
├── sace-frontend/                 # SACE Frontend (systemd)
│   ├── src/                       # Source code
│   ├── public/                    # Static assets
│   ├── .env.production            # Production config
│   ├── package.json               # Node dependencies
│   └── .next/                     # Built application
│
├── sace-backend/                  # SACE Backend (FastAPI on systemd)
│   ├── app/                       # Application code
│   │   ├── api/                   # API routes
│   │   ├── core/                  # Core config
│   │   ├── models/                # Database models
│   │   └── main.py                # Entry point
│   ├── venv/                      # Python virtual environment
│   ├── .env                       # Environment variables
│   ├── alembic/                   # Database migrations
│   └── requirements.txt           # Python dependencies
│
├── thrivesend/                    # ThriveSend (Next.js on PM2)
│   ├── src/                       # Source code
│   ├── public/                    # Static assets
│   ├── .env.production            # Production config
│   ├── package.json               # Node dependencies
│   └── .next/                     # Built application (1.2GB)
│
├── prospect-iq/                   # ProspectIQ (5 systemd services)
│   ├── frontend/                  # Frontend application
│   ├── backend/                   # Backend API
│   ├── celery/                    # Celery task queue
│   └── worker/                    # Background workers
│
├── umdoni/                        # Umdoni Municipality Website
│   ├── public/                    # PHP application
│   ├── assets/                    # Static assets
│   └── index.php                  # Entry point (284MB)
│
├── nhlanhla-portfolio/            # Portfolio Website
│   ├── build/                     # Built static files
│   └── assets/                    # Static assets (30MB)
│
├── isutech-frontend/              # iSuTech Corporate Frontend
│   └── [application files]
│
├── isutech-backend/               # iSuTech Corporate Backend (PM2)
│   └── [application files]
│
├── isutech-web/                   # iSuTech Corporate Web
│   └── [application files]
│
├── matchmind/                     # MatchMind (Development - stopped)
│   └── [application files]
│
└── docs/                          # Documentation Hub
    ├── repo/                      # Git repository
    │   ├── docs/                  # Markdown content
    │   ├── site/                  # Built static site
    │   └── mkdocs.yml             # MkDocs configuration
    └── scripts/
        └── update-docs.sh         # Auto-update script

Development Directory

/projects/active/
├── thrive-send-b2b2g/            # ThriveSend development (2.0GB)
├── isutech-prospect-iq/          # ProspectIQ development (696MB)
├── umdoni-website/               # Umdoni development (271MB)
├── mnyandu-portfolio_from_root/  # Portfolio development (147MB)
├── matchmind/                    # MatchMind development (19MB)
├── iSuMonitor/                   # Monitor development (7.6MB)
├── autoslip/                     # AutoSlip development (1.7MB)
└── MyProfile/                    # Profile docs (56KB)

Note: Development happens at /projects/active/ and is deployed to /var/www/ for production.

System Configuration Directories

/etc/
├── nginx/
│   ├── sites-available/          # Available site configs
│   │   ├── api.isutech.co.za
│   │   ├── autoslip
│   │   ├── docs.isutech.co.za
│   │   ├── isutech.co.za
│   │   ├── monitor.isutech.co.za
│   │   ├── nhlanhla.isutech.co.za
│   │   ├── prospectiq
│   │   ├── sace.isutech.co.za
│   │   ├── sansa.isutech.co.za
│   │   ├── thrivesend
│   │   ├── tumi.isutech.co.za
│   │   ├── umdoni.gov.za
│   │   └── (matchmind - not active)
│   └── sites-enabled/            # Active site configs (13 symlinks)
│
├── systemd/system/               # Systemd service files
│   ├── sansa-backend.service
│   ├── tumi-backend.service
│   ├── sace-backend.service
│   ├── sace-frontend.service
│   ├── autoslip.service
│   ├── prospectiq-api.service
│   ├── prospectiq-frontend.service
│   ├── prospectiq-celery.service
│   ├── prospectiq-celery-beat.service
│   ├── prospectiq-worker.service
│   └── docs-update.timer        # Auto-update scheduler
│
└── letsencrypt/                  # SSL certificates
    ├── live/
    │   ├── api.isutech.co.za/
    │   ├── docs.isutech.co.za/
    │   ├── isutech.co.za/
    │   ├── monitor.isutech.co.za/
    │   ├── nhlanhla.isutech.co.za/
    │   ├── sace.isutech.co.za/
    │   ├── sansa.isutech.co.za/
    │   ├── tumi.isutech.co.za/
    │   └── umdoni.gov.za/
    └── renewal/                  # Auto-renewal configs

---

🚀 Application Management

SACE Backend (FastAPI)

Service Control:

# Check status
sudo systemctl status sace-backend

# Start service
sudo systemctl start sace-backend

# Stop service
sudo systemctl stop sace-backend

# Restart service
sudo systemctl restart sace-backend

# Enable auto-start on boot
sudo systemctl enable sace-backend

# View logs
sudo journalctl -u sace-backend -f

# View last 100 lines
sudo journalctl -u sace-backend -n 100

Manual Operations:

# Navigate to directory
cd /var/www/sace-backend

# Activate virtual environment
source venv/bin/activate

# Check environment variables
cat .env

# Run migrations
alembic upgrade head

# Test API directly
uvicorn app.main:app --host 0.0.0.0 --port 8001

# Run Python scripts
python seed_demo_data.py
python seed_activities.py

Configuration Files:

• /var/www/sace-backend/.env - Environment variables
• /etc/systemd/system/sace-backend.service - Systemd service

---

SACE Frontend (Next.js)

Service Control:

# Check status
sudo systemctl status sace-frontend

# Start service
sudo systemctl start sace-frontend

# Stop service
sudo systemctl stop sace-frontend

# Restart service
sudo systemctl restart sace-frontend

# Enable auto-start on boot
sudo systemctl enable sace-frontend

# View logs
sudo journalctl -u sace-frontend -f

Manual Operations:

# Navigate to directory
cd /var/www/sace-frontend

# Check environment variables
cat .env.production

# Install dependencies
npm install

# Build application
npm run build

# Start manually (for testing)
npm start

# Development mode (not for production)
npm run dev

Configuration Files:

• /var/www/sace-frontend/.env.production - Environment variables
• /etc/systemd/system/sace-frontend.service - Systemd service

---

SANSA Backend (FastAPI)

Service Control:

# Check status
sudo systemctl status sansa-backend

# Start service
sudo systemctl start sansa-backend

# Stop service
sudo systemctl stop sansa-backend

# Restart service
sudo systemctl restart sansa-backend

# Enable auto-start on boot
sudo systemctl enable sansa-backend

# View logs
sudo journalctl -u sansa-backend -f

# View last 100 lines
sudo journalctl -u sansa-backend -n 100

Manual Operations:

# Navigate to directory
cd /var/www/sansa-backend

# Activate virtual environment
source venv/bin/activate

# Check environment variables
cat .env

# Run migrations
alembic upgrade head

# Test API directly
uvicorn app.main:app --host 0.0.0.0 --port 8003

# Run database seeding
python scripts/seed_database.py

Configuration Files:

• /var/www/sansa-backend/.env - Environment variables
• /etc/systemd/system/sansa-backend.service - Systemd service

API Endpoints:

• Base URL: https://sansa.isutech.co.za/api/v1
• API Docs: https://sansa.isutech.co.za/api/docs
• Health Check: https://sansa.isutech.co.za/api/v1/health

---

SANSA Frontend (Next.js/PM2)

Service Control:

# Check status
pm2 status sansa-frontend

# Start service
pm2 start sansa-frontend

# Stop service
pm2 stop sansa-frontend

# Restart service
pm2 restart sansa-frontend

# View logs
pm2 logs sansa-frontend

# View last 100 lines
pm2 logs sansa-frontend --lines 100

Manual Operations:

# Navigate to directory
cd /var/www/sansa-frontend

# Check environment variables
cat .env.production

# Install dependencies
npm install

# Build application
npm run build

# Start manually (for testing)
npm start

Configuration Files:

• /var/www/sansa-frontend/.env.production - Environment variables
• PM2 managed (no systemd service)

Test Users:

Email	Password	Role
admin@sansa.gov.za	Admin@2024	Admin (Superuser)
dan.matsapola@sansa.gov.za	Sansa@2024	Admin
facilitator@sansa.gov.za	Sansa@2024	Facilitator
viewer@sansa.gov.za	Sansa@2024	Viewer

---

Tumi Backend (FastAPI)

Service Control:

# Check status
sudo systemctl status tumi-backend

# Start service
sudo systemctl start tumi-backend

# Stop service
sudo systemctl stop tumi-backend

# Restart service
sudo systemctl restart tumi-backend

# View logs
sudo journalctl -u tumi-backend -f

# View last 100 lines
sudo journalctl -u tumi-backend -n 100

Manual Operations:

# Navigate to directory
cd /var/www/tumi-backend

# Activate virtual environment
source venv/bin/activate

# Check environment variables
cat .env

# Run migrations
alembic upgrade head

# Test API directly
uvicorn app.main:app --host 0.0.0.0 --port 8004

Configuration Files:

• /var/www/tumi-backend/.env - Environment variables
• /etc/systemd/system/tumi-backend.service - Systemd service

API Endpoints:

• Base URL: https://tumi.isutech.co.za/api
• Health Check: https://tumi.isutech.co.za/api/health

---

Tumi Frontend (Static Site)

Deployment: The Tumi frontend is a static site served directly by nginx from /var/www/tumi-frontend/.

Manual Operations:

# Navigate to directory
cd /var/www/tumi-frontend

# View files
ls -lah

# Check nginx config
cat /etc/nginx/sites-available/tumi.isutech.co.za

No service management needed - nginx serves static files directly.

---

ThriveSend (Next.js/PM2)

Service Control:

# Check status
pm2 status thrivesend

# Start service
pm2 start thrivesend

# Stop service
pm2 stop thrivesend

# Restart service
pm2 restart thrivesend

# View logs
pm2 logs thrivesend

# View last 100 lines
pm2 logs thrivesend --lines 100

Manual Operations:

# Navigate to directory
cd /var/www/thrivesend

# Check environment variables
cat .env.production

# Install dependencies
npm install

# Build application
npm run build

# Start manually
npm start

Configuration Files:

• /var/www/thrivesend/.env.production - Environment variables
• PM2 managed (no systemd service)

---

ProspectIQ (5 Systemd Services)

ProspectIQ runs with 5 separate systemd services for scalability:

Service Control:

# Check all ProspectIQ services
sudo systemctl status prospectiq-api prospectiq-frontend prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Restart all services
sudo systemctl restart prospectiq-api prospectiq-frontend prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Start individual services
sudo systemctl start prospectiq-api
sudo systemctl start prospectiq-frontend
sudo systemctl start prospectiq-celery
sudo systemctl start prospectiq-celery-beat
sudo systemctl start prospectiq-worker

# View logs for specific service
sudo journalctl -u prospectiq-api -f
sudo journalctl -u prospectiq-celery -f

Service Descriptions:

• prospectiq-api - Backend API service (FastAPI)
• prospectiq-frontend - Frontend application (Next.js)
• prospectiq-celery - Celery worker for async tasks
• prospectiq-celery-beat - Celery beat scheduler
• prospectiq-worker - Background worker service

Manual Operations:

# Navigate to directory
cd /var/www/prospect-iq

# View service configurations
ls -lah /etc/systemd/system/prospectiq-*

# Check nginx config
cat /etc/nginx/sites-available/prospectiq

Configuration Files:

• /var/www/prospect-iq/.env - Environment variables
• /etc/systemd/system/prospectiq-*.service - Systemd services (5 files)
• /etc/nginx/sites-available/prospectiq - Nginx configuration

---

Umdoni Municipality Website

Deployment: The Umdoni website is a PHP-based application served by nginx from /var/www/umdoni/.

Manual Operations:

# Navigate to directory
cd /var/www/umdoni

# View files
ls -lah

# Check nginx config
cat /etc/nginx/sites-available/umdoni.gov.za

# Check PHP-FPM status (if applicable)
sudo systemctl status php*-fpm

Configuration Files:

• /etc/nginx/sites-available/umdoni.gov.za - Nginx configuration
• PHP files served directly by nginx

---

Nhlanhla Portfolio Website

Deployment: Portfolio website is a static site served by nginx from /var/www/nhlanhla-portfolio/.

Manual Operations:

# Navigate to directory
cd /var/www/nhlanhla-portfolio

# View files
ls -lah

# Check nginx config
cat /etc/nginx/sites-available/nhlanhla.isutech.co.za

No service management needed - nginx serves static files directly.

---

iSuTech Corporate Site

Service Control:

# Check PM2 backend status
pm2 status isutech-backend

# Restart backend
pm2 restart isutech-backend

# View logs
pm2 logs isutech-backend

Manual Operations:

# Navigate to directories
cd /var/www/isutech-frontend
cd /var/www/isutech-backend
cd /var/www/isutech-web

# Check nginx configs
cat /etc/nginx/sites-available/isutech.co.za
cat /etc/nginx/sites-available/api.isutech.co.za

---

AutoSlip (Receipt Automation)

Service Control:

# Check status
sudo systemctl status autoslip

# Start service
sudo systemctl start autoslip

# Stop service
sudo systemctl stop autoslip

# Restart service
sudo systemctl restart autoslip

# View logs
sudo journalctl -u autoslip -f

Manual Operations:

# Development code location
cd /projects/active/autoslip

# Check service configuration
cat /etc/systemd/system/autoslip.service

# Check nginx config
cat /etc/nginx/sites-available/autoslip

Note: AutoSlip runs as a systemd service. Development code is at /projects/active/autoslip/.

---

Documentation Hub (MkDocs)

Auto-Update System:

# Check update timer status
sudo systemctl status docs-update.timer

# Check update service status
sudo systemctl status docs-update.service

# View update logs
sudo journalctl -u docs-update.service -n 50

# Manual update
sudo /var/www/docs/scripts/update-docs.sh

# View recent update log
tail -n 30 /var/log/docs-update.log

Manual Operations:

# Navigate to repository
cd /var/www/docs/repo

# Pull latest changes
git pull origin main

# Build documentation
mkdocs build

# Serve locally (for testing)
mkdocs serve

# Deploy to production
mkdocs build
# Site built to: /var/www/docs/repo/site

Configuration Files:

• /var/www/docs/repo/mkdocs.yml - MkDocs configuration
• /var/www/docs/scripts/update-docs.sh - Update script

---

🌐 Nginx Configuration

Viewing Configurations

# List all site configs
ls -la /etc/nginx/sites-available/

# View docs config
cat /etc/nginx/sites-available/docs.isutech.co.za

# View SACE config
cat /etc/nginx/sites-available/sace.isutech.co.za

# Check enabled sites
ls -la /etc/nginx/sites-enabled/

Managing Nginx

# Test configuration syntax
sudo nginx -t

# Reload configuration (no downtime)
sudo systemctl reload nginx

# Restart Nginx
sudo systemctl restart nginx

# Check status
sudo systemctl status nginx

# View error logs
sudo tail -f /var/log/nginx/error.log

# View access logs
sudo tail -f /var/log/nginx/access.log

Adding New Site

# 1. Create configuration file
sudo nano /etc/nginx/sites-available/newsite.isutech.co.za

# 2. Enable site (create symlink)
sudo ln -s /etc/nginx/sites-available/newsite.isutech.co.za /etc/nginx/sites-enabled/

# 3. Test configuration
sudo nginx -t

# 4. Reload Nginx
sudo systemctl reload nginx

# 5. Install SSL certificate
sudo certbot --nginx -d newsite.isutech.co.za

---

🗄️ Database Management

PostgreSQL

Service Control:

# Check status
sudo systemctl status postgresql

# Start service
sudo systemctl start postgresql

# Stop service
sudo systemctl stop postgresql

# Restart service
sudo systemctl restart postgresql

Database Operations:

# Connect as postgres user
sudo -u postgres psql

# List databases
sudo -u postgres psql -c "\l"

# Connect to SACE database
sudo -u postgres psql -d sace_production

# Using SACE user credentials
PGPASSWORD="SaceProd2025Secure" psql -U sace_user -d sace_production -h localhost

# Backup database
sudo -u postgres pg_dump sace_production > /tmp/sace_backup_$(date +%Y%m%d).sql

# Restore database
sudo -u postgres psql sace_production < /tmp/sace_backup_20251111.sql

# Check database size
sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('sace_production'));"

Common Queries:

-- Inside psql
\l                              -- List databases
\du                             -- List users
\dt                             -- List tables
\d table_name                   -- Describe table
\q                              -- Quit

-- Sample queries
SELECT COUNT(*) FROM providers;
SELECT COUNT(*) FROM educators;
SELECT COUNT(*) FROM activities;
SELECT COUNT(*) FROM enrollments;

---

Redis

Service Control:

# Check status
sudo systemctl status redis

# Start service
sudo systemctl start redis

# Stop service
sudo systemctl stop redis

# Restart service
sudo systemctl restart redis

Redis Operations:

# Connect to Redis CLI
redis-cli

# Inside redis-cli:
# ping                          # Test connection
# keys *                        # List all keys
# flushall                      # Clear all data (DANGER!)
# info                          # Server info
# quit                          # Exit

---

🔒 SSL Certificates

Certbot Management

Check Certificate Status:

# List all certificates
sudo certbot certificates

# Check specific certificate
sudo certbot certificates --cert-name docs.isutech.co.za

Renew Certificates:

# Test renewal (dry run)
sudo certbot renew --dry-run

# Manually renew
sudo certbot renew

# Renew specific certificate
sudo certbot renew --cert-name docs.isutech.co.za

Install New Certificate:

# For Nginx
sudo certbot --nginx -d newdomain.isutech.co.za

# Manual mode (no auto-config)
sudo certbot certonly --nginx -d newdomain.isutech.co.za

Auto-Renewal:

# Certbot sets up automatic renewal via systemd timer
# Check timer status
sudo systemctl status certbot.timer

# View renewal logs
sudo journalctl -u certbot.service

---

📊 Monitoring & Logs

Quick Reference Available

For a comprehensive monitoring command reference including PostgreSQL monitoring, emergency procedures, and health check scripts, see the Server Monitoring Quick Reference.

System Health

Disk Space:

# Check disk usage
df -h

# Check directory sizes
du -sh /var/www/*

# Find large files
sudo find / -type f -size +100M -exec ls -lh {} \;

Memory & CPU:

# Memory usage
free -h

# CPU usage
top

# Process list
htop  # If installed

# Running processes
ps aux | grep python
ps aux | grep node

Network:

# Check open ports
sudo netstat -tulpn

# Check listening services
sudo ss -tulpn

# Test port connectivity
curl http://localhost:8001/api/health
curl http://localhost:3000

Application Logs

SACE Backend:

# Live logs
sudo journalctl -u sace-backend -f

# Last 100 lines
sudo journalctl -u sace-backend -n 100

# Logs from today
sudo journalctl -u sace-backend --since today

# Logs with error level
sudo journalctl -u sace-backend -p err

SACE Frontend:

# Live logs
sudo journalctl -u sace-frontend -f

# Last 100 lines
sudo journalctl -u sace-frontend -n 100

Nginx:

# Access logs
sudo tail -f /var/log/nginx/access.log

# Error logs
sudo tail -f /var/log/nginx/error.log

# Search for errors
sudo grep "error" /var/log/nginx/error.log

# Site-specific logs (if configured)
sudo tail -f /var/log/nginx/sace-access.log
sudo tail -f /var/log/nginx/docs-access.log

---

🔧 Maintenance Tasks

Regular Maintenance (Weekly)

# 1. Update system packages
sudo apt update
sudo apt upgrade -y

# 2. Check disk space
df -h

# 3. Clean old logs
sudo journalctl --vacuum-time=7d

# 4. Check service status
sudo systemctl status sansa-backend tumi-backend sace-backend sace-frontend \
  autoslip prospectiq-api prospectiq-frontend prospectiq-celery \
  prospectiq-celery-beat prospectiq-worker nginx postgresql redis
pm2 status

# 5. Verify SSL certificate expiry
sudo certbot certificates

# 6. Backup database
sudo -u postgres pg_dump sace_production > ~/backups/sace_backup_$(date +%Y%m%d).sql

Monthly Tasks

# 1. Review and rotate logs
sudo logrotate -f /etc/logrotate.conf

# 2. Check for security updates
sudo apt list --upgradable | grep -i security

# 3. Review disk usage trends
du -sh /var/www/* /var/log/*

# 4. Test backup restoration
# Restore to test database

# 5. Review failed systemd services
sudo systemctl --failed

---

🚨 Troubleshooting

Service Won’t Start

# 1. Check service status
sudo systemctl status service-name

# 2. View detailed logs
sudo journalctl -u service-name -n 50

# 3. Check configuration syntax
# For Nginx:
sudo nginx -t

# For Python/FastAPI:
cd /var/www/sace-backend
source venv/bin/activate
python -m app.main

# 4. Check port conflicts
sudo netstat -tulpn | grep :8001

Application Not Accessible

# 1. Check Nginx status
sudo systemctl status nginx

# 2. Check DNS resolution
nslookup sace.isutech.co.za

# 3. Check firewall rules
sudo ufw status

# 4. Test local connectivity
curl http://localhost:8001/api/health
curl http://localhost:3000

# 5. Check SSL certificate
sudo certbot certificates

# 6. Review Nginx error logs
sudo tail -f /var/log/nginx/error.log

Database Connection Issues

# 1. Check PostgreSQL status
sudo systemctl status postgresql

# 2. Test connection
sudo -u postgres psql -c "SELECT 1;"

# 3. Check user permissions
sudo -u postgres psql -c "\du"

# 4. Test SACE user connection
PGPASSWORD="SaceProd2025Secure" psql -U sace_user -d sace_production -h localhost -c "SELECT 1;"

# 5. Check PostgreSQL logs
sudo tail -f /var/log/postgresql/postgresql-14-main.log

High Memory/CPU Usage

# 1. Identify resource-hungry processes
top
htop

# 2. Check specific service resources
systemctl status sace-backend
systemctl status sace-frontend

# 3. Restart problematic service
sudo systemctl restart service-name

# 4. Check for memory leaks
# Review application logs for errors

---

🔒 Security Best Practices

User Access

• ✅ Use SSH key authentication (disable password auth)
• ✅ Create separate users for different team members
• ✅ Use sudo instead of logging in as root
• ✅ Regularly review user accounts and permissions

Firewall

# Check UFW status
sudo ufw status

# Allow specific ports only
sudo ufw allow 80/tcp    # HTTP
sudo ufw allow 443/tcp   # HTTPS
sudo ufw allow 22/tcp    # SSH

# Enable firewall
sudo ufw enable

isutech-svr-01 (Dube TradePort) goes further

On top of UFW, the new server also runs CrowdSec + crowdsec-firewall-bouncer (bans IPs on brute-force/probing patterns — not fail2ban, which is retired) and auditd + AIDE (file-integrity monitoring, catches tampering with system binaries/cron/sudoers/SSH keys). Hetzner currently only has the UFW layer. See the Dube TradePort SSH Access section above for the full hardening list.

Application Security

• ✅ Keep all software updated
• ✅ Use environment variables for secrets (never commit)
• ✅ Enable HTTPS for all applications
• ✅ Use strong passwords and rotate regularly
• ✅ Implement rate limiting on API endpoints
• ✅ Regular security audits

Database Security

• ✅ Use strong passwords
• ✅ Restrict database access to localhost
• ✅ Regular backups with tested restoration
• ✅ Limit user permissions (principle of least privilege)

---

📦 Deployment Procedures

Deploying SACE Updates

Backend Updates:

# 1. SSH to server
ssh root@46.224.40.5

# 2. Navigate to backend
cd /var/www/sace-backend

# 3. Backup current code
cp -r app app.backup

# 4. Pull latest changes (if using Git)
git pull origin main

# 5. Activate virtual environment
source venv/bin/activate

# 6. Install new dependencies
pip install -r requirements.txt

# 7. Run migrations
alembic upgrade head

# 8. Restart service
sudo systemctl restart sace-backend

# 9. Verify
curl http://localhost:8001/api/health
sudo journalctl -u sace-backend -n 20

Frontend Updates:

# 1. Navigate to frontend
cd /var/www/sace-frontend

# 2. Backup current build
cp -r .next .next.backup

# 3. Pull latest changes
git pull origin main

# 4. Install dependencies
npm install

# 5. Build application
npm run build

# 6. Restart service
sudo systemctl restart sace-frontend

# 7. Verify
curl http://localhost:3000
sudo journalctl -u sace-frontend -n 20

Deploying Documentation Updates

Automatic Process:

• Documentation auto-updates every 10 minutes via systemd timer
• Pulls from GitHub and rebuilds automatically

Manual Update:

# Run update script manually
sudo /var/www/docs/scripts/update-docs.sh

# View logs
tail -n 30 /var/log/docs-update.log

---

📝 Quick Reference Commands

Most Used Commands

# Check all systemd service statuses
sudo systemctl status sansa-backend tumi-backend sace-backend sace-frontend \
  autoslip prospectiq-api prospectiq-frontend prospectiq-celery \
  prospectiq-celery-beat prospectiq-worker nginx postgresql redis

# Check all PM2 processes
pm2 status

# Restart all backend services
sudo systemctl restart sansa-backend tumi-backend sace-backend \
  prospectiq-api prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Restart all frontend services (systemd)
sudo systemctl restart sace-frontend prospectiq-frontend

# Restart all PM2 applications
pm2 restart all

# View all logs in real-time
sudo journalctl -f

# Check disk space
df -h

# Check memory
free -h

# Test API health endpoints
curl https://sansa.isutech.co.za/api/v1/health
curl https://tumi.isutech.co.za/api/health
curl https://sace.isutech.co.za/api/health

# Update documentation manually
sudo /var/www/docs/scripts/update-docs.sh

# Backup all databases
sudo -u postgres pg_dump sace_production > ~/backups/sace_$(date +%Y%m%d).sql
sudo -u postgres pg_dump sansa_production > ~/backups/sansa_$(date +%Y%m%d).sql
sudo -u postgres pg_dump tumi_production > ~/backups/tumi_$(date +%Y%m%d).sql

# Check Nginx config and reload
sudo nginx -t && sudo systemctl reload nginx

# View all nginx sites
ls -la /etc/nginx/sites-enabled/

# Check all SSL certificates
sudo certbot certificates

---

📞 Support & Escalation

When to Restart Services

• Application not responding
• High memory usage
• After configuration changes
• After code deployment

When to Contact Support

• Persistent service failures
• Database corruption
• Security incidents
• SSL certificate issues
• Hardware problems

Emergency Procedures

1. Service Down: Restart service → Check logs → Verify config
2. Database Issue: Stop writes → Backup → Restore from backup
3. Security Breach: Isolate server → Review logs → Reset credentials
4. Disk Full: Clear old logs → Remove temp files → Expand disk

---

📚 Additional Resources

Documentation Links

• Nginx Documentation
• FastAPI Documentation
• Next.js Documentation
• PostgreSQL Documentation
• MkDocs Material

Internal Documentation

• SACE System Documentation

---

Last Updated: 02/01/2026 | Document Owner: Operations Team | Version: 2.0

---

📋 Application Inventory Summary

#	Application	Location	Domain	Service Type	Status
1	SANSA Frontend	/var/www/sansa-frontend/	sansa.isutech.co.za	PM2 (port 3003)	✅ Running
2	SANSA Backend	/var/www/sansa-backend/	sansa.isutech.co.za/api	systemd (port 8003)	✅ Running
3	Tumi Frontend	/var/www/tumi-frontend/	tumi.isutech.co.za	nginx static	✅ Serving
4	Tumi Backend	/var/www/tumi-backend/	tumi.isutech.co.za/api	systemd (port 8004)	✅ Running
5	SACE Frontend	/var/www/sace-frontend/	sace.isutech.co.za	systemd	✅ Running
6	SACE Backend	/var/www/sace-backend/	sace.isutech.co.za/api	systemd (port 8001)	✅ Running
7	ThriveSend	/var/www/thrivesend/	thrivesend	PM2	✅ Running
8	ProspectIQ API	/var/www/prospect-iq/	prospectiq	systemd	✅ Running
9	ProspectIQ Frontend	/var/www/prospect-iq/	prospectiq	systemd	✅ Running
10	ProspectIQ Celery	/var/www/prospect-iq/	-	systemd	✅ Running
11	ProspectIQ Beat	/var/www/prospect-iq/	-	systemd	✅ Running
12	ProspectIQ Worker	/var/www/prospect-iq/	-	systemd	✅ Running
13	DocsHub	/var/www/docs/	docs.isutech.co.za	nginx static	✅ Serving
14	Umdoni	/var/www/umdoni/	umdoni.gov.za	nginx + PHP	✅ Serving
15	iSuTech Corporate	/var/www/isutech-*/	isutech.co.za	nginx + PM2	⚠️ Partial
16	Portfolio	/var/www/nhlanhla-portfolio/	nhlanhla.isutech.co.za	nginx static	⚠️ Unknown
17	AutoSlip	Service only	autoslip	systemd	✅ Running
18	MatchMind	/var/www/matchmind/	-	-	🔴 Stopped

Total: 18 services across 14 applications Running: 14 services Partially Running: 1 (iSuTech) Unknown: 1 (Portfolio) Stopped: 2 (MatchMind services)