Server Management Guide — Dube TradePort (current) + Hetzner (legacy, being emptied)

Server Management Guide

Migration status — 2026-08-09

The migration has flipped: most applications now run live on isutech-svr-01 (Dube TradePort), not Hetzner. production-server-01 (Hetzner) is being emptied app-by-app — some apps’ code still runs there as dead weight pending decommission, a few haven’t moved yet at all (PoleloDMS, the isutech.co.za flagship site, DocsHub itself, SANSA/SACE/AutoSlip). The Project Inventory below reflects where each app actually serves traffic from today, not the historical Hetzner-only state. See the migration plan and wave runbooks (internal ops tracker) for exact status and what’s still outstanding.

---

Server Overview

Property	isutech-svr-01 (Dube TradePort — primary, live)	production-server-01 (Hetzner — legacy, being emptied)
Hostname	isutech-svr-01	production-server-01
Provider	Dube TradePort / iConnect	Hetzner
Public IP	41.78.233.110	46.224.40.5
Internal IP	10.2.38.10	—
OS	Ubuntu 24.04.4 LTS	Ubuntu 22.04 LTS
Architecture	x86_64	ARM64
vCPU / RAM	16 vCPU / 24 GB RAM	— (not tracked)
Storage	200 GB OS + 800 GB data (/data)	— (not tracked)
SSH	ssh -p 33822 isutech-admin@41.78.233.110 (port 33822, key-only, no root login)	ssh root@46.224.40.5 (port 22, root login — legacy, being retired with the box)
Web Server	Nginx	Nginx
Database	PostgreSQL 16, one role+DB per app	PostgreSQL 14
Cache	Redis (shared instance, one DB index per app that needs a real datastore, not just cache)	Redis
Process Manager	systemd + PM2 (pm2-isutech-admin.service resurrects on reboot — verified)	systemd + PM2
Container Runtime	Docker + Containerd (provisioned; PoleloDMS not yet deployed here)	Docker + Containerd
Firewall/IPS	UFW default-deny + CrowdSec + firewall-bouncer	UFW only
File integrity	auditd + AIDE	Not deployed

Where each app actually lives now is tracked precisely — path, port, runtime, domain, database — in the internal ops tracker’s box map (updated as apps cut over; ask Nhlanhla if you need current access). The table below is the human-readable summary.

---

Project Inventory

Live on Dube (isutech-svr-01) — 15 apps, 41.78.233.110

#	Project	Path (on Dube)	Domain	Runtime	Database
1	Portfolio	/data/www/nhlanhla-portfolio/	nhlanhla.isutech.co.za	nginx static	—
2	DealVault	/data/www/dealvault/	dealvault.isutech.co.za	PM2 (3050)	dealvault
3	BuildQ API	/data/www/buildq/apps/api/	buildq.isutech.co.za/api	PM2 (4040)	buildq
4	BuildQ Web	/data/www/buildq/apps/web/	buildq.isutech.co.za	PM2 (3040)	buildq (shared)
5	Buka	/opt/buka/	buka.isutech.co.za	systemd (8000)	SQLite buka.db
6	Tumi Backend	/data/www/tumi-backend/	tumi.isutech.co.za/api	systemd (8004)	SQLite wedding.db
7	Tumi Frontend	/data/www/tumi-frontend/	tumi.isutech.co.za	nginx static	—
8	ThriveSend	/data/www/thrivesend/	thrivesend.isutech.co.za	PM2 (3005)	thrivesend_b2b2g
9	ConformEdge	/data/www/conformedge/	conformedge.co.za (external domain — GoDaddy, not Afrihost)	PM2 (3020)	conformedge
10	KopanoWorks API	/data/www/kopanoworks/apps/api/	kopanoworks.isutech.co.za	PM2 (3010)	kopanoworks
11	KopanoWorks Admin	/data/www/kopanoworks/apps/web-admin/	admin.kopanoworks.isutech.co.za	PM2 (3011)	kopanoworks (shared)
12	ProspectIQ	/data/www/prospect-iq/	prospectiq.isutech.co.za	systemd (4 services: api, frontend, worker, beat)	prospectiq_prod
13	QueueHere	/data/www/queue-here/	queuehere.isutech.co.za	systemd (3000)	queuehere_prod
14	PropertyData Engine	/data/www/propertydata-engine/	internal only — called by AssessFlow over 127.0.0.1:8001	systemd (8001)	propertydata_engine + reads AssessFlow’s DB directly
15	AssessFlow	/data/www/assess-flow/	assess-flow.isutech.co.za	PM2, 3 processes (backend :5004, meims-worker :5005, frontend :3006)	assessflow

Still on Hetzner (production-server-01) — not yet migrated

#	Project	Location	Domain	Runtime	Notes
1	PoleloDMS	/opt/polelo-dms/	polelodms.co.za + *.polelodms.isutech.co.za (wildcard, multi-tenant)	Docker Compose — app, worker, Nextcloud, Meilisearch, Redis; prod and staging both running	Not started. Detached HEAD on the deployed commit — confirm intentional before touching. Nextcloud data volume size never measured.
2	iSu Tech Frontend (flagship)	/var/www/isutech-frontend/	isutech.co.za	nginx static	Not started. Unversioned — needs git init + salvage before cloning anywhere, same pattern used for the portfolio site.
3	DocsHub (this site)	/var/www/docs/repo/	docs.isutech.co.za	Astro Starlight static build + authelia (auth gate) + security-api (backing API) + a */10 * * * * auto-build cron	Not started. Three coupled pieces, not just a static site — plan for all three.
4	SANSA Frontend	/var/www/sansa-frontend/	sansa.isutech.co.za	PM2 (3003)	Backend confirmed running on Dube-migration audit; frontend unit found disabled — confirm with Nhlanhla whether that’s deliberate before migrating it.
5	SANSA Backend	/var/www/sansa-backend/	sansa.isutech.co.za/api	systemd (8003, bound 0.0.0.0 — fix to localhost on migration)	Largest DB of the whole estate.
6	SACE Frontend/Backend	/var/www/sace-frontend/, /var/www/sace-backend/	sace.isutech.co.za	systemd	Ownership genuinely unresolved — confirm whether this is a real standalone app or part of SANSA/AutoSlip before migrating.
7	AutoSlip	production systemd service; dev at /projects/active/autoslip/	— (WhatsApp-based)	systemd	Never assigned a migration wave; no known special risk, just not yet picked up.
8	Umdoni Municipality	/var/www/umdoni/	umdoni.gov.za	nginx + PHP	Live-vs-retired status flagged for confirmation 2026-07-30, never resolved.
9	MatchMind	/var/www/matchmind/	—	🔴 stopped	23,029 DB rows but no running process — orphan or intentionally retired, never confirmed. Primary deployment is Railway, not this box.

Already-migrated apps whose old Hetzner copy is still running as dead weight (safe to stop once each is confirmed stable for several days, not yet done): BuildQ, DealVault, ConformEdge, KopanoWorks (×2), Buka, PropertyData Engine, ProspectIQ (×4), QueueHere, Tumi. Decommissioning these is the actual “empty the server” step — don’t do it early for tidiness, do it last per app, after its Dube copy has proven itself.

api.isutech.co.za and monitor.isutech.co.za — nginx configs exist on Hetzner for both, but nothing is listening behind either (confirmed 2026-08-09). Not migration targets; nothing to move.

Development Projects: /projects/active/ (see Development Workflow)

---

Project Details

1. SANSA (Vision 2030 Educator Platform)

Frontend:

• Path: /var/www/sansa-frontend/
• Size: 712MB
• Type: Next.js application
• Port: 3003
• Process Manager: PM2
• Domain: https://sansa.isutech.co.za

Commands:

# Status
pm2 status sansa-frontend

# Restart
pm2 restart sansa-frontend

# Logs
pm2 logs sansa-frontend

# Manual start
cd /var/www/sansa-frontend
npm run build
pm2 start npm --name sansa-frontend -- start

Backend:

• Path: /var/www/sansa-backend/
• Size: 590MB
• Type: FastAPI (Python)
• Port: 8003
• Service: sansa-backend.service
• Domain: https://sansa.isutech.co.za/api

Commands:

# Status
systemctl status sansa-backend

# Restart
systemctl restart sansa-backend

# Logs
journalctl -u sansa-backend -f

# Manual operation
cd /var/www/sansa-backend
source venv/bin/activate
uvicorn app.main:app --host 0.0.0.0 --port 8003

---

2. Tumi Platform

Frontend:

• Path: /var/www/tumi-frontend/
• Size: 145MB
• Type: Static site
• Domain: https://tumi.isutech.co.za

Backend:

• Path: /var/www/tumi-backend/
• Size: 94MB
• Type: FastAPI (Python)
• Port: 8004
• Service: tumi-backend.service
• Domain: https://tumi.isutech.co.za/api

Commands:

# Backend status
systemctl status tumi-backend

# Backend restart
systemctl restart tumi-backend

# Backend logs
journalctl -u tumi-backend -f

# Frontend (static - no restart needed)
# Files served directly by nginx

---

3. SACE (Provider Intelligence)

Frontend:

• Path: /var/www/sace-frontend/
• Size: 564MB
• Type: Next.js application
• Service: sace-frontend.service
• Domain: https://sace.isutech.co.za

Backend:

• Path: /var/www/sace-backend/
• Size: 552MB
• Type: FastAPI (Python)
• Service: sace-backend.service
• Domain: https://sace.isutech.co.za/api

Commands:

# Status
systemctl status sace-backend sace-frontend

# Restart both
systemctl restart sace-backend sace-frontend

# Logs
journalctl -u sace-backend -f
journalctl -u sace-frontend -f

---

4. ThriveSend (B2B2G Platform)

• Path: /var/www/thrivesend/
• Size: 1.2GB
• Type: Next.js full-stack application
• Process Manager: PM2
• Nginx Config: /etc/nginx/sites-enabled/thrivesend

Commands:

# Status
pm2 status thrivesend

# Restart
pm2 restart thrivesend

# Logs
pm2 logs thrivesend

# View all ThriveSend processes
pm2 list | grep thrive

---

5. ProspectIQ (Lead Intelligence Platform)

• Path: /var/www/prospect-iq/
• Size: 3.6MB (code) + node_modules
• Stack: Python FastAPI + Next.js + Celery + PostgreSQL + Redis
• Nginx Config: /etc/nginx/sites-enabled/prospectiq

Services (5 total):

1. prospectiq-api.service - FastAPI Backend
2. prospectiq-frontend.service - Next.js Frontend
3. prospectiq-celery.service - Celery Worker
4. prospectiq-celery-beat.service - Celery Beat Scheduler
5. prospectiq-worker.service - Async Task Processor

Commands:

# Status all services
systemctl status prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Restart all
systemctl restart prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

# Logs
journalctl -u prospectiq-api -f
journalctl -u prospectiq-frontend -f
journalctl -u prospectiq-celery -f

---

6. DocsHub (Knowledge Hub — this site)

• Path: /var/www/docs/repo/
• Domain: https://docs.isutech.co.za (Authelia-gated for some sections, not blanket HTTP Basic Auth)
• Type: Astro Starlight static site (migrated from MkDocs 2026-04 — if you find MkDocs commands or a mkdocs.yml reference anywhere, it’s stale)
• Git Remote: git@github.com:gedeza/isutech-knowledge-hub.git
• Backing services, not just a static site:
  • authelia.service — auth portal gating some doc sections
  • security-api.service — “Security Status API for DocsHub”

Commands:

# Build
cd /var/www/docs/repo
git pull origin main
npm run build          # astro build — outputs to ./dist

# Preview locally
npm run preview

# Restart the backing services after a config change
systemctl restart authelia security-api

Publishing model: commit + push to main — not a manual deploy step. A cron job (*/10 * * * * /var/www/docs/scripts/update-docs.sh, not a systemd timer) pulls, builds, and atomically swaps the live site within ~10 minutes. That script (v3, 2026-07-19) already has failure-alerting (via an optional webhook in /var/www/docs/alert-webhook.conf), a disk-space guard, and health-marker files (.last-successful-check, .last-successful-deploy) — check those before assuming the pipeline is stuck vs. just between runs.

⚠️ Two other cron jobs (sync-prospectiq.sh on 0 5 * * *, sync-deals.sh on */15 * * * *) feed content into this site from ProspectIQ and DealVault’s data. Both apps’ real databases now live on Dube — confirm these sync scripts aren’t reading a stale local copy before trusting any ProspectIQ/DealVault-derived content on this site.

---

7. Umdoni Municipality Website

• Path: /var/www/umdoni/
• Size: 284MB
• Domain: https://umdoni.gov.za
• Type: PHP-based application
• Status: ✅ Serving

Commands:

# View nginx configuration
cat /etc/nginx/sites-available/umdoni.gov.za

# Check PHP-FPM status
systemctl status php*-fpm

# View logs
tail -f /var/log/nginx/umdoni-access.log

---

8. iSu Tech Corporate Website

Frontend:

• Path: /var/www/isutech-frontend/
• Size: 3.2MB
• Domain: https://isutech.co.za
• Type: Static site

Backend:

• Path: /var/www/isutech-backend/
• Size: 371MB
• Domain: https://api.isutech.co.za
• Process Manager: PM2
• PM2 Names: isutech-backend, isu-api

Commands:

# Backend status
pm2 status isutech-backend

# Backend restart
pm2 restart isutech-backend

# Backend logs
pm2 logs isutech-backend

---

9. AutoSlip (Receipt Automation)

• Production: Running as systemd service
• Development Code: /projects/active/autoslip/
• Service: autoslip.service
• Purpose: WhatsApp-based receipt processing

Commands:

# Status
systemctl status autoslip

# Restart
systemctl restart autoslip

# Logs
journalctl -u autoslip -f

---

10. Nhlanhla Portfolio

• Path: /var/www/nhlanhla-portfolio/
• Size: 30MB
• Domain: https://nhlanhla.isutech.co.za
• Type: Static site
• Status: Configured, serving status unknown

---

11. MatchMind (Sports Prediction)

• Path: /var/www/matchmind/
• Size: 568MB
• Status: 🔴 Stopped (code exists, no service running)
• Development: /projects/active/matchmind/
• Primary Deployment: Railway

---

Development Workflow

Development Directory: /projects/active/ (3.1GB)

Active Development Projects:

1. thrive-send-b2b2g (2.0GB)
2. isutech-prospect-iq (696MB)
3. umdoni-website (271MB)
4. mnyandu-portfolio_from_root (147MB)
5. matchmind (19MB)
6. iSuMonitor (7.6MB)
7. autoslip (1.7MB)
8. MyProfile (56KB)

Workflow:

Development → Testing → Production
/projects/active/[project] → test → /var/www/[project]

See: Development Workflow Guide

---

Infrastructure Services

Nginx (Web Server / Reverse Proxy)

Enabled Sites (13):

• api.isutech.co.za
• autoslip
• docs.isutech.co.za
• isutech.co.za
• monitor.isutech.co.za
• nhlanhla.isutech.co.za
• prospectiq
• sace.isutech.co.za
• sansa.isutech.co.za
• thrivesend
• tumi.isutech.co.za
• umdoni.gov.za
• (matchmind - configured but not serving)

Commands:

# Status
systemctl status nginx

# Test configuration
nginx -t

# Reload configuration (no downtime)
systemctl reload nginx

# Restart nginx
systemctl restart nginx

# View enabled sites
ls -la /etc/nginx/sites-enabled/

# View specific config
cat /etc/nginx/sites-available/docs.isutech.co.za

---

PostgreSQL 14

Status: ✅ Running

Commands:

# Status
systemctl status postgresql@14-main

# Connect to database
sudo -u postgres psql

# List databases
sudo -u postgres psql -c "\l"

# Backup database
sudo -u postgres pg_dump dbname > backup_$(date +%Y%m%d).sql

# Restore database
sudo -u postgres psql dbname < backup_20260201.sql

---

Redis

Status: ✅ Running

Commands:

# Status
systemctl status redis-server

# Connect to Redis CLI
redis-cli

# Check info
redis-cli INFO

# Inside redis-cli:
# ping              # Test connection
# keys *            # List all keys
# info              # Server info
# quit              # Exit

---

Docker & Containerd

Status: ✅ Running

Commands:

# Docker status
systemctl status docker

# Containerd status
systemctl status containerd

# List containers
docker ps

# List images
docker images

---

Directory Structure

This tree is Hetzner’s (/var/www/...) — still accurate there since the old app copies haven’t been removed yet. On Dube, the convention is /data/www/<app>/ for everything except a few apps that kept their original path (/opt/buka) — see the Project Inventory tables above for the exact path per app.

Gotcha — not all app dirs are owned by isutech-admin. /data and /data/www themselves are world-readable, so cd/ls always works and can mask the real problem. Four app directories are owned by their own service user/group instead: propertydata-engine (propertydata:propertydata), prospect-iq (prospectiq:prospectiq), queue-here and both tumi-* dirs (www-data:www-data). isutech-admin isn’t in those groups by default, so you can browse them but writes fail without sudo. Fix once per box: sudo usermod -aG propertydata,prospectiq,www-data isutech-admin, then log out/in (or newgrp <group>) for it to take effect — group changes don’t apply to an already-open session.

/var/www/
├── docs/                        # DocsHub (237MB)
│   └── repo/                    # Git repository + built site
├── sansa-frontend/              # SANSA Frontend (712MB)
├── sansa-backend/               # SANSA Backend (590MB)
├── tumi-frontend/               # Tumi Frontend (145MB)
├── tumi-backend/                # Tumi Backend (94MB)
├── sace-frontend/               # SACE Frontend (564MB)
├── sace-backend/                # SACE Backend (552MB)
├── thrivesend/                  # ThriveSend (1.2GB)
├── prospect-iq/                 # ProspectIQ (3.6MB + modules)
├── umdoni/                      # Umdoni Municipality (284MB)
├── isutech-frontend/            # iSu Tech Frontend (3.2MB)
├── isutech-backend/             # iSu Tech Backend (371MB)
├── isutech-web/                 # iSu Tech Web (6.5MB)
├── nhlanhla-portfolio/          # Portfolio (30MB)
├── matchmind/                   # MatchMind (568MB - stopped)
└── html/                        # Default nginx (unused)

/projects/active/                # Development (3.1GB)
├── thrive-send-b2b2g/           # ThriveSend dev
├── isutech-prospect-iq/         # ProspectIQ dev
├── umdoni-website/              # Umdoni dev
├── mnyandu-portfolio_from_root/ # Portfolio dev
├── matchmind/                   # MatchMind dev
├── autoslip/                    # AutoSlip dev
├── iSuMonitor/                  # Monitoring assets
└── MyProfile/                   # CV and profiles

/etc/nginx/sites-enabled/
├── api.isutech.co.za
├── autoslip
├── docs.isutech.co.za
├── isutech.co.za
├── monitor.isutech.co.za
├── nhlanhla.isutech.co.za
├── prospectiq
├── sace.isutech.co.za
├── sansa.isutech.co.za
├── thrivesend
├── tumi.isutech.co.za
└── umdoni.gov.za

---

Service Management Commands

Quick Reference

# View all running systemd services
systemctl list-units --type=service --state=running

# View all PM2 processes
pm2 list

# View nginx enabled sites
ls /etc/nginx/sites-enabled/

# Check what's listening on ports
ss -tlnp | grep LISTEN

# Check disk usage
df -h
du -sh /var/www/*

# Check memory usage
free -h

---

Restart All Application Services

# All systemd backend services
systemctl restart sansa-backend tumi-backend sace-backend sace-frontend \
  prospectiq-api prospectiq-frontend prospectiq-celery \
  prospectiq-celery-beat prospectiq-worker autoslip

# All PM2 frontend services
pm2 restart all

# Nginx
systemctl reload nginx

---

Restart by Application

# SANSA
systemctl restart sansa-backend
pm2 restart sansa-frontend

# Tumi
systemctl restart tumi-backend
# (Frontend is static, no restart needed)

# SACE
systemctl restart sace-backend sace-frontend

# ThriveSend
pm2 restart thrivesend

# ProspectIQ
systemctl restart prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

# AutoSlip
systemctl restart autoslip

# iSu Tech
pm2 restart isutech-backend
# (Frontend is static, no restart needed)

---

Deployment Workflows

General Deployment Pattern

# 1. Backup current version
cp -r /var/www/[project] /var/www/[project].backup.$(date +%Y%m%d)

# 2. Pull latest code (if using Git)
cd /var/www/[project]
git pull origin main

# 3. Install dependencies
# For Python:
source venv/bin/activate && pip install -r requirements.txt
# For Node:
npm install

# 4. Build if needed
# For Next.js:
npm run build
# For MkDocs:
mkdocs build

# 5. Restart service
# For systemd:
systemctl restart [service-name]
# For PM2:
pm2 restart [app-name]

# 6. Verify
curl http://localhost:[port]/health
journalctl -u [service] -n 20  # or pm2 logs [app]

---

Specific Application Deployments

SANSA:

# Backend
cd /var/www/sansa-backend
source venv/bin/activate
git pull && pip install -r requirements.txt
alembic upgrade head
systemctl restart sansa-backend

# Frontend
cd /var/www/sansa-frontend
git pull && npm install && npm run build
pm2 restart sansa-frontend

DocsHub:

cd /var/www/docs/repo
git pull origin main
mkdocs build
# No restart needed - nginx serves static files

ProspectIQ:

cd /var/www/prospect-iq
git pull origin main
pip install -r requirements.txt
npm install && npm run build
systemctl restart prospectiq-api prospectiq-frontend \
  prospectiq-celery prospectiq-celery-beat prospectiq-worker

---

Monitoring & Logs

View Service Logs

Systemd Services:

# Live logs (follow)
journalctl -u SERVICE_NAME -f

# Last 100 lines
journalctl -u SERVICE_NAME -n 100

# Today's logs
journalctl -u SERVICE_NAME --since today

# Error level only
journalctl -u SERVICE_NAME -p err

# Multiple services
journalctl -u sansa-backend -u tumi-backend -f

PM2 Applications:

# Live logs
pm2 logs APP_NAME

# Last 100 lines
pm2 logs APP_NAME --lines 100

# All PM2 apps
pm2 logs

# PM2 monitoring dashboard
pm2 monit

Nginx:

# Access logs
tail -f /var/log/nginx/access.log

# Error logs
tail -f /var/log/nginx/error.log

# Site-specific logs (if configured)
tail -f /var/log/nginx/sansa-access.log
tail -f /var/log/nginx/docs-access.log

---

Check Resource Usage

# Overall system
htop  # or top

# Disk usage
df -h
du -sh /var/www/*

# Memory usage
free -h

# CPU and memory per process
ps aux | grep python
ps aux | grep node

# Network connections
netstat -tulpn
ss -tulpn

# PM2 resource monitoring
pm2 monit

---

Known Issues & Notes — Hetzner (current, does not apply to isutech-svr-01)

Issue	Project	Status	Notes
Multiple restarts	isutech-backend (PM2)	⚠️ Monitor	Currently 23 restarts
Not running	MatchMind	🔴 Stopped	Code exists, no active service
Not running	isu-api (PM2)	🔴 Stopped	Check if needed
Development on prod	All projects	⚠️ Notice	Development happens directly on production server (Mac broken)

These are point-in-time operational notes about Hetzner’s current app estate — none of them are meaningful for isutech-svr-01, which is a fresh commission with no history yet.

---

Security Notes

production-server-01 (Hetzner, current)

• SSH Access: Key-based authentication only, root login via ssh root@46.224.40.5
• Firewall: UFW enabled (ports 80, 443, 22)
• SSL Certificates: Let’s Encrypt (auto-renewal via certbot.timer)
• Password Protection: DocsHub uses HTTP Basic Auth
• Git Authentication: SSH keys or Personal Access Tokens
• Database: PostgreSQL restricted to localhost
• Redis: Restricted to localhost

isutech-svr-01 (Dube TradePort, current standard — stronger baseline)

• SSH Access: Key-only auth, no root login at all — isutech-admin (sudo) on port 33822, PasswordAuthentication no enforced at the sshd level
• Firewall: UFW default-deny (only 22 internal/80/443 open) plus CrowdSec + crowdsec-firewall-bouncer (bans on SSH brute-force/probing patterns — not fail2ban, retired)
• File integrity: auditd + AIDE — watches system binaries, cron, sudoers, SSH keys; catches both successful and failed tamper attempts. Gap Hetzner still has open.
• SSL Certificates: not yet provisioned — no apps deployed yet
• Database/Cache: not yet provisioned — no apps deployed yet

Check Security (either server):

# Firewall status
ufw status

# SSL certificate expiry
certbot certificates

# Failed login attempts
journalctl _SYSTEMD_UNIT=ssh.service | grep "Failed"

# Open ports
ss -tlnp

On isutech-svr-01 specifically, also:

# CrowdSec active bans
cscli decisions list

# auditd status + recent events
auditctl -s
ausearch -k cron_change -ts recent   # example: any of the watched keys from the ruleset

---

Contacts & Resources

• Server Provider (primary, live): Dube TradePort / iConnect — Server IP: 41.78.233.110 (internal 10.2.38.10)
• Server Provider (legacy, being emptied): Hetzner Cloud — Server IP: 46.224.40.5
• SSL Provider: Let’s Encrypt / Certbot
• Git Repositories: github.com/gedeza
• Documentation: https://docs.isutech.co.za

---

Quick Emergency Commands

# Service down - restart
systemctl restart [service-name]

# Check why service failed
systemctl status [service-name]
journalctl -u [service-name] -n 50

# High memory usage
free -h
ps aux --sort=-%mem | head -10
systemctl restart [problematic-service]

# Disk full
df -h
du -sh /var/www/* /var/log/*
journalctl --vacuum-time=7d  # Clear old logs

# Nginx issues
nginx -t  # Test config
systemctl restart nginx
tail -f /var/log/nginx/error.log

# Database issues
systemctl status postgresql@14-main
sudo -u postgres psql -c "SELECT 1;"

---

Document Version: 3.0 — rewritten for the Dube TradePort cutover Last Updated: 2026-08-09 Previous Update: 2026-02-01 Author: iSu Technologies Operations Team