Security Incident Response Tools

Security Incident Response Tools

Last Updated: 2026-02-15 Incident Date: 2026-02-12 to 2026-02-14 Tools Location: /root/ on production server Status: Active & Operational

---

Overview

Following the security incident on February 12-14, 2026, a comprehensive suite of security command tools was developed to detect, investigate, and remediate threats. These tools are designed to work together as a complete incident response toolkit.

Incident Summary

Attack Vector: Exposed credentials in /tmp/login.json (admin@kopanoworks.co.za) Impact: Unauthorized access, log deletion (wtmp, lastlog, btmp) Response: Automated remediation tools deployed within 48 hours Current Status: ✅ Secured with ongoing monitoring

---

Security Tools Suite

1. Backdoor Detection Scanner

File: /root/backdoor-detection.sh Purpose: Comprehensive scan for persistence mechanisms and backdoors Run Frequency: Weekly or after suspicious activity

What It Detects

Check	Description	Severity
Cron Jobs	Suspicious scheduled tasks across all users	HIGH
Systemd Services	Recently modified service files (30-day window)	HIGH
SSH Keys	Unauthorized keys in all user directories	CRITICAL
Startup Scripts	Malicious /etc/rc.local or /etc/init.d entries	HIGH
Webshells	PHP/JSP/ASP shells in web directories	CRITICAL
Reverse Shells	Netcat, socat, perl, python shell processes	CRITICAL
Network Listeners	Unauthorized listening ports	MEDIUM
SUID Binaries	Privilege escalation via SUID/SGID files	HIGH
Hidden Files	Concealed files in /tmp and /var/tmp	MEDIUM
LD_PRELOAD Rootkits	Library injection attacks	CRITICAL
Modified Binaries	Tampered system binaries in /bin, /sbin	CRITICAL
PAM Backdoors	Authentication module tampering	CRITICAL

Usage

# Run full backdoor scan
bash /root/backdoor-detection.sh

# Save output to file for review
bash /root/backdoor-detection.sh > /root/backdoor-scan-$(date +%Y%m%d).txt

# Email results to security team
bash /root/backdoor-detection.sh | mail -s "Security Scan Results" security@isutech.co.za

When to Run

• ✅ Every Monday morning (scheduled maintenance)
• ✅ After deploying new code or services
• ✅ Following any security alert or suspicious activity
• ✅ Before major infrastructure updates
• ✅ After onboarding new team members with server access

---

2. Forensic Evidence Collector

File: /root/investigate-attack.sh Purpose: Comprehensive forensic data collection for incident analysis Run Frequency: Only during active incident response

Data Collected

1. System Information

   • OS version, uptime, hostname
   • Active kernel and loaded modules

2. Network Activity

   • All ESTABLISHED connections
   • Listening ports and services
   • Recent network traffic patterns

3. Authentication Logs

   • Last 100 successful SSH logins
   • Last 50 failed login attempts
   • User session history

4. User Accounts

   • All users with shell access
   • SSH authorized keys (root & sibonga)
   • Sudo privileges and configurations

5. Scheduled Tasks

   • All cron jobs for all users
   • Systemd timers
   • /etc/cron.d contents

6. File System Changes

   • Files modified in last 3 days
   • Focus on /root and /home directories

7. Web Access Patterns

   • Suspicious POST/PUT requests
   • Filtered by incident date range
   • Nginx access logs

8. Security Controls

   • fail2ban current status and bans
   • Firewall rules
   • Running processes (especially shells)

Usage

/root/security-investigation-20260214-071359.txt

# Run investigation (creates timestamped report)
bash /root/investigate-attack.sh

Output File Contains

• Section 1-5: User activity and authentication
• Section 6-7: System files and privileges
• Section 8-10: File changes and web activity
• Section 11-12: Security status and running processes

When to Run

• 🚨 Immediately upon detecting suspicious activity
• 🚨 When fail2ban bans spike unexpectedly
• 🚨 After receiving alerts from monitoring systems
• 🚨 Before contacting security consultants (provide evidence)
• 🚨 For monthly security audit trail documentation

---

3. Web Application Security Auditor

File: /root/webapp-security-audit.sh Purpose: Audit all running web applications for common vulnerabilities Run Frequency: Monthly or after application updates

Applications Monitored

Application	Technology	Ports	URL
KopanoWorks	Node.js + Next.js	3010, 3011	admin.kopanoworks.isutech.co.za
Prospectiq	Python + Next.js	3002, 8002	prospectiq.isutech.co.za
Sansa Backend	Python FastAPI	8003	Internal API
Tumi Backend	Python FastAPI	8004	Internal API
AutoSlip	Python Gunicorn	5000	Internal Service
SACE	Full-stack	3000, 8001	sace.isutech.co.za

Security Checks

HIGH PRIORITY:

• ✅ .env file permissions (must be 600, not 644)
• ✅ Exposed credentials in application logs
• ✅ Hardcoded secrets detection

MEDIUM PRIORITY:

• ⚠️ Missing rate limiting on API endpoints
• ⚠️ Missing CSRF protection on forms
• ⚠️ Missing authentication audit logging

LOW PRIORITY:

• ℹ️ Security headers (HSTS, X-Frame-Options, CSP)
• ℹ️ Dependency versions (outdated packages)
• ℹ️ API versioning implementation

Usage

# Run web application audit
bash /root/webapp-security-audit.sh

# Check specific application logs
bash /root/webapp-security-audit.sh | grep -A5 "KopanoWorks"

Recommendations Provided

The script automatically provides prioritized security recommendations:

1. Change exposed admin password (admin@kopanoworks.co.za)
2. Audit all .env files for exposed secrets
3. Enable rate limiting on all API endpoints
4. Add CSRF protection to all forms
5. Enable API request logging with IP tracking

When to Run

• ✅ Monthly security review (first Monday of month)
• ✅ After deploying new applications
• ✅ Before penetration testing
• ✅ After any credential rotation
• ✅ Following security incidents

---

4. Credential Rotation Guide

File: /root/change-compromised-passwords.sh Purpose: Identify exposed credentials and guide rotation process Run Frequency: After security incidents or quarterly reviews

Credentials Identified

The script scans and identifies:

1. Exposed Web Credentials

   • admin@kopanoworks.co.za (found in /tmp/login.json)
   • All web admin accounts

2. Database Credentials

   • PostgreSQL root password
   • All application database passwords in .env files
   • Redis passwords

3. Application Secrets

   • JWT secrets (7 applications)
   • API keys (AWS, Stripe, SendGrid, Twilio)
   • Session secrets

4. Infrastructure Credentials

   • Grafana admin password
   • Monitoring system credentials

Step-by-Step Instructions

The script provides detailed rotation steps for each credential type:

Example - JWT Secret Rotation:

# Generate new JWT secret
openssl rand -hex 32

# Update .env files
nano /var/www/kopanoworks/apps/api/.env
# Replace JWT_SECRET=old_value with new value

# Restart service
systemctl restart pm2-root

Example - Database Password Change:

# Change PostgreSQL password
sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'NEW_SECURE_PASSWORD';"

# Update all .env files that use this database
find /var/www -name ".env" -exec grep -l "DATABASE_URL" {} \;

# Restart affected services
bash /root/restart-services.sh

Usage

# View credential rotation guide
bash /root/change-compromised-passwords.sh

# Get specific section (JWT secrets only)
bash /root/change-compromised-passwords.sh | sed -n '/ROTATE JWT SECRETS/,/ROTATE API KEYS/p'

Helper Commands Included

# Generate strong password (32 chars)
openssl rand -base64 24

# Generate JWT secret
openssl rand -hex 32

# List all .env files
find /var/www /opt -name '.env' -type f

# Check PostgreSQL users
sudo -u postgres psql -c '\du'

---

5. Service Restart Manager

File: /root/restart-services.sh Purpose: Safely restart all services after configuration changes Run Frequency: After credential rotation or .env file updates

Services Managed

Service	Purpose	Dependencies
tumi-backend	Tumi API service	Database, Redis
prospectiq-api	Prospectiq backend	Database, Celery
autoslip	AutoSlip service	Database
sansa-frontend	Sansa UI service	Backend APIs
pm2-root	Node.js process manager	Multiple Node apps
nginx	Web server	All backend services

Features

• ✅ Graceful restarts - No dropped connections
• ✅ Error handling - Reports failed restarts
• ✅ Status verification - Checks service status after restart
• ✅ Visual feedback - Clear success/failure indicators

Usage

# Restart all services
bash /root/restart-services.sh

# Check if services are running after restart
systemctl status tumi-backend prospectiq-api autoslip sansa-frontend pm2-root nginx

# Restart specific service only
systemctl restart tumi-backend

When to Run

• ✅ After updating .env files
• ✅ After rotating database passwords
• ✅ After changing JWT secrets
• ✅ After modifying service configurations
• ✅ During scheduled maintenance windows

⚠️ WARNING: Restarting pm2-root will temporarily interrupt all Node.js applications. Schedule during low-traffic periods.

---

6. Password Hash Generator

File: /root/change-admin-password.py Purpose: Generate bcrypt hashes for new admin passwords Run Frequency: One-time use during password changes

Usage

# Generate password hash
python3 /root/change-admin-password.py

# Output:
# New password: 7hPGJuV4H8AD/euQttRD2VsRxTsDGX4f
# Bcrypt hash: $2b$10$...

Integration with Database

-- Update admin password in database
UPDATE users
SET password = '$2b$10$...'
WHERE email = 'admin@kopanoworks.co.za';

⚠️ SECURITY NOTE: This file contains a hardcoded password. After use:

# Securely delete the file
shred -u /root/change-admin-password.py

---

Automated Monitoring

Security Monitor Daemon

File: /usr/local/bin/security-monitor Schedule: Every 5 minutes via cron Log File: /var/log/security-alerts.log

What It Monitors

• SSH authorized_keys changes
• /etc/sudoers modifications
• Critical system binary changes
• Suspicious running processes
• New SUID/SGID files
• Unexpected network listeners

Viewing Alerts

# View recent alerts
tail -50 /var/log/security-alerts.log

# Filter for specific alert type
grep "ALERT" /var/log/security-alerts.log | tail -20

# Check last security scan timestamp
tail -1 /var/log/security-alerts.log

Current Known False Positives

The following processes are flagged but are legitimate:

• AutoSlip gunicorn workers (4 processes)
• Docker runc init processes (2 processes)

Recommendation: Update security monitor to exclude these:

nano /usr/local/bin/security-monitor
# Add exclusions for: gunicorn, runc init

---

Complete Incident Response Workflow

Phase 1: Detection (0-15 minutes)

1. Identify Threat

   # Check fail2ban for unusual bans
   fail2ban-client status sshd
   
   # Review recent SSH logins
   grep "Accepted\|Failed" /var/log/auth.log | tail -50
   
   # Check security monitoring alerts
   tail -100 /var/log/security-alerts.log

2. Initial Assessment

   # Run quick backdoor scan
   bash /root/backdoor-detection.sh | grep -E "ALERT|WARNING"
   
   # Check for suspicious processes
   ps aux | grep -E "(nc|netcat|socat)" | grep -v grep

Phase 2: Investigation (15-60 minutes)

3. Collect Evidence

   /root/security-investigation-YYYYMMDD-HHMMSS.txt

   # Run full forensic collection
   bash /root/investigate-attack.sh
   
   

4. Analyze Attack Vector

   # Check web application security
   bash /root/webapp-security-audit.sh
   
   # Review exposed credentials
   bash /root/change-compromised-passwords.sh

5. Document Timeline

   • Note first detection time
   • Identify attack window
   • List affected systems/accounts
   • Document attacker actions

Phase 3: Containment (1-4 hours)

6. Isolate Affected Systems (if necessary)

   # Block attacker IP immediately
   fail2ban-client set sshd banip <ATTACKER_IP>
   
   # Or use UFW
   ufw deny from <ATTACKER_IP>

7. Disable Compromised Accounts

   # Lock user account
   usermod -L <compromised_user>
   
   # Remove SSH keys
   > /home/<compromised_user>/.ssh/authorized_keys

8. Verify No Backdoors

   # Run comprehensive backdoor scan
   bash /root/backdoor-detection.sh > /root/backdoor-scan-$(date +%Y%m%d).txt
   
   # Review results carefully
   less /root/backdoor-scan-$(date +%Y%m%d).txt

Phase 4: Eradication (4-8 hours)

9. Rotate All Credentials

   # Follow rotation guide
   bash /root/change-compromised-passwords.sh
   
   # Generate new database passwords
   openssl rand -base64 24
   
   # Generate new JWT secrets
   openssl rand -hex 32

10. Update All .env Files

    # Find all .env files
    find /var/www /opt -name ".env" -type f
    
    # Update each with new credentials
    # Ensure permissions are 600
    chmod 600 /var/www/*/. env

11. Restart Services

    # Apply new configurations
    bash /root/restart-services.sh
    
    # Verify all services running
    systemctl list-units --state=running | grep -E "tumi|prospectiq|sansa|autoslip"

Phase 5: Recovery (8-24 hours)

12. Verify System Integrity

    # Check all services are operational
    curl -I https://kopanoworks.isutech.co.za
    curl -I https://prospectiq.isutech.co.za
    curl -I https://sace.isutech.co.za
    
    # Verify database connectivity
    sudo -u postgres psql -c "SELECT version();"

13. Monitor for Reinfection

    # Watch authentication logs
    tail -f /var/log/auth.log
    
    # Monitor security alerts
    tail -f /var/log/security-alerts.log
    
    # Watch fail2ban activity
    tail -f /var/log/fail2ban.log

Phase 6: Post-Incident (24-72 hours)

14. Document Incident

    • Create incident report
    • Include timeline
    • List remediation actions
    • Note lessons learned

15. Improve Defenses

    # Install additional monitoring (if needed)
    apt install aide rkhunter
    
    # Initialize file integrity monitoring
    aide --init
    mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

16. Team Briefing

    • Share incident details with team
    • Update security procedures
    • Schedule security training
    • Review access controls

---

Quick Reference Commands

Daily Security Checks

# Morning security status check
fail2ban-client status sshd                # Check banned IPs
tail -20 /var/log/security-alerts.log     # Review overnight alerts
grep "Accepted" /var/log/auth.log | tail -10  # Recent logins

Weekly Security Tasks

# Monday morning routine
bash /root/backdoor-detection.sh > /root/backdoor-scan-$(date +%Y%m%d).txt
bash /root/webapp-security-audit.sh
tail -100 /var/log/fail2ban.log

Monthly Security Review

# First Monday of month
bash /root/investigate-attack.sh  # Full forensic report
bash /root/webapp-security-audit.sh  # Application audit
# Review and rotate credentials as needed

Emergency Response

# Immediate threat response
fail2ban-client set sshd banip <ATTACKER_IP>  # Block attacker
bash /root/investigate-attack.sh  # Collect evidence
bash /root/backdoor-detection.sh  # Scan for persistence

Security Dashboard API

# Check API status
systemctl status security-api

# View API logs
journalctl -u security-api -f

# Test API health
curl http://127.0.0.1:5001/api/health

# Get live security status
curl http://127.0.0.1:5001/api/security-status | python3 -m json.tool

# Restart API (if needed)
systemctl restart security-api

# View nginx API logs
tail -f /var/log/nginx/docs-access.log | grep "/api/"

Browser Console Commands

// Clear dashboard dismissal (for testing)
localStorage.removeItem('isutech_security_dashboard_dismissed');
location.reload();

// Check dismissal status
localStorage.getItem('isutech_security_dashboard_dismissed');

// Manually show dashboard (if dismissed)
localStorage.removeItem('isutech_security_dashboard_dismissed');
window.location.reload();

---

Security Metrics

Current Protection Status (as of 2026-02-15)

Metric	Value	Status
fail2ban Bans	189 IPs	✅ Active
.env Permissions	All 600	✅ Secured
Database Passwords	Rotated	✅ Updated
JWT Secrets	Rotated (7 apps)	✅ Updated
SSH Keys	8 root, 3 sibonga	✅ Verified
Backdoors Detected	0	✅ Clean
Monitoring	Every 5 min	✅ Active

Security Tools Usage Statistics

# View when tools were last run
ls -lth /root/security-investigation-*.txt 2>/dev/null | head -5
ls -lth /root/backdoor-scan-*.txt 2>/dev/null | head -5

---

Troubleshooting

Issue: Script Fails to Run

Symptom: Permission denied or command not found

Solution:

# Ensure scripts are executable
chmod +x /root/backdoor-detection.sh
chmod +x /root/investigate-attack.sh
chmod +x /root/webapp-security-audit.sh
chmod +x /root/change-compromised-passwords.sh
chmod +x /root/restart-services.sh

# Verify they exist
ls -lh /root/*.sh

Issue: Service Restart Fails

Symptom: Service fails to start after restart

Solution:

# Check service status
systemctl status <service-name>

# View service logs
journalctl -u <service-name> -n 50

# Check configuration syntax
# For nginx:
nginx -t

# For Python services:
python3 -m py_compile /var/www/<app>/app.py

Issue: False Positive Alerts

Symptom: Security monitor constantly alerts on legitimate processes

Solution:

# Edit security monitor script
nano /usr/local/bin/security-monitor

# Add exclusions to process check
ps aux | grep -E "(nc|netcat|socat)" \
  | grep -v "gunicorn" \
  | grep -v "runc init" \
  | grep -v grep

---

Additional Resources

Related Documentation

• fail2ban Configuration Guide
• SSH Hardening Guide
• Firewall Configuration
• Kernel Hardening

External References

• OWASP Top 10
• CIS Benchmarks
• NIST Cybersecurity Framework

Security Contacts

Internal: security@isutech.co.za Emergency: Contact on-call engineer via PagerDuty Vendor Support: Hetzner support for infrastructure issues

---

Maintenance Schedule

Task	Frequency	Responsibility
Run backdoor scan	Weekly (Monday)	DevOps Team
Review security alerts	Daily	DevOps Team
Application security audit	Monthly	Security Lead
Credential rotation	Quarterly	Security Lead
Full security review	Quarterly	Management + Security Lead
Penetration testing	Annually	External Consultant

---

Changelog

Date	Version	Changes
2026-02-15	1.0	Initial documentation created
2026-02-14	0.9	Security tools deployed
2026-02-14	0.5	Incident remediation completed
2026-02-12	0.0	Security incident occurred

---

Document Status: ✅ Active Next Review: 2026-03-15 Owner: iSu Technologies DevOps Team