Client Onboarding Guide

Last Updated: 2026-01-07

Overview

This guide covers the process of adding new clients to the PLGT Observability Stack. The onboarding process sets up website monitoring, metrics collection, activity logging, and custom dashboards.

Onboarding Checklist

Prerequisites

Before starting, collect the following information:

Item	Example
Client ID	`new-client`
Client Name	`New Client Company`
Domain	`client.com`
Contact Email	`admin@client.com`
Hosting Provider	Afrihost, Hetzner, AWS
Server Access	SSH or cPanel
Database Type	MySQL, PostgreSQL

Step 1: Configure Prometheus Targets

Add HTTP probes for website monitoring in prometheus/prometheus.yml:

scrape_configs:
  - job_name: 'blackbox-http-clients'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://client.com
        labels:
          client: new-client
          client_name: 'New Client'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

For Metrics Endpoints

If the client has a custom metrics endpoint:

  - job_name: 'new-client-metrics'
    scheme: https
    metrics_path: /api/metrics.php
    params:
      token: ['SECRET_TOKEN']
    static_configs:
      - targets: ['client.com']
        labels:
          client: new-client
          client_name: 'New Client'

Step 2: Deploy Client Scripts

PHP Metrics Endpoint

Deploy metrics.php to the client’s server to expose application metrics:

<?php
/**
 * Prometheus Metrics Endpoint
 * Deploy to: /public/api/metrics.php
 */

// Security token
define('METRICS_TOKEN', 'your_secret_token');

// Validate token
if (($_GET['token'] ?? '') !== METRICS_TOKEN) {
    http_response_code(403);
    die("# Access denied\n");
}

header('Content-Type: text/plain; charset=utf-8');

// Database connectivity
try {
    $pdo = new PDO($dsn, $user, $pass);
    echo "app_database_up 1\n";
} catch (Exception $e) {
    echo "app_database_up 0\n";
}

// Application metrics
echo "app_users_total " . getUserCount($pdo) . "\n";
echo "app_sessions_active " . getActiveSessions($pdo) . "\n";

// Disk usage
$free = disk_free_space('/');
$total = disk_total_space('/');
echo "app_disk_usage_percent " . round((($total - $free) / $total) * 100, 2) . "\n";
echo "app_disk_free_bytes " . $free . "\n";

Activity Log Shipper

Deploy loki-shipper.php to ship activity logs to Loki:

<?php
/**
 * Loki Log Shipper
 * Deploy to: /scripts/loki-shipper.php
 * Cron: */15 * * * * php /scripts/loki-shipper.php
 */

define('CLIENT_ID', 'new-client');
define('CLIENT_NAME', 'New Client');
define('LOKI_URL', 'https://monitor.isutech.co.za/loki/api/v1/push');
define('BATCH_SIZE', 100);
define('STATE_FILE', __DIR__ . '/.loki-shipper-state');

// Database configuration
$dbConfig = [
    'host' => getenv('DB_HOST') ?: 'localhost',
    'name' => getenv('DB_NAME') ?: 'database',
    'user' => getenv('DB_USER') ?: 'username',
    'pass' => getenv('DB_PASS') ?: 'password'
];

// Log table configuration
$logTableConfig = [
    'table' => 'activity_logs',
    'id_column' => 'id',
    'user_column' => 'user_id',
    'action_column' => 'action',
    'data_column' => 'data',
    'ip_column' => 'ip_address',
    'time_column' => 'created_at'
];

// ... (full implementation in templates/client-onboarding/loki-shipper.php)

Step 3: Create Loki Alert Rules

Create alert rules in loki/rules/<client>/activity-alerts.yml:

groups:
  - name: new-client-activity
    rules:
      - alert: NewClientFailedLoginSpike
        expr: |
          sum(count_over_time({client="new-client"}
            |= "failed" |~ "login|Login" [10m])) > 5
        for: 0m
        labels:
          severity: warning
          client: new-client
          category: security
        annotations:
          summary: "Multiple failed logins on New Client"
          description: "{{ $value }} failed login attempts in the last 10 minutes"

      - alert: NewClientBruteForceDetected
        expr: |
          sum(count_over_time({client="new-client"}
            |= "failed" |~ "login|Login" [5m])) > 10
        for: 0m
        labels:
          severity: critical
          client: new-client
          category: security
        annotations:
          summary: "Possible brute force attack on New Client"
          description: "{{ $value }} failed login attempts in 5 minutes"

      - alert: NewClientNoActivity
        expr: |
          absent(count_over_time({client="new-client"}[1h]))
        for: 0m
        labels:
          severity: warning
          client: new-client
        annotations:
          summary: "No activity logs from New Client"
          description: "No logs received in the last hour - check log shipper"

Step 4: Configure Alert Routing

Add client-specific routes in alertmanager/alertmanager.yml:

route:
  routes:
    # New Client alerts
    - match:
        client: new-client
      receiver: 'new-client-alerts'
      group_wait: 30s
      repeat_interval: 2h
      routes:
        - match:
            severity: critical
          receiver: 'new-client-critical'
          group_wait: 10s
          repeat_interval: 30m

receivers:
  - name: 'new-client-alerts'
    email_configs:
      - to: 'admin@client.com'
        send_resolved: true
        headers:
          Subject: '[New Client] {{ template "email.subject" . }}'
    slack_configs:
      - channel: '#alerts'
        send_resolved: true

  - name: 'new-client-critical'
    email_configs:
      - to: 'admin@client.com'
        send_resolved: true
    slack_configs:
      - channel: '#critical'
    telegram_configs:
      - bot_token: 'YOUR_BOT_TOKEN'
        chat_id: YOUR_CHAT_ID

Step 5: Create Grafana Dashboard

Dashboard Template

Create a client dashboard in grafana/dashboards/<client>.json:

Key Panels:

Panel	Purpose	Data Source
Website Status	Up/Down indicator	Prometheus
Response Time	HTTP response latency	Prometheus
SSL Expiry	Days until certificate expires	Prometheus
Activity Timeline	Log volume over time	Loki
Login Activity	Successful/failed logins	Loki
Error Events	Application errors	Loki
User Activity	Actions by user	Loki

Sample Panel Configuration

{
  "title": "Website Status",
  "type": "stat",
  "targets": [{
    "expr": "probe_success{instance=\"https://client.com\"}",
    "legendFormat": "Status"
  }],
  "options": {
    "colorMode": "background",
    "graphMode": "none",
    "orientation": "auto",
    "reduceOptions": {
      "calcs": ["lastNotNull"],
      "fields": "",
      "values": false
    }
  },
  "mappings": [{
    "type": "value",
    "options": {
      "1": {"text": "ONLINE", "color": "green"},
      "0": {"text": "OFFLINE", "color": "red"}
    }
  }]
}

Step 6: Configure Cron Jobs

On the client server, set up scheduled tasks:

# Activity log shipping (every 15 minutes)
*/15 * * * * php /home/user/scripts/loki-shipper.php >> /home/user/logs/loki.log 2>&1

# Optional: Health ping (every 5 minutes)
*/5 * * * * curl -s https://client.com/api/health > /dev/null

Step 7: Test and Verify

Test Website Monitoring

# From PLGT server
curl -s "http://localhost:9115/probe?target=https://client.com&module=http_2xx" | grep probe_success

Test Log Shipping

# On client server
php /scripts/loki-shipper.php

# Verify in Grafana Explore
{client="new-client"}

Test Alerts

Temporarily add a test alert or simulate a failure to verify the notification chain.

Step 8: Client Access

Create Grafana User

Login to Grafana as admin
Administration → Users → New User
Set appropriate permissions (Viewer role recommended)
Send credentials to client

Provide Documentation

Send the client:

Dashboard URL
Login credentials
Basic usage guide
Support contact information

Quick Setup Script

Use the automated setup script for faster onboarding:

# Copy template
cp -r templates/client-onboarding clients/new-client

# Edit configuration
vim clients/new-client/config.yaml

# Run setup
./clients/new-client/setup.sh

The script handles:

Prometheus target generation
Loki rule creation
Dashboard template customization
Alertmanager route configuration

Maintenance

Regular Checks

Weekly: Verify log shipping is running
Monthly: Check for alert noise/tune thresholds
Quarterly: Review dashboard relevance
Annually: Audit client access

Offboarding

When removing a client:

Remove Prometheus targets
Delete Loki alert rules
Archive/delete Grafana dashboard
Remove Alertmanager routes
Revoke Grafana access
Remove cron jobs on client server

Next: Query Reference