skywatch/REALTIME_GUIDE.md

Real-Time Updates & Observability - Quick Reference

🔔 Real-Time Notifications

User Experience:

• Toast notifications appear in top-right corner when:
  • Incident is detected (resource down)
  • Incident is resolved (resource back up)
  • Critical alerts are triggered
• Auto-dismiss after 5 seconds
• Multiple notifications stack vertically

WebSocket Message Types:

// Dashboard update
{ type: 'dashboard', data: { resources: [...], groups: [...] } }

// Alert
{ type: 'alert', data: { resourceName: '...', message: '...' } }

// Incident
{ type: 'incident', data: { type: 'started'|'resolved', resourceName: '...' } }

// Metrics
{ type: 'metrics', data: { uptime: ..., apiStats: [...], errorCount: ... } }

---

📊 Observability Dashboard

Access:

Navigation Bar → Observability or /observability

Tab 1: API Metrics

Shows performance insights:

• Server Uptime: How long the server has been running
• Total API Requests: Total requests since startup
• Errors Logged: Total errors logged
• Unique Errors: Number of distinct error types

Endpoint Performance Table:

Column	Meaning
Endpoint	HTTP method + path
Requests	Total requests to this endpoint
Success Rate %	Percentage successful (200-399 status)
Avg Response	Average response time in milliseconds
Min/Max	Best and worst response times

Tab 2: Errors

Lists recent application errors:

• Error Message: What went wrong
• Context: Where it happened (endpoint, IP)
• Occurred At: Timestamp of error

Tab 3: Audit Logs

Complete change history with filtering:

Summary Section:

• Shows events from last 7 days
• Breakdown by action type (create, update, delete)

Filters:

• Entity Type: resource, settings, group, maintenance_window
• Entity ID: ID of the changed entity
• User ID: Who made the change (usually "system" for automated)
• Action: create, update, delete

Audit Log Columns:

Column	Meaning
Action	What happened (create/update/delete)
Entity	Type and ID of affected resource
User	Who made the change
Changes	What changed (shown as JSON snippet)
Time	When it happened

---

📈 API Endpoints for Integration

Get Metrics

GET /api/observability/metrics

Response:

{
  "uptime": 3600,
  "uptimeFormatted": "1h 0m",
  "apiStats": [
    {
      "endpoint": "/api/resources",
      "method": "GET",
      "totalRequests": 150,
      "successRate": "99.33",
      "avgResponseTime": 45,
      "minResponseTime": 12,
      "maxResponseTime": 234
    }
  ],
  "errorCount": 2,
  "uniqueErrors": 1
}

Get Recent Errors

GET /api/observability/errors?limit=50

Get Audit Logs

GET /api/observability/audit-logs?entityType=resource&userId=system&limit=100&offset=0

Get Audit Summary

GET /api/observability/audit-summary?days=7

---

🔍 Audit Log Examples

Resource Creation

{
  "action": "create",
  "entity_type": "resource",
  "entity_id": "42",
  "changes": {
    "name": "API Server",
    "url": "https://api.example.com",
    "type": "http",
    "enabled": true
  },
  "user_id": "system",
  "created_at": "2024-01-14T10:30:00Z"
}

Resource Update

{
  "action": "update",
  "entity_type": "resource",
  "entity_id": "42",
  "changes": {
    "name": "API Server",
    "enabled": false,
    "maintenance_mode": true
  },
  "user_id": "system",
  "created_at": "2024-01-14T10:35:00Z"
}

Settings Change

{
  "action": "update",
  "entity_type": "settings",
  "entity_id": "email_enabled",
  "changes": {
    "old": false,
    "new": true
  },
  "user_id": "system",
  "created_at": "2024-01-14T10:40:00Z"
}

---

💡 Use Cases

1. Monitoring API Performance

• Check GET /api/observability/metrics regularly
• Set up alerts if success rate drops below threshold
• Track response time trends over time

2. Debugging Issues

• Visit Errors tab in Observability dashboard
• See what went wrong with full stack traces
• Check context (which endpoint, from where)

3. Compliance Auditing

• Use Audit Logs tab to see who changed what
• Filter by date range and user
• Export logs for compliance reports

4. Root Cause Analysis

• Check audit log for recent changes
• Correlate with error logs
• Verify metrics before/after changes

5. Performance Optimization

• Identify slowest endpoints from metrics
• Check which endpoints have lowest success rate
• Focus optimization efforts on high-traffic endpoints

---

⚙️ Configuration

Metrics Collection

• Automatically enabled, no configuration needed
• Tracks all API endpoints
• Data available via /api/observability/metrics

Audit Logging

• Automatically enabled, no configuration needed
• Logs all resource CRUD operations
• Logs all settings changes
• Stores permanently in database

Error Tracking

• Automatically enabled, no configuration needed
• Captures all exceptions
• Stores in error_logs table

---

🚀 Performance Notes

• Metrics: In-memory collection, resets on server restart
• Audit Logs: Database storage, grows indefinitely (plan archive strategy)
• Error Logs: Database storage, grows indefinitely (plan archive strategy)
• WebSocket: All connected clients receive broadcast messages

---

📋 Troubleshooting

No notifications appearing?

• Check browser console for WebSocket errors
• Verify WebSocket connection is established
• Check that incidents are actually being created

Audit logs not showing changes?

• Verify you're filtering correctly
• Check user_id (usually "system" for API operations)
• Ensure entity type matches what you're looking for

Metrics showing as 0?

• Server was just restarted (metrics reset)
• No API calls have been made yet
• Check that endpoint path matches exactly

Performance dashboard slow?

• Too many audit logs in database
• Consider archiving old logs
• Limit query results with pagination