Data Flow

Overview

Centralize Management System memiliki 4 alur data utama:

1. Application Logs → OpenSearch
2. Security Alerts → OpenSearch
3. Audit Trail → OpenSearch
4. Service Metrics → Dashboard

1. Application Logs (logs-*)

Docker Containers ──▶ Fluent Bit ──▶ Fluentd ──▶ OpenSearch (logs-YYYY.MM.DD)
                         │
                         ├── /var/lib/docker/containers/*/*.log
                         └── /var/log/*.log

Sumber: Docker container logs, system logs Index Pattern: logs-YYYY.MM.DD Retention: Sesuai kebijakan

Fluent Bit Collection

Fluent Bit dikonfigurasi untuk mengumpulkan:

• Docker container logs dari /var/lib/docker/containers/
• System logs dari /var/log/
• Application-specific logs via custom parsers

Fluentd Forwarding

Fluentd menerima logs dari Fluent Bit dan melakukan:

• Enrichment dengan metadata (tenant_id, app_slug)
• Routing ke OpenSearch berdasarkan index pattern
• Multi-tenant isolation via alias

2. Security Alerts (wazuh-alerts-*)

Servers/Endpoints ──▶ Wazuh Agent ──▶ Wazuh Manager ──▶ alerts.json ──▶ Fluentd ──▶ OpenSearch
                           │               │                                    │
                           │               ├── Rule Engine                      │
                           │               ├── Correlation                      ▼
                           │               └── Active Response         wazuh-alerts-YYYY.MM.DD
                           │
                           └── Collects:
                               ├── auth.log
                               ├── secure
                               ├── syslog
                               └── Custom logs

Sumber: Wazuh agents pada monitored servers Index Pattern: wazuh-alerts-YYYY.MM.DD Retention: 30 hari

Wazuh Agent Collection

Wazuh agent dikonfigurasi untuk mengumpulkan:

• auth.log - Authentication logs
• secure - Secure logs
• syslog - System logs
• Custom logs - Application-specific logs

Wazuh Manager Processing

Wazuh Manager melakukan:

• Rule Engine - Evaluasi rules terhadap incoming logs
• Correlation - Korelasi events lintas agent
• Active Response - Otomatis response terhadap threats

3. Audit Trail (audit-log-*)

CLM Backend ──▶ OpenSearch (audit-log-YYYY.MM.DD)
     │
     └── Logs:
         ├── User login/logout
         ├── Create/Update/Delete operations
         ├── Configuration changes
         └── API access

Sumber: CLM Backend application Index Pattern: audit-log-YYYY.MM.DD Retention: 90 hari

Audit Events

Event	Description
login	User login attempt
logout	User logout
search	Log search operation
create	Create operation
update	Update operation
delete	Delete operation
export	Data export

4. Service Dashboard (dashboard-services)

CLM Backend ──▶ OpenSearch (dashboard-services)
     │
     └── Metrics:
         ├── Service status
         ├── Request count
         ├── Error rate
         ├── Latency
         └── Log count

Sumber: CLM Backend (aggregated metrics) Index: dashboard-services Update Frequency: Real-time / Periodic

Data Pipeline Components

Fluent Bit → Fluentd → OpenSearch

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Fluent Bit  │────▶│   Fluentd    │────▶│  OpenSearch  │
│  (Collector) │     │  (Forwarder) │     │  (Storage)   │
│              │     │              │     │              │
│ • Docker logs│     │ • Enrichment │     │ • Indexing    │
│ • System logs│     │ • Routing    │     │ • Searching   │
│ • Parsers    │     │ • Filtering  │     │ • Aggregation │
└──────────────┘     └──────────────┘     └──────────────┘

Wazuh Agent → Wazuh Manager → OpenSearch

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Wazuh Agent │────▶│ Wazuh Manager│────▶│  OpenSearch  │
│  (per host)  │     │  (Processing)│     │  (Storage)   │
│              │     │              │     │              │
│ • Log collect│     │ • Rule eval  │     │ • Indexing    │
│ • FIM        │     │ • Correlation│     │ • Searching   │
│ • Rootcheck  │     │ • Alert gen  │     │ • Dashboard   │
└──────────────┘     └──────────────┘     └──────────────┘

PostgreSQL → Logstash → OpenSearch

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  PostgreSQL  │────▶│   Logstash   │────▶│  OpenSearch  │
│  (Source DB) │     │   (Sync)     │     │  (Storage)   │
│              │     │              │     │              │
│ • RBAC data  │     │ • JDBC input │     │ • Metrics    │
│ • Audit trail│     │ • Transform  │     │ • Dashboard   │
│ • Tenants    │     │ • Output     │     │ • Alerting   │
└──────────────┘     └──────────────┘     └──────────────┘

Message Bus Flow

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Backend    │────▶│  RabbitMQ    │────▶│   Consumer   │
│  (Producer)  │     │  (Queue)     │     │  (Consumer)  │
│              │     │              │     │              │
│ • Alert gen  │     │ • Queuing    │     │ • Processing │
│ • Event emit │     │ • Routing    │     │ • Indexing   │
└──────────────┘     └──────────────┘     └──────────────┘

Telegram Notification Flow

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Backend    │────▶│     n8n      │────▶│  Telegram    │
│  (Webhook)   │     │  (Workflow)  │     │  (Bot API)   │
│              │     │              │     │              │
│ • Alert gen  │     │ • Verify     │     │ • Broadcast  │
│ • Chat IDs   │     │ • Format     │     │ • Send msg   │
└──────────────┘     └──────────────┘     └──────────────┘

Next Steps

• Network Topology - Topologi jaringan
• Docker Compose - Konfigurasi services
• Data Pipeline - Detail pipeline components