Data Pipeline Overview
Pipeline Architecture
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Data Sources│────▶│ Collectors │────▶│ Processors │────▶ OpenSearch
└──────────────┘ └──────────────┘ └──────────────┘
Components
| Component | Technology | Function |
|---|---|---|
| Fluent Bit | Fluent Bit 4.2.5 | Log collector (Docker & system logs) |
| Fluentd | Fluentd v1.17-1 | Log forwarding & enrichment |
| Logstash | Logstash (custom) | PostgreSQL → OpenSearch sync |
| Apache Camel | Camel (custom) | Integration routing |
| RabbitMQ | RabbitMQ 3 | Message bus |
| Consumer | Go service | RabbitMQ message consumer |
Data Flows
1. Application Logs
Docker Containers ──▶ Fluent Bit ──▶ Fluentd ──▶ OpenSearch (logs-YYYY.MM.DD)
2. Security Alerts
Servers ──▶ Wazuh Agent ──▶ Wazuh Manager ──▶ Fluentd ──▶ OpenSearch (wazuh-alerts-YYYY.MM.DD)
3. Audit Trail
CLM Backend ──▶ OpenSearch (audit-log-YYYY.MM.DD)
4. PostgreSQL Metrics
PostgreSQL ──▶ Logstash ──▶ OpenSearch (postgresql-metrics)
Next Steps
- Fluent Bit - Log collector
- Fluentd - Log forwarder
- Logstash - DB sync
- Apache Camel - Integration
- RabbitMQ - Message bus