Skip to main content

Fluentd

Overview​

Fluentd adalah log forwarder yang menerima logs dari Fluent Bit dan mengirimkannya ke OpenSearch.

Configuration​

Docker Compose​

fluentd:
  image: clm-fluentd:latest
  build:
    context: ./configs/fluentd
  container_name: fluentd
  user: root
  ports:
    - "24224:24224"
    - "24224:24224/udp"
    - "9880:9880"
  environment:
    DEFAULT_TENANT: bjb-syariah
    DEFAULT_APP_SLUG: clm-backend
  volumes:
    - ./configs/fluentd/fluent.conf:/fluentd/etc/fluent.conf
    - wazuh-logs:/var/ossec/logs:ro
  mem_limit: 512m
  cpus: "0.25"

fluent.conf​

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter docker.**>
  @type record_transformer
  <record>
    tenant_id "#{ENV['DEFAULT_TENANT']}"
    app_slug "#{ENV['DEFAULT_APP_SLUG']}"
    source_type "docker"
  </record>
</filter>

<filter system.**>
  @type record_transformer
  <record>
    tenant_id "#{ENV['DEFAULT_TENANT']}"
    app_slug "#{ENV['DEFAULT_APP_SLUG']}"
    source_type "system"
  </record>
</filter>

<match docker.**>
  @type opensearch
  host opensearch
  port 9200
  scheme https
  user admin
  password MyStrong@OpenSearch2026!
  ssl_verify false
  
  index_name logs-${tag_parts[1]}
  include_tag_key true
  
  <buffer>
    @type file
    path /fluentd/log/buffer
    flush_mode interval
    flush_interval 5s
    retry_type exponential_backoff
    retry_max_interval 30s
    chunk_limit_size 2M
    queue_limit_length 32
  </buffer>
</match>

<match wazuh.**>
  @type opensearch
  host opensearch
  port 9200
  scheme https
  user admin
  password MyStrong@OpenSearch2026!
  ssl_verify false
  
  index_name wazuh-alerts-${Time.now.strftime('%Y.%m.%d')}
  include_tag_key true
  
  <buffer>
    @type file
    path /fluentd/log/buffer-wazuh
    flush_mode interval
    flush_interval 5s
    retry_type exponential_backoff
    retry_max_interval 30s
    chunk_limit_size 2M
    queue_limit_length 32
  </buffer>
</match>

Data Enrichment​

Adding Metadata​

<filter docker.**>
  @type record_transformer
  enable_ruby true
  <record>
    tenant_id "#{ENV['DEFAULT_TENANT']}"
    app_slug "#{ENV['DEFAULT_APP_SLUG']}"
    source_type "docker"
    hostname "#{Socket.gethostname}"
    timestamp "${Time.now.utc.iso8601}"
  </record>
</filter>

Parsing JSON Logs​

<filter docker.**>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

Output to OpenSearch​

Index Rotation​

<match docker.**>
  @type opensearch
  index_name logs-${Time.now.strftime('%Y.%m.%d')}
</match>

Buffer Configuration​

<buffer>
  @type file
  path /fluentd/log/buffer
  flush_mode interval
  flush_interval 5s
  retry_type exponential_backoff
  retry_max_interval 30s
  chunk_limit_size 2M
  queue_limit_length 32
</buffer>

Monitoring​

Health Check​

curl http://localhost:9880/plugins

Metrics​

curl http://localhost:9880/metrics

Troubleshooting​

Check Logs​

docker logs fluentd --tail 100

Test Configuration​

fluentd -c /fluentd/etc/fluent.conf --dry-run

Next Steps​