Skip to main content

Fluent Bit

Overview​

Fluent Bit adalah lightweight log collector yang mengumpulkan logs dari Docker containers dan system logs.

Configuration​

Docker Compose​

fluent-bit:
  image: clm-fluent-bit:latest
  build:
    context: ./configs/fluent-bit
    dockerfile: Dockerfile
  container_name: fluent-bit
  volumes:
    - ./configs/fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    - ./configs/fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro
    - ./configs/fluent-bit/scripts:/fluent-bit/scripts:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
    - /var/log:/var/log:ro
    - /var/run/docker.sock:/var/run/docker.sock:ro
  mem_limit: 128m
  cpus: "0.25"

fluent-bit.conf​

[SERVICE]
    Flush         5
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Path          /var/lib/docker/containers/*/*.log
    Parser        docker
    Tag           docker.*
    Mem_Buf_Limit 5MB

[INPUT]
    Name          tail
    Path          /var/log/*.log
    Parser        syslog
    Tag           system.*
    Mem_Buf_Limit 5MB

[FILTER]
    Name          record_modifier
    Match         docker.*
    Record        source_type docker
    Record        app_slug clm-backend

[FILTER]
    Name          record_modifier
    Match         system.*
    Record        source_type system
    Record        app_slug clm-system

[OUTPUT]
    Name          forward
    Match         *
    Host          fluentd
    Port          24224

parsers.conf​

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER]
    Name        syslog
    Format      regex
    Regex       ^(?<time>[^ ]+ [^ ]+) (?<ident>[a-zA-Z0-9_\/\.]+)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
    Time_Key    time
    Time_Format %b %d %H:%M:%S

Collecting Docker Logs​

Automatic Container Discovery​

#!/bin/bash
# scripts/generate-container-map.sh

# Generate container map for Fluent Bit
docker ps --format '{{.Names}}' | while read container; do
    log_path=$(docker inspect --format='{{.LogPath}}' "$container")
    echo "$container: $log_path"
done

Manual Configuration​

[INPUT]
    Name          tail
    Path          /var/lib/docker/containers/docker-container-id/*.log
    Parser        docker
    Tag           docker.clm-backend

Collecting System Logs​

[INPUT]
    Name          tail
    Path          /var/log/messages
    Parser        syslog
    Tag           system.messages

[INPUT]
    Name          tail
    Path          /var/log/secure
    Parser        syslog
    Tag           system.secure

[INPUT]
    Name          tail
    Path          /var/log/auth.log
    Parser        syslog
    Tag           system.auth

Monitoring​

Health Check​

curl http://localhost:2020/api/v1/health

Metrics​

curl http://localhost:2020/api/v1/metrics

Troubleshooting​

Check Logs​

docker logs fluent-bit --tail 100

Test Parser​

fluent-bit -c /fluent-bit/etc/fluent-bit.conf -p parsers_file=/fluent-bit/etc/parsers.conf -T

Next Steps​