Logstash 配置基础

云计算 2023-01-04 大白菜程序猿 手机阅读

1. Logstash 的基本原理

3.1 标准出入到标准输出

1
2
3
4
5
6

input {
    stdin {}
}
output {
    stdout { codec => rubydebug }
}

3.2 文件到 ES

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

input {
    file {
        path => [ "/data/nginx/logs/nginx_access.log" ]
        start_position => "beginning"
        ignore_older => 0
    }
}
filter {
    1. 使用 grok 插件对日志内容进行格式化
    grok {
        match => {
            "message" => "%{COMBINEDAPACHELOG}"
        }
    }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "nginx-access"
    }
}

3.3 Filebeats 到 ES

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

input { 
    beats { 
        port => 8088 
    }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "beats"
    }
}

3.4 Kafka 到 ES

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

input {
  kafka {
    bootstrap_servers => "${KAFKA_URL:localhost:2187}"
    topics => "${KAFKA_TOPIC:kafka-topic-test}"
    group_id => "${KAFKA_GROUP_ID:logstash-es}"
    consumer_threads => "${KAFKA_CONSUMER_THREADS:6}"
  }
}
filter {
  if ![message] {
    drop { }
  }
  mutate {
    remove_field => [ "headers", "result"]
  }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "kafka"
    }
}