ELK5.5nginx访问日志grok切割(filebeat)

监控nginx访问日志filebeat+redis+logstashfilebeat收集日志后传给redis，logstash读取redis后grok后存储

安装filebeat

[root@linuxea.com-Node117 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-x86_64.rpm
[root@linuxea.com-Node117 ~]# yum install filebeat-5.5.1-x86_64.rpm -y

传递给redis

配置文件如下

[root@linuxea.com-Node117 /etc/filebeat]# cat filebeat.yml 
filebeat.prospectors:
 - input_type: log
   paths:
    - /data/logs/access_nginx.log
   document_type: nginx-access117
output.redis:
  hosts: ["10.10.0.98"]
  password: "OTdmOWI4ZTM4NTY1M2M4OTZh"
  key: "default_list"
  db: 1
  timeout: 5
  keys:
    - key: "%{[type]}"
      mapping:
      "nginx-access117": "nginx-access117"

启动程序

[root@linuxea.com-Node117 /etc/filebeat]# systemctl restart filebeat
[root@linuxea.com-Node117 /etc/filebeat]# tail -f /var/log/filebeat/filebeat
2017-08-25T20:53:09+08:00 INFO States Loaded from registrar: 11
2017-08-25T20:53:09+08:00 INFO Loading Prospectors: 1
2017-08-25T20:53:09+08:00 INFO Prospector with previous states loaded: 1
2017-08-25T20:53:09+08:00 WARN DEPRECATED: document_type is deprecated. Use fields instead.
2017-08-25T20:53:09+08:00 INFO Starting prospector of type: log; id: 12123466383741208858 
2017-08-25T20:53:09+08:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-08-25T20:53:09+08:00 INFO Metrics logging every 30s
2017-08-25T20:53:09+08:00 INFO Starting Registrar
2017-08-25T20:53:09+08:00 INFO Start sending events to output
2017-08-25T20:53:09+08:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-08-25T20:53:29+08:00 INFO Harvester started for file: /data/logs/access_nginx.log
2017-08-25T20:53:39+08:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.publisher.published_events=243 libbeat.redis.publish.read_bytes=1367 libbeat.redis.publish.write_bytes=126046 publish.events=245 registrar.states.current=11 registrar.states.update=245 registrar.writes=2
2017-08-25T20:54:09+08:00 INFO No non-zero metrics in the last 30s

redis查看

但启动后，写入access_nginx.log日志后就会写到redis,这个时候如果没有被拿走是可以看到的，如下

[root@linuxea.com-Node98 ~]# redis-cli -h 10.10.0.98 -a OTdmOWI4ZTM4NTY1M2M4OTZh
10.10.0.98:6379> select 1
OK
10.10.0.98:6379[1]> keys *
1) "nginx-access117"
10.10.0.98:6379[1]> type "nginx-access117"
list
10.10.0.98:6379[1]> lrange nginx-access117 0 -1
  1) "{"@timestamp":"2017-08-25T12:53:29.279Z","beat":{"hostname":"linuxea.com-Node117.cluster.com","name":"linuxea.com-Node117.cluster.com","version":"5.5.1"},"input_type":"log","message":"10.10.0.96 - - [25/Aug/2017:12:53:21 +0000] GET / HTTP/1.1 - 304 0 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36 -","offset":48321607,"source":"/data/logs/access_nginx.log","type":"nginx-access117"}"

创建patterns目录和文件

现看下具体格式有哪些：nginx日志格式如下:

log_format upstream2  '$proxy_add_x_forwarded_for $remote_user [$time_local] "$request" $http_host'
        '[$body_bytes_sent] $request_body "$http_referer" "$http_user_agent" [$ssl_protocol] [$ssl_cipher]'
        '[$request_time] [$status] [$upstream_status] [$upstream_response_time] [$upstream_addr]';

这个是在logstash机器上创建patterns.d目录存放grok格式

[root@linuxea.com-Node49 /etc/logstash/conf.d]# mkdir /etc/logstash/patterns.d/ -p

把patterns写到文件

[root@linuxea.com-Node49 /etc/logstash/conf.d]# cat /etc/logstash/patterns.d/nginx 
NGUSERNAME [a-zA-Z.@-+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IP:clent_ip} (?:-|%{USER:ident}) [%{HTTPDATE:log_date}] "%{WORD:http_verb} (?:%{PATH:baseurl}?%{NOTSPACE:params}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})" (%{IPORHOST:url_domain}|%{URIHOST:ur_domain}|-)[(%{BASE16FLOAT:request_time}|-)] %{NOTSPACE:request_body} %{QS:referrer_rul} %{GREEDYDATA:User_Agent} [%{GREEDYDATA:ssl_protocol}] [(?:%{GREEDYDATA:ssl_cipher}|-)][%{NUMBER:time_duration}] [%{NUMBER:http_status_code}] [(%{BASE10NUM:upstream_status}|-)] [(%{NUMBER:upstream_response_time}|-)] [(%{URIHOST:upstream_addr}|-)]

但是在安装完成kibana后，在dev tools中有grok debugger,如果日志格式不同，增减后直接simulate测试即可，如下图：写进elasticsearch配置如下其中用了GeoLite2-City.mmdb，但是发现没有什么卵用下载地址：https://dev.maxmind.com/zh-hans/geoip/geoip2/geolite2-%E5%BC%80%E6%BA%90%E6%95%B0%E6%8D%AE%E5%BA%93/或者就用自带的，注释database即可

[root@linuxea-Node49 /etc/logstash/conf.d]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[root@linuxea.com-Node49 /etc/logstash]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%   
-> Installed ingest-user-agent
[root@linuxea.com-Node49 /etc/logstash]# 

input从redis取数据发送给elasticsearch

[root@linuxea.com-Node49 /etc/logstash/conf.d]# cat /etc/logstash/conf.d/redis_input.conf 
input { 
    redis {
        host => "10.10.0.98"
        port => "6379"
        key => "nginx-access117"
        data_type => "list"
        password => "OTdmOWI4ZTM4NTY1M2M4OTZh"
        threads => 10
        db => "1"
    }
}
filter {
   if [type] == "nginx-access-117" {
    grok {
        patterns_dir => [ "/etc/logstash/patterns.d" ]
        match => { "message" => "%{NGINXACCESS}" }
        overwrite => [ "message" ]
        }
    geoip {
        source => "clent_ip"
        target => "geoip"
#        database => "/etc/logstash/GeoLiteCity.dat"
        database => "/etc/logstash/GeoLite2-City.mmdb"
         }
    useragent {
        source => "User_Agent"
        target => "userAgent"
        }
    urldecode {
        all_fields => true
        }
     mutate {
            gsub => ["User_Agent","["]",""]        #将user_agent中的 " 换成空
            convert => [ "response","integer" ]
            convert => [ "body_bytes_sent","integer" ]
            convert => [ "bytes_sent","integer" ]
            convert => [ "upstream_response_time","float" ]
            convert => [ "upstream_status","integer" ]
            convert => [ "request_time","float" ]
            convert => [ "port","integer" ]
       }
    date {
    match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        }
        }
    }
output {
    if [type] == "nginx-access117" {
    elasticsearch {
        hosts => ["10.0.1.49:9200"]
        index => "logstash-nginx-access-117-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "linuxea"
    }
    }
    stdout {codec => rubydebug}
}

最后几步

在启动logstash时候可以观察下日志：打开kibana，在management-->create即可，输入logstash-nginx-access-117-*,如下图：当日志写入，字段会grok，在kibana上表现这样ok,基本上日志切割完成