您好,歡迎來到網暖!
?
當前位置:網暖 » 站長資訊 » 建站基礎 » 網絡技術 » 文章詳細 訂閱RssFeed

Logstash基礎操作-Filter

來源:網絡整理 瀏覽:166次 時間:2019-12-06

Grok配置案例:

##啟動文件配置:#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{}}filter?{grok?{match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]???}}output?{??stdout{????codec?=>?"rubydebug"??}}##輸出文件內容172.16.213.132?[07/Feb/2018:16:24:19?+0800]?"GET?/?HTTP/1.1"?403?5039##顯示內容{??????"@version"?=>?"1",????"@timestamp"?=>?2019-11-10T06:02:42.865Z,??????????"host"?=>?"localhost.localdomain",???????"message"?=>?"172.16.213.132?[07/Feb/2018:16:24:19?+0800]?\"GET?/?HTTP/1.1\"?403?5039",?????"timestamp"?=>?"07/Feb/2018:16:24:19?+0800",?????????"bytes"?=>?"5039",??????"response"?=>?"403",??????"clientip"?=>?"172.16.213.132",??????"referrer"?=>?"\"GET?/?HTTP/1.1\""}

Grok 過濾重復字段

##?配置文件#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{?}}filter?{??grok?{??match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\???%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]??remove_field?=>?["message"]???}}output?{??stdout{??codec?=>?"rubydebug"??}}

Grok搭配Date時間插件配置

#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{??}}filter?{grok?{?match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\??%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]?remove_field?=>?["message"]???}date?{??match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]??}}output?{??stdout{??codec?=>?"rubydebug"??}}

Date 過濾重復得字段配置

#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{??}}filter?{?grok?{???match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\????%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]???remove_field?=>?["message"]???}date?{??match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]????}mutate?{???remove_field?=>?[?"timestamp"?]????}}output?{?stdout{??codec?=>?"rubydebug"??}}

綜合練習配置參數

#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{??}}filter?{??grok?{???match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\????%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]???remove_field?=>?["message"]??}?date?{??match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]???}?mutate{????rename?=>?{"response"?=>?"response_new"}????gsub?=>?["referrer",?"\"",?""]????remove_field?=>?[?"timestamp"?]????split?=>?["clientip",?"."]??}}output?{?stdout{??codec?=>?"rubydebug"??}}

Geoip 地理位置插件操作方式

#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{??}}filter?{????grok?{?????match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\??????%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]?????remove_field?=>?["message"]???}???date?{????match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]???}???mutate{??????remove_field?=>?[?"timestamp"?]??}??geoip?{????source?=>?"clientip"????database?=>?"/usr/local/include/GeoLite2-ASN_20191105/GeoLite2-ASN.mmdb"???}}output?{??stdout{????codec?=>?"rubydebug"??}?}

Geoip輸出指定屬性值

#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{??}}filter?{????grok?{?????match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\??????%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"]?????remove_field?=>?["message"]???}???date?{????match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]??}???mutate{??????remove_field?=>?[?"timestamp"?]??}geoip?{source?=>?"clientip"#database?=>?"/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb"database?=>?"/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb"fields?=>?["city_name",?"region_name",?"country_name",?"ip",?"latitude",?"longitude",?"timezone"]???}}output?{??stdout{????codec?=>?"rubydebug"??}}模擬數據:36.7.152.182?[07/Feb/2018:16:24:19?+0800]?"GET?/?HTTP/1.1"?403?5039

綜合實戰

#?Sample?Logstash?configuration?for?creating?a?simple#?Beats?->?Logstash?->?Elasticsearch?pipeline.input?{??stdin{}}filter{grok{??match?=>?{"message"?=>?"%{TIMESTAMP_ISO8601:localtime}\|\~\|%{IP:clientip}??\|\~\|%{GREEDYDATA:http_user_agent}\|\~\|%{GREEDYDATA:url}??\|\~\|%{GREEDYDATA:mediaid}\|\~\|%{GREEDYDATA:osid}"}??remove_field?=>?[?"message"?]???}date?{????match?=>?["localtime",?"yyyy-MM-dd'T'HH:mm:ssZZ"]????target?=>?"@timestamp"???}mutate?{??????remove_field?=>?["localtime"]???}geoip?{?source?=>?"clientip"?#database?=>?"/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb"?database?=>?"/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb"?fields?=>?["city_name",?"region_name",?"country_name",?"ip",?"latitude",?"longitude",?"timezone"]??}}output?{???stdout?{???codec?=>?"rubydebug"???}}示例:2018-02-09T10:57:42+08:00|~|123.87.240.97|~|Mozilla/5.0(iPhone;CPU?iPhone?OS?11_2_2?like?Mac?OS?X)AppleWebKit/604.4.7?Version/11.0?Mobile/15C202?Safari/604.1|~|http://m.sina.cn/cm/ads_ck_wap.html|~|12434785489009|~|DF45566587855P



推薦站點

  • 騰訊騰訊

    騰訊網(www.QQ.com)是中國瀏覽量最大的中文門戶網站,是騰訊公司推出的集新聞信息、互動社區、娛樂產品和基礎服務為一體的大型綜合門戶網站。騰訊網服務于全球華人用戶,致力成為最具傳播力和互動性,權威、主流、時尚的互聯網媒體平臺。通過強大的實時新聞和全面深入的信息資訊服務,為中國數以億計的互聯網用戶提供富有創意的網上新生活。

    www.qq.com
  • 搜狐搜狐

    搜狐網是全球最大的中文門戶網站,為用戶提供24小時不間斷的最新資訊,及搜索、郵件等網絡服務。內容包括全球熱點事件、突發新聞、時事評論、熱播影視劇、體育賽事、行業動態、生活服務信息,以及論壇、博客、微博、我的搜狐等互動空間。

    www.sohu.com
  • 網易網易

    網易是中國領先的互聯網技術公司,為用戶提供免費郵箱、游戲、搜索引擎服務,開設新聞、娛樂、體育等30多個內容頻道,及博客、視頻、論壇等互動交流,網聚人的力量。

    www.163.com
  • 新浪新浪

    新浪網為全球用戶24小時提供全面及時的中文資訊,內容覆蓋國內外突發新聞事件、體壇賽事、娛樂時尚、產業資訊、實用信息等,設有新聞、體育、娛樂、財經、科技、房產、汽車等30多個內容頻道,同時開設博客、視頻、論壇等自由互動交流空間。

    www.sina.com.cn
  • 百度一下百度一下

    百度一下,你就知道

    www.baidu.com
?
最牛一尾中特规律