主页 / server / 问题

问题[filebeat](server)

Martin Hope
Peter
Asked: 2021-01-26 05:44:15 +0800 CST
  • 0

我正在学习使用 ELK 并拥有一台作为测试客户端运行的 debian PC。每 30 秒它会记录一条消息：

021-01-18T08:29:59.656-0500#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":4096}}}},"cpu":{"system":{"ticks":171310,"time":{"ms":11}},"total":{"ticks":433770,"time":{"ms":24},"value":433770},"user":{"ticks":262460,"time":{"ms":13}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":13},"info":{"ephemeral_id":"e83bbdd5-2482-4ac4-85b9-5b50f2b64e7c","uptime":{"ms":444690082}},"memstats":{"gc_next":20543200,"memory_alloc":12804128,"memory_total":20951973208},"runtime":{"goroutines":54}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":2,"batches":2,"total":2},"read":{"bytes":681},"write":{"bytes":4714}},"pipeline":{"clients":3,"events":{"active":0,"published":2,"total":2},"queue":{"acked":2}}},"registrar":{"states":{"current":15,"update":2},"writes":{"success":2,"total":2}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}

我不想每 30 秒就有一次，所以我创建了一个 filebeat.yml 在 filebeat 中过滤掉它

在我写的路径部分下

exclude_lines :['^INFO#011[monitoring]#011log/log.go:145#011Non-zero']

没有效果，该行应该如何？

我有几个附带问题，

  • 如果不是每 30 秒记录一次，那对我来说没问题，这个状态可以每 5 分钟记录一次吗？
  • 好吧，我是 ELK 的新手，我认为这条线不应该在 filebeat 中（因为它应该记录系统的内存统计信息，或者我读错了吗？）另一方面，知道这些统计信息也不错，但是为什么它在 Gui 中没有很好地显示
elk filebeat
Martin Hope
vane
Asked: 2020-01-31 12:38:28 +0800 CST
  • 0

我正在尝试让 filebeat 使用 kafka 输入来使用来自 kafka 的消息。由于某种原因，我无法使用 SASL 进行身份验证，我不确定这是为什么。尝试将 Kafka 和 Filebeat 与 SASL 一起使用时，它的文档有点缺乏。

我的filebeat配置如下：

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
- type: kafka
  hosts: 'the.kafka.server.com:9092'
  topics: 'my_topic'
  group_id: 'my_group'
  ssl.enabled: yes
  username: "$ConnectionString"
  password: "org.apache.kafka.common.security.plain.PlainLoginModule required username='my_username' password='my_password';"

processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~

output.console:
  pretty: true

输出显示

INFO    input/input.go:114      Starting input of type: kafka; ID: 14409252276502564738
INFO    kafka/log.go:53 kafka message: Initializing new client
INFO    kafka/log.go:53 client/metadata fetching metadata for all topics from broker the.kafka.server.com:9092

INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
INFO    cfgfile/reload.go:171   Config reloader started
INFO    cfgfile/reload.go:226   Loading of config files completed.
INFO    kafka/log.go:53 kafka message: Successful SASL handshake. Available mechanisms: %!(EXTRA []string=[PLAIN OAUTHBEARER])
INFO    kafka/log.go:53 Failed to read response while authenticating with SASL to broker the.kafka.server.com:9092: EOF

INFO    kafka/log.go:53 Closed connection to broker the.kafka.server.com:9092

INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: EOF

我不确定这里发生了什么。我还尝试添加compression: none没有帮助的内容，并使用 openssl 验证服务器证书可以被验证。我在这里做错了什么？有问题的 kafka 服务器是云托管的 kafka 服务器，我看不到服务器配置，我从 kafka 的云 UI 中获得了“连接字符串”。

kafka filebeat