当事件没有发生时，管理员如何概括警报？

通常，我的用户要求我同样负责知道事件是否尚未发生。

我总是不得不使用 cron'ed shell 脚本和大量的日期边缘案例测试来构建自定义和脆弱的解决方案。

集中式日志记录应该允许一种更好、更易于维护的方式来掌握过去 N 小时内没有发生的事情。诸如logstash 通知和nagios 警报之类的东西。

更新

toppledwagon 的回答非常有帮助。o O（灯泡。）我现在有十几个批处理作业正在检查新鲜度。我想让他彻底的回答公正，并跟进我是如何实施他的想法的。

我将 jenkins 配置为发出系统日志，logstash 捕获它们并通过 nsca 将状态更新发送到 nagios。我还使用 check_mk 来保持一切干燥并在 nagios 中组织。

Logstash 过滤器

:::ruby
filter {
  if [type] == "syslog" {
    grok {
      match => [ "message", '%{SYSLOGBASE} job="%{DATA:job}"(?: repo="%{DATA:repo}")?$',
                 "message", "%{SYSLOGLINE}" ]
      break_on_match => true
    }
    date { match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ] }
  }
}

神奇之处在于 grok 的 match 参数中的那对模式以及 break_on_match => true。Logstash 将依次尝试每个模式，直到其中一个匹配。

Logstash 输出

我们使用 logstash nagios_nsca 输出插件让 icinga 知道我们在 syslog 中看到了 jenkins 作业。

:::ruby
output {
  if [type] == "syslog"
    and [program] == "jenkins"
    and [job] == "Install on Cluster"
    and "_grokparsefailure" not in [tags] {
      nagios_nsca {
        host => "icinga.example.com"
        port => 5667
        send_nsca_config => "/etc/send_nsca.cfg"
        message_format => "%{job} %{repo}"
        nagios_host => "jenkins"
        nagios_service => "deployed %{repo}"
        nagios_status => "2"
      }
   } # if type=syslog, program=jenkins, job="Install on Cluster"
} # output

冰糖 (nagios)

最后，我们通过 nsca 到达了 icinga (nagios)。现在，我们将需要为我们想要注意的每项工作定义的被动服务检查没有按时发生。这可能是很多工作，所以让我们使用check_mk将 Python 工作列表转换为 nagios 对象定义。

check_mk那样很酷。

/etc/check_mk/conf.d/freshness.mk

# check_mk requires local variables be prefixed with '_'

_dailies = [ 'newyork' ]
_day_stale = 86400 * 1.5

_weeklies = [ 'atlanta', 'denver', ]
_week_stale = 86400 * 8

_monthlies = [ 'stlouis' ]
_month_stale = 86400 * 32

_service_opts = [
    ("active_checks_enabled", "0"),
    ("passive_checks_enabled", "1"),
    ("check_freshness", "1"),
    ("notification_period", "workhours"),
    ("contacts", "root"),
    ("check_period", "workhours"),
]

# Define a new command 'check-periodically' that sets the service to UKNOWN.
# This is called after _week_stale seconds have passed since the service last checked in.

extra_nagios_conf += """
  define command {
    command_name check-periodicaly
    command_line $USER1$/check_dummy 3 $ARG1$
  }

  """
# Loop through all passive checks and assign the new check-period command to them.

for _repo in _dailies + _weeklies + _monthlies:
    _service_name = 'deployed %s' % _repo
    legacy_checks += [(('check-periodicaly', _service_name, False), ['lead'])]


# Look before you leap - python needs the list defined before appending to it.
# We can't assume it already exists because it may be defined earlier.

if "freshness_threshold" not in extra_service_conf:
    extra_service_conf["freshness_threshold"] = []

# Some check_mk wizardry to set when the check has passed its expiration date.
# Results in (659200, ALL_HOSTS, [ 'atlanta', 'denver' ]) for weeklies, etc.

extra_service_conf["freshness_threshold"] += [
    (_day_stale,   ALL_HOSTS, ["deployed %s"   % _x for _x in _dailies]  ),
    (_week_stale,  ALL_HOSTS, ["deployed %s"  % _x for _x in _weeklies] ),
    (_month_stale, ALL_HOSTS, ["deployed %s" % _x for _x in _monthlies] ),
]

# Now we assign all the other nagios directives listed in _service_opts

for _k,_v in _service_opts:
    if _k not in extra_service_conf:
        extra_service_conf[_k] =  []

    extra_service_conf[_k] += [(_v, ALL_HOSTS, ["deployed "]) ]