Mtail gets stuck
The MtailNotProcessingLogs alert is firing quite often in the latest release.
Here's what happens in the mtail log:
May 28 14:42:36 latitanza mtail[1088]: I0528 14:42:36.230137 1088 store.go:107] Running Store.Expire()
May 28 15:42:36 latitanza mtail[1088]: I0528 15:42:36.230168 1088 store.go:107] Running Store.Expire()
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.230129 1088 store.go:107] Running Store.Expire()
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.230204 1088 tail.go:255] read /dev/fd/3: file already closed
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.230532 1088 log_watcher.go:249] No abspath in watched list, added new one for /etc/mtail/postfix.mtail
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.232411 1088 checker.go:217] declaration of capture group reference `hostname' at postfix.mtail:19:3-170 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.232471 1088 checker.go:217] declaration of capture group reference `date' at postfix.mtail:19:3-170 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.232499 1088 checker.go:217] declaration of capture group reference `date' at postfix.mtail:19:3-170 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.232523 1088 checker.go:217] declaration of capture group reference `hostname' at postfix.mtail:19:3-170 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.233552 1088 checker.go:217] declaration of capture group reference `score' at postfix.mtail:152:3-62 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.233617 1088 checker.go:217] declaration of capture group reference `score' at postfix.mtail:152:3-62 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.233726 1088 checker.go:217] declaration of capture group reference `score' at postfix.mtail:155:3-64 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.233768 1088 checker.go:217] declaration of capture group reference `score' at postfix.mtail:155:3-64 appears to be unused
May 28 16:42:36 latitanza mtail[1088]: I0528 16:42:36.235017 1088 loader.go:229] Loaded program postfix.mtail
May 28 17:42:36 latitanza mtail[1088]: I0528 17:42:36.230105 1088 store.go:107] Running Store.Expire()
no Ansible push was running at the time, configuration files are unchanged, yet it seems something triggered a reload, and the tail loop is likely hung after that "file already closed" error...