Matrix, Zabbix and Bots
Have you already heared about Matrix? The new chat protocol all the cool kids are using. All jokes aside, the cool thing about Matrix is that it allows bridging different chat protocols together. I always like to take the example of the chat room of the study association (Scintilla) I volunteer at. The same chat room is availble over IRC, Telegram, Slack and Matrix. I believe there were some efforts to bridge Discord into as well. Benefit of this approach is that people can decide for themselves which client they prefer to use, rather than being forced to use Telegram, for example. If you would like to try out Matrix, at SNT (Studenten Net Twente) we have set up a public homeserver. There you can register for free and try it out. The homeserver can be found here.
If you are a system administrator you probably have heard of Zabbix. A monitoring system. At the time of writing I manage four Zabbix installs to monitor two different study associations, a measurement project and my personal systems. If Zabbix is set up correctly, the trigger notifications are very useful. The default templates are very quick to notify you of things. For example, if you have hosts that are frequently busy, but it is no problem that they are busy, Zabbix will tell you everytime the host is a bit busy. The Linux Journal has a nice post about how to value triggers/notifications. Essentially, it boils down to this; critical triggers are triggers you want to be woken up at 3 A.M for, since they mean everything is on fire. Warnings triggers are triggers which will develop into high or critical if left unchecked, but at the moment they are still “OK”. These ideas should create an environment where Zabbix only
complains notifies you of things worth notifying you about. While we are working towards such a situation, at the study associations currently it is not quite the case. Often we see triggers going off (host A idle time below X), only to turn green a minute later.
To combine Zabbix with Matrix I wrote two pieces of code. The first is the matrix_alert.py. This allows Zabbix to report triggers to a configured Matrix room. It makes use of the matrix-python-sdk which is quite easy to use😁
The second part is a Zabbix bot. This was made with the python-matrix-bot-api (also easy to use). The main function of the bot is to hang around in rooms and respond to “!zabbix” commands. Without any argument it responds with all the unacknowledged triggers from Zabbix. However, if instructed it can also list acknowledged triggers. This is usefull, since normally you would need to open the webpage to see those triggers, now you can simply read them from your chat client. Another usefull feature of the bot is the ability to acknowledge triggers from within the chat client. Acknowledged triggers are not repeated every X (defaults to an hour) time.
All the code for the Zabbix alerter and the bot can be found over at my Github page.