You should have monitors in place to detect problems in your enterprise. These can be individual monitors defined for an agent, or queries/thresholds defined for data collected by an observability platform. Either way, at some point, you need to notify someone about what went wrong.
The following is an email notification we set up for a customer:
The important things to note are:
- What failed? The "Tivoli CTH Health Check" failed in PROD.
- What needs to be done? Run all of the checks that are listed at the end of the email.
While this amount of actionable information is just normal to some number of people, many organizations simply don't have this kind of information-rich notification configured. The part I like the best is the "run book", basically the "What needs to be done" part. This could have a lot more detail, but it is sufficient for the known target audience of this email. The additional details (like in a run book) would be the exact steps needed to perform the checks, along with maybe a video showing what it should normally look like.
No comments:
Post a Comment