Avoid Making a Nervous Change Detection System
In the Visible Ops methodology, we stress the use of automatic change detection systems. To accrue the benefits, it is important that the tools be implemented correctly. The reason for this caveat is that the improper use of a change detection tool can result in a nervous system that generates a constant stream of alerts. As a result, the flood of alerts reduces reliance on the system and rather than detecting and reporting on the proverbial needle in the haystack, the system generates a new haystack! Thus, we need to review three basic principles to maximize the value of a change detection tool.
Defined Process – To maximize benefits, a detective control must reinforce a defined well conceived change management process. An integrity management system is a tool that aids in reinforcing processes and culture but it is not a process in and of itself. Identifying roles and responsibilities, when changes can occur, in what system the change records are maintained and so on are all things that must be thought out in advance. In general, always define processes first based on requirements. After that, then purchase and implement a tool.
Defined Scope – The configuration items (CIs) to monitor should be clearly identified and prioritized based on risk. If you have 3,000 CIs (servers and network devices) but only 60 of them really matter, then only invest the time and resources to monitor just the critical ones. Any detective control can be set up rather quickly but it will also take time to review alerts and ensure that all changes can be accounted for. Unless the risks merit the ongoing operation expense, it is better to limit the scope.
Focused Targets – If the scope sets the breadth of what is monitored, then the next step is to focus on what is to be monitored on each CI. For example, if you have a Windows server, monitoring the whole server is counter-productive as there are files and registry entries that are constantly changing and will generate alerts constantly. In these situations, rather than having meaningful alerts, the notifications will quickly degenerate into meaningless noise that is ignored. Instead, the specific areas of interest should be defined and only those areas monitored. For example, perhaps you monitor certain branches of the registry, the executables and DLLs in the Windows\System32 folder and then some specific application folders. The intent must always be to only monitor what matters.
In conclusion, Visible Ops strongly recommends detective controls as they can truly assist organizations in their implementation and ongoing monitoring of changes. This serves to foster cultural change, lower levels of unplanned work, and improved availability. To achieve this, make sure that the change management process is defined, that the scope is appropriately defined and that what matters is what is monitored.