Messages.suppress File Can Hide Serious Issues

Historically, when I was brought into situations where a problem within the EMR system was difficult to track down, I used the message logs quite extensively. They helped me understand what was happening and where the troubleshooting should begin, and they explained any application-specific problems. This method worked very effectively until Millennium® 2007.18 TP4, when a new file in the $cer_mgr directory — the messages.suppress file — started shipping with some troubling exclusions, namely ScriptError and  ScriptErrMsg events.

The messages.suppress file stops Millennium from writing events to the message logs. This file was designed to stop creating message log events when an engineer accidentally sent an event to clients with an incorrectly set LogLevel. It’s a legitimate place for extraneous events that would otherwise clutter your log files and potentially overwrite messages that need attention. For example, the event CRM_ClientTimeZone writes the current time zone to the message log. Since domains rarely change time zones, the event is seldom of benefit when troubleshooting an issue. Yet it can cause a lot of application node writes, since every transaction going through the application server with this set has to write a message log row into the system message log as well as the specific server message log. To reduce the application node’s disk I/O and performance variability, it’s good to have CRM_ClientTimeZone in the messages.suppress file and exclude these events from the message logs.

If you hide the wrong events, however, you miss out on important information. A problem, failure or issue that should be written to the message log goes undocumented, and your clinical or technical support team has no record to assist them in identifying and resolving issues.

So it makes no sense to me that with the initial TP4, ScriptError and ScriptErrMsg events were included in the messages.suppress exclusions and that the file has since expanded to include ScriptErrors-Fatal and ScriptErrors-NonFatal events. You need visibility into these events. They indicate that an application or Oracle has a problem (often a code or build issue) and needs help to fix it.

What might the problem be? ScriptError events include blood bank transfusion script errors, MDI download problems, scripts failing to run (such as those getting PharmNet frequencies) and Oracle deadlock issues. These events all have a negative outcome to the clinicians using Millennium and the business office needing to bill accurately for services delivered.

ScriptErrors-Fatal events include memory problems that prevent clinical information from being updated and from being viewed by the clinicians who need this information.

ScriptErrors-NonFatal events show a wide range of issues. Expert Knowledge System (EKS) compile errors, Visual Explorer errors, SurgiNet documentation errors, getting orders asynchronously for PowerChart, Lights On Network data-gathering script errors and sequence number problems are all events that can degrade Millennium performance or even cause a failure. These errors must be logged so that problems can be seen and corrected.

Prognosis: Keeping your messages.suppress file properly set will ensure that the Millennium message logs contain the information that your support team needs to identify and resolve issues in a timely manner.