I am frequently asked about maintenance of the Cerner Millennium® system’s MQ queues, specifically what to do with the Admin.Reply, Audit.Exception, Exception and Z_Exception queues. It’s an excellent question, and the answer is not as clear-cut as you might believe. Each of these queues has different reasons for filling up or keeping transactions and different impacts if they do fill up.
Let’s look at the purpose of each of these queues.
The Admin.Reply queue is used to ensure that appropriate communication lines exist for a server starting a transaction. Millennium executables check with MQ to see if the needed queues exist on the node. So, for example, if an SSREQ/SSREP or RDM queue is needed, the Admin.Reply queue either finds the needed queues or creates them. At that point, the transaction is supposed to be deleted from the queue. The problem with this queue is that various Millennium versions and executables do not always delete the transaction correctly and the queue fills up. Once the queue is full, no new Millennium servers can be started from the Server Control Panel (SCP), command line, Panther or Olympus. So any automated cycle-server jobs as well as any interactive server cycling will fail.
This problem can very, very quickly cause significant performance and stability issues in your domain. I helped one client whose Admin.Reply queue, which had a Max_Depth of 15,000 rows, filled up in less than 10 minutes. The client cycled servers between midnight and 7 a.m., but because this queue was full, all of the cycle jobs failed and caused performance issues.
So how do you prevent a similar scenario? By the time you see a transaction in this queue, the work the transaction was doing is complete. These transactions do not contain any clinical or financial information, so it’s safe to delete them. I recommend that you check this queue on each application node at least once a week and delete any transactions you find.
The Audit.Exception queue is connected with the queues CPMSRVAUDIT, CPMSRVAUDITBATCH, CPMSRVAUDITOUT, SSREP.SRVAUDITMGR and SSREQ.SRVAUDITMGR. It is designed to capture transactions that could not be sent to P2Sentinel, which can happen for any variety of reasons. (Be aware that P2Sentinel is the third-party product used to capture information and could cause HIPAA violations or show inappropriate use of Millennium by your organization’s staff.)
Begin your troubleshooting of this queue by verifying that P2Sentinel was up fully and able to take transactions. If P2Sentinel can take transactions, then you should be able to replay these transactions. If you do not replay or recover these transactions, there will be gaps in the auditing information that you need to be aware of. It is possible to have incorrectly formed transactions, which you might have to delete. Before you do, I suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance. Your organization might need to complete additional documentation prior to any deletions.
The Exception queue is designed to hold transactions that could not be put on an MQ queue. How does that make any sense? Here’s the basis of MQ: A process or executable has a transaction to hand off to another process or executable, and application programmers want to make certain these transactions make it from one process to another. So the executable that wants to hand off a transaction puts that transaction into a queue. From there, the receiving executable retrieves the transaction and processes it. Still confused? Suppose the transaction is my electric bill. The electric company wants me to get the bill (transaction), which it delivers to my mailbox (MQ queue), where I go and get the bill (transaction). If for some reason the postal carrier cannot put my bill into my mailbox, the bill will go to the infamous dead-letter office (Exception queue). Someone at the central post office will then do additional processing to send me my bill.
In the case of Millennium, the bill is actually a clinical transaction, so you can’t ignore it. Any transactions sent to the Exception queue have clinical or financial data that was supposed to be processed and acted upon but never made it to queue it was originally sent to. It’s important to recover this information, which you generally can do simply by replaying the transaction to the appropriate queue. This is a little like UPS leaving a message on your door stating they need a signature before leaving a package, and since you were not home, they will try again tomorrow. The next day you stay home and get the package.
A replay generally works with Exception queue transactions because the failure reasons are often transient. If a replay does not work, you will want to log an SR to get additional assistance in either processing or deleting the transactions. As with the Audit.Exception queue, Exception queue transactions generally include clinical or financial data, so deleting them could create gaps in patient records. I would suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance before deleting these transactions. You might need to complete additional documentation prior to any deletions.
The Z_Exception queue is my favorite. It is used by the Queue Control Panel (QCP) when the front-end QCPView.exe or Quemon.exe or any other Millennium executable cannot communicate with the QCP. Since the QCP does not handle any clinical or financial data, I would suggest deleting the transactions in this queue. If the Z_Exception queue fills up, the QCP will not work normally and you will not be able to use QCPView.exe or Quemon.exe.
Prognosis: A little housekeeping with the Millennium Exception queues will help make your EMR environment more stable.
