Today I'll tackle a topic new to this blog: successful failover. Your organization, like many that I talk with, might have brought in a very highly paid consultant to set up failover for your Cerner Millennium® system. Yet, when you need to fail one application node to another to perform maintenance, the interfaces don't fail over correctly. Let me give you a few ideas that might help — and other than a change in your thinking, my ideas won't cost you a thing.
Cerner has extended some functionality in Open Engine's (OEN) Oracle tables that makes failover less painful for its clients and their Millennium support team. Users can set up the system so that the base Millennium failover scripts automatically start the interfaces on the failover node. (In the OEN_PERSONALITY table, where the column Name is PRODUCTION, the flag would be set to Y.)
I know I'm in danger of losing the OEN team here, but keep breathing and hear me out. Typically, the OEN team believes that nothing should be automatically failed over because you could have synchronization issues when more than one registration system sends Millennium ADT transactions. Even single ADT systems, where other systems send Millennium orders, order statuses, transcriptions, etc., can have synchronization issues. These issues can result in an order coming through before the patient has been added to Millennium. I don't dispute your concern. However, I absolutely do not agree that none of the interface servers can be automatically failed over.
ESI Servers and Outbound Interfaces
Two types of servers, in particular, can be set to fail over automatically with no synchronization risk for incoming interfaces. They are:
- The ESI servers, or ESI COMM servers. These are not SCP Entry ID 250 and its clones but the servers that take the Inbound client transactions and route them to SCP Entry ID 250 (CPM ESI) and its clones as well as SCP Entry ID 335 (PM Trans Async).
- The Outbound interfaces, those that send transactions to systems outside of Millennium.
Finding these interface servers can be tricky — Millennium OpenView and Olympus make you go through several screens and tabs — so I suggest you start an interactive CCL session and run the following query to find the ESI COMM servers:
Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='ESI')
order by pi.SCP_EID asc
GO
The resulting five columns will show you the ESI COMM servers, which can all have their PRODUCTION flag safely set to Y. They will only process transactions after you have started the Inbound COMM server that feeds these servers' transactions. That gives you one less set of servers that have to be started for the Inbound interfaces. I would suggest that the primary ADT Inbound interface be enabled for automatic failover. If during a controlled outage, you do not want the primary ADT interface to be up and running first after the failover, you can easily change the PRODUCTION flag back to N.
The interfaces configured to send transactions to systems outside of Millennium can be found by running this CCL command:
Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PACKESO' and op.value > '0')
order by pi.SCP_EID asc
GO
As with the ESI COMM servers, you can safely set the PRODUCTION flag to Y. Now as your clinicians are doing their normal work, the transactions configured to leave Millennium will not get hung up in Millennium for any length of time.
You might want to take one extra safety measure if you have Outbound pharmacy orders. SCP Entry ID 625 (Medication Dispense and Dose Server) and ID 626 (Medication Fill/Report Server) have a property called FsiCqmDowntimePref that can be added and configured. Set this property to 1 so that Outbound pharmacy transactions can be recovered if SCP Entry ID 252, the FSI ESO-CQM server, is down for any reason.
Additional Servers for Failover Node
Another factor that can impede interface failovers is if any of the following servers are not already running on the failover node:
- SCP Entry ID 250, the CPM ESI servers, along with any clones or copies. If no interface server is running on the failover node, then nothing should be sending these servers any transactions.
- SCP Entry ID 107, ORM Order Write Synch for FSI.
- SCP Entry ID 252, the FSI ESO-CQM server.
- SCP Entry ID 205, the Clinical Event server for ESI, and any of its clones or copies.
The cerner.vars file should be reviewed for the single-instance servers. The file should include at least the following servers: SCP Entry ID 240, 243, 245, 255, 270, 275, 276, 277, 280, 281, 282, 283, 284, 285, 291 and 292. I excluded MDI servers from this list because I am only reviewing interfaces and they involve a great number of variables, which I'll talk about later.
Keeping TCP/IP Ports Free
The last area I want to cover involves the TCP/IP ports. Interfaces will not fail over properly if the TCP/IP ports that you have configured the interfaces to communicate with are in use. How can the ports be in use? Through the normal cycling of servers, ports that initially were free can become used over time. You can prevent this by accessing the file named "/etc/services" and adding the incoming ports for the interfaces and even the MDI servers. For example, telnet is a service that runs on a port, normally 23. So the entry for this would look like:
telnet 23/tcp
If you want to reserve your Inbound interface and MDI ports, all you have to do is add the ports and a name to this file. I suggest you use the SCP Description (minus spaces) for the name along with the SCP Entry ID or OEN PID for clarity, as shown below:
telnet 23/tcp
ADTin 10046/tcp
ORUin 11001/tcp
Once added, these ports will not be used for anything else.
To create a list of the TCP ports you need to put in the "/etc/services" file, use the following command in CCL:
Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PORT' and op.value > '0')
order by pi.SCP_EID asc
GO
Dealing with MDIs
For MDIs, you have to run the command enumlog –all on the two nodes and verify that all of the MDI configurations exist on both nodes and match exactlyon both nodes. Because typographical errors are extremely painful to find, I recommend you output the data to Excel and use the EXACT function to compare cells. It will quickly find any typographical errors.
If you want the MDIs to fail over automatically, you can use the following command for CCL to find them and then use OEN_PROC_VIEW GO to modify their PRODUCTION flag to Y:
Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='CUSTOM')
order by pi.SCP_EID asc
GO
I did not include the MDI servers in the cerner.vars list because too many variables exist for these. Which MDI servers need to be in the file and not able to run on the failover node depend on the code level you are on, the Millennium Distribution Packages you have taken and your volumes. If you have MDIs that are not in the new CareAware or iBus framework, log a Service Request to verify if your code level can have the MDI servers running on multiple application nodes or if you can only run a single instance on a single node. Review the server guide for the MDI servers, looking for the section titled "Considerations for Multiple-Node and Multiple-Instance Usage." This will let you know if you can have more than a single instance and if you can run the server on multiple nodes. I want to emphasize that if you have an MDI server, like SCP Entry ID 230 (MDI Genlab Result Client) and you can run multiple instances on multiple nodes, then on the failover node the server will take very little CPU or memory after it is started. It will idle until the failover event.
Prognosis: A little preparation work can dramatically ease interface and MDI failovers and make your night, weekend and holiday events much shorter and quieter.
