Occasionally I have clients who are unable to start new instances of Millennium®application servers or cycle their servers. Let’s walk through the steps to troubleshoot this problem.
The first place to look is the ADMIN.REPLY queue. If the queue is full, no new Millennium servers can be started.
The next place to check is the operating system’s semaphores that Millennium is using. The executable called semdump, which is used on the Unix platforms, AIX, HP-UX and Linux, shows you on each application node if there are any locks on any of the domain semaphores. Access the data by typing the command:
./semdump –v
Group [x] semaphores
Assuming you have a LIBPATH statement where $cer_exe is or in the directory of $cer_exe, you will see a table that looks something like the following (columns have been removed for space reasons):
Look at the Waiters column, which contains four digits followed by a slash (/) and four more digits. The first row in the table, Entry ID 35 IPC Key Table, indicates a persistent lock on that resource. In this case, it stopped the ability of new Millennium servers. It can also indicate why an application is hanging or frozen.
To break the lock, execute the following command:
ps –ef | grep 1492
The command could return more than a single row or line. You want to look for the second column being an exact match to the PID number. Information on server 1492 from the example above appears below:
If you still have access to SCPVIEW from Citrix or a fat client or access to Servers from Panther, you want to stop that server gracefully. If you cannot do it from these tools, you need to have root access and perform a kill on that PID. (Please do not do a kill -9 as this can cause other problems on the application node.)
The last thing to do is to restart the server that had to be stopped so that you are running the appropriate number of instances for your configuration.
Prognosis: When you are unable to start a new server or cycle an existing server, a few simple commands can find and resolve the problem.
