Log4j dynamic log level changes

Recently, I've been working with a colleague on implementing a solution to use Spring JMS to pass messages to other JVMs in a clustered environment for the purpose of setting log4j logging levels to something else.  For example, by configuration file, I've got 'org.foo.bar' logger set to level INFO.  I've found that during some production issue I need to set 'org.foo.bar' to level DEBUG.  Without redeploying the application, I want to set that level for every JVM in my cluster.  I don't want to have to go to each JVM and set that level I want to go to one and let the application handle the rest.

We're using WebSphere 7.0 for z/OS and taking advantage of the included SIBUS. SIBUS is an all java implementation of MQ. I don't believe it is as robust as MQ, but for this purpose it is good enough.

This solution was chosen over others because it can all be implemented within our application except for the SIBUS configuration. We discussed using JMX, but found that we would have potential security limitations. Further we could switch out SIBUS for MQ if need be without application changes.

The solution includes a JSP page to display and set the different log levels for each logger defined. The page reads the loggers that are defined at runtime and code time. The server processes requests from the JSP and builds the message to be processed. A listener processes the message in each JVM to update the logger.

A challenge we ran across was that in WAS z/OS the messaging bus runs as a parallel task (adjunct) to the application  server (servant).  Those tasks start in parallel when a bounce of the server is performed.  We discussed with IBM the possibility of existing implementation to serialize those start ups such that the adjunct starts before the servant.  They didn't have that as a current implementation and suggested that we open an enhancement request.  The challenge is that Spring JMS is looking for the messaging bus to be active during startup.  If the messaging bus is not active, Spring JMS will try to refresh the connection at intervals that you set.  The key to all of this is that Spring JMS run in a separate thread than the thread that is used to start the application context.  If they run in the same thread and the messaging bus is not found on a bounce, the application will fail to start.  IBM recommended some code that is part of their documentation to search for the messaging bus and delay further processing until it is found.  The logic is WebSphere specific and we wanted to remain platform agnostic.

Follow the information in this developerWorks article to a tee and you should have no issues.

http://www.ibm.com/developerworks/websphere/techjournal/0609_alcott/0609_alcott.html

Credit to Srini Ivaturi for this solution. It was his design. I helped with the implementation.