Heya, In tools and around arquillian I'm seeing more and more of $subject error that I can't figure out the cause of. I've found https://issues.jboss.org/browse/WFLY-3152 which David responds the error message is clear. I might be clear, but I can't figure out why starting wildfly the exact same way will in one case result in this error but then on second start it just works. I first thought that our tools might not be setting the LogManager (which it does not), but when I look at standalone.sh in the same server there is also no setting of this property hence afaics the server is being obtuse. The answer in all cases seem to have been "have tried restarting it?" and then things starts working, until it doesn't - and then we restart again. Any idea on what is going on here ? Thanks, /max http://about.me/maxandersen _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

As Stuart asked more details would help. The only way this could happen is if something, an agent for example, attempts to get a logger before JBoss Modules initializes logging. JBoss Modules looks for a META-INF/services/ java.util.logging.LogManager service which JBoss Log Manager has. If an agent needs to be used the java.util.logging.manager needs to be set to org.jboss.logmanager.LogManager before the agent initializes a logger.

On 10/19/2014 12:42 AM, Stuart Douglas wrote: > Are you using a java agent by any chance? > > If you are then the java agent is probably causing the issue, by > attempting to log something before JBoss modules can set the log > manager. > > If you are not using a java agent then I am not sure exactly what the > cause might be. Looking at the code it looks like it is possible for > a > custom security manager to break logging as the security manager is > initialized first, however that does not seem likely. > > Can you share the details on exactly what command line is being used > to > start the server? > > Stuart > > Max Rydahl Andersen wrote: >> Heya, >> >> In tools and around arquillian I'm seeing more and more of $subject >> error that I can't figure out the cause of. >> >> I've found https://issues.jboss.org/browse/WFLY-3152 which David >> responds the error message is clear. >> >> I might be clear, but I can't figure out why starting wildfly the >> exact >> same way will in one case result in this >> error but then on second start it just works. >> >> I first thought that our tools might not be setting the LogManager >> (which it does not), but when I look at standalone.sh >> in the same server there is also no setting of this property hence >> afaics the server is being obtuse. >> >> The answer in all cases seem to have been "have tried restarting >> it?" >> and then things starts working, until it doesn't - and >> then we restart again. >> >> Any idea on what is going on here ? >> >> Thanks, >> /max >> http://about.me/maxandersen >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev -- James R. Perkins JBoss by Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 20 Oct 2014, at 11:20, Max Rydahl Andersen wrote:> On 20 Oct 2014, at 1:48, James R. Perkins wrote: > >> As Stuart asked more details would help. >> >> The only way this could happen is if something, an agent for example, >> attempts to get a logger before JBoss Modules initializes logging. >> JBoss Modules looks for a META-INF/services/ >> java.util.logging.LogManager service which JBoss Log Manager has. If >> an agent needs to be used the java.util.logging.manager needs to be >> set to org.jboss.logmanager.LogManager before the agent initializes a >> logger. > > It is all in the jira. > > And no, these errors have been happening on what is just plain vanilla > WildFly or EAP/JPP bundles. > > We do not do any agent funkyness afaik. > > And as said the "workaround" so far have been "restart it" and it > works. > > zero changes to the state or arguments. okey so there been some new movement in this. https://bugzilla.redhat.com/show_bug.cgi?id=1037970 Indicating there actually is a race condition if something pings the jvm via jmx during the startup *before* jboss main boot process manages to set the logmanager. Something that will happen because we actually have a feature that scans for locally running jvm's and attach an agent to gather info. Suggested workaround is to add this to our launch: JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager" JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/main/jboss-logmanager-1.3.1.Final-redhat-1.jar" -logging org.jboss.logmanager.LogManager Question from us is, is the above workaround proper ? Will EAP/WildFly change their boot arguments to match to avoid this race condition issue for bat/sh launched servers too ?

>> >> We do not do any agent funkyness afaik. >> >> And as said the "workaround" so far have been "restart it" and it >> works. >> >> zero changes to the state or arguments. > > okey so there been some new movement in this. > > https://bugzilla.redhat.com/show_bug.cgi?id=1037970 > > Indicating there actually is a race condition if something pings the > jvm > via jmx during the startup > *before* jboss main boot process manages to set the logmanager. > Something that will happen because > we actually have a feature that scans for locally running jvm's and > attach an agent to gather info. > > Suggested workaround is to add this to our launch: > JAVA_OPTS="$JAVA_OPTS > -Djava.util.logging.manager=org.jboss.logmanager.LogManager" > JAVA_OPTS="$JAVA_OPTS > -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/main/jboss-logmanager-1.3.1.Final-redhat-1.jar" > -logging org.jboss.logmanager.LogManager > > Question from us is, is the above workaround proper ? Will > EAP/WildFly > change their boot arguments to match to avoid this race condition > issue > for bat/sh launched servers too ? No, there's a very good chance that will cause weird stuff to happen, like two copies of the log manager being installed. It may work, mostly, but this is not a risk-free solution.

The "proper fix" is to come up with some kind of agent mode for jboss-modules, but it's not really clear how this can be accomplished without yet more goofy side-effects.

On 11/24/2014 10:40 AM, Max Rydahl Andersen wrote: >>>> >>>> We do not do any agent funkyness afaik. >>>> >>>> And as said the "workaround" so far have been "restart it" and it >>>> works. >>>> >>>> zero changes to the state or arguments. >>> >>> okey so there been some new movement in this. >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1037970 >>> >>> Indicating there actually is a race condition if something pings >>> the jvm >>> via jmx during the startup >>> *before* jboss main boot process manages to set the logmanager. >>> Something that will happen because >>> we actually have a feature that scans for locally running jvm's and >>> attach an agent to gather info. >>> >>> Suggested workaround is to add this to our launch: >>> JAVA_OPTS="$JAVA_OPTS >>> -Djava.util.logging.manager=org.jboss.logmanager.LogManager" >>> JAVA_OPTS="$JAVA_OPTS >>> -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/main/jboss-logmanager-1.3.1.Final-redhat-1.jar" >>> >>> -logging org.jboss.logmanager.LogManager >>> >>> Question from us is, is the above workaround proper ? Will >>> EAP/WildFly >>> change their boot arguments to match to avoid this race condition >>> issue >>> for bat/sh launched servers too ? >> >> No, there's a very good chance that will cause weird stuff to >> happen, >> like two copies of the log manager being installed. It may work, >> mostly, but this is not a risk-free solution. > > Yeah, that was what I feared...btw. this is what the official > recommendation is on the knowledge base article about this issue ;/ > > But can you confirm that the issue is can and will happen if a remote > process tries to install an agent and even if the agent has zero > jboss > logging usage > that can somehow result in jboss logging getting activated before it > has > settings configured ? > > I'm just wondering what code actually is being invoked that needs > this > log manager when we are installing an agent very early in the start > up > phase ? Conceptually we don't need or want our log manager to be installed until we have something interesting to log. But, you only get one chance to install the log manager, and the JVM initializes it whenever it wants, so that's the basic summary of the problem. The only way to "beat" the JVM is to get in there ahead of it. You could theoretically dig in with reflection and swap out the log manager after the fact, but then you'd also have to somehow swap out all the loggers made with the original log manager, otherwise what happens is some log messages just go missing. Another option is to install some kind of "proxy" log manager that has a facility to swap out all the backing loggers.