Re: [jboss-dev] Delayed Shutdown
by Brian Stansberry
Thanks, Ron. Yeah, I don't think there is any remoting issue here;
seems like it was just bad luck that its finalize() method logged
something in the middle of whatever the real problem was.
And, doh, for sure it's not the JBAS-5590 change, as the EJB3 connector
isn't even touched by the UnifiedInvoker.
Ron Sigal wrote:
> Hi Brian,
>
> Sorry for the delay. I had an eye test this afternoon, and the world is
> even murkier than usual today. :-)
>
> 1. Like you, I don't see how JBAS-5590 could cause this problem.
>
> 2. If there were anything in Remoting that would concern me, it would
> be some changes I made in response to JBREM-821 "JBoss Remoting fails
> under load" - in particular, the part where Tim Fox said
>
>> 2) The connector close process is not clean. I would suggest the code
>> should be changed so the server threads are shut down at the beginning
>> of the connector close process.
>> I.e.
>> a) Wait for current invocations on that server invoker to complete -
>> and don't allow any more.
>> b) Once all are complete shut it down.
>
> So my first thought was to worry that some worker threads were hanging
> up. But the lines
>
>>
>> 2008-06-11 05:31:02,486 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker] (JBoss
>> Shutdown Hook) SocketServerInvoker[10.18.6.175:3873] stopped threads
>> in clientpool
>> 2008-06-11 05:31:02,486 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker]
>> (ServerSocketRefresh) ServerSocketRefresh shutting down
>> 2008-06-11 05:31:02,487 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker] (JBoss
>> Shutdown Hook) SocketServerInvoker[10.18.6.175:3873] stopped threads
>> in threadpool
>
> convince me that all is well in that regard.
>
> 3. The EJB3 server shouldn't even have any ServerInvokerCallbackHandlers
> to shut down.
>
> 4. So, I don't *think* there are any Remoting problems here. This is
> the kind of event that really makes me believe in gremlins.
>
> 5. I suppose the only reason for the existence of
> SocketServerInvoker.finalize() is just to be super sure someone doesn't
> forget to call stop(). But if it's causing a problem for some as yet
> unknown reason, I guess we could just remove it.
>
> -Ron
>
> Brian Stansberry wrote:
>> Ron,
>>
>> I'd better defer this one to you for an opinion. :)
>>
>> I don't really see how the JBAS-5590 change would cause this, but that
>> was a minor issue affecting just one unit test of an edge-case, so for
>> now I'll revert that and re-open the JIRA. Once we know it's safe to
>> proceed I can restore the fix.
>>
>> Here's what I've found:
>>
>> The SocketServerInvoker that might be taking a long time to finalize
>> is the one for the EJB3 connector:
>>
>> 2008-06-11 05:31:12,707 DEBUG
>> [org.jboss.deployers.plugins.deployers.DeployersImpl] (JBoss Shutdown
>> Hook) Fully Undeployed
>> vfsfile:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/build/output/jboss-5.0.0.CR1/server/all/conf/jboss-service.xml
>>
>> 2008-06-11 05:31:54,744 DEBUG [org.jboss.remoting.ServerInvoker]
>> (Finalizer) SocketServerInvoker[10.18.6.175:3873] stopped
>>
>> Looking earlier in the log, I see that SocketServerInvoker was already
>> stopped as part of the undeploy of the connector:
>>
>> 2008-06-11 05:31:02,486 DEBUG [org.jboss.system.ServiceController]
>> (JBoss Shutdown Hook) stopping service:
>> jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
>> 2008-06-11 05:31:02,486 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker] (JBoss
>> Shutdown Hook) SocketServerInvoker[10.18.6.175:3873] stopped threads
>> in clientpool
>> 2008-06-11 05:31:02,486 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker]
>> (ServerSocketRefresh) ServerSocketRefresh shutting down
>> 2008-06-11 05:31:02,487 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker] (JBoss
>> Shutdown Hook) SocketServerInvoker[10.18.6.175:3873] stopped threads
>> in threadpool
>> 2008-06-11 05:31:02,487 DEBUG
>> [org.jboss.remoting.transport.socket.SocketServerInvoker] (JBoss
>> Shutdown Hook) SocketServerInvoker[10.18.6.175:3873] exiting
>> 2008-06-11 05:31:02,487 DEBUG [org.jboss.remoting.ServerInvoker]
>> (JBoss Shutdown Hook) SocketServerInvoker[10.18.6.175:3873] stopped
>> 2008-06-11 05:31:02,487 DEBUG [org.jboss.system.ServiceController]
>> (JBoss Shutdown Hook) destroying service:
>> jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
>> 2008-06-11 05:31:02,487 DEBUG [org.jboss.system.ServiceController]
>> (JBoss Shutdown Hook) removing service:
>> jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
>> 2008-06-11 05:31:02,487 DEBUG [org.jboss.system.ServiceCreator] (JBoss
>> Shutdown Hook) Removing mbean from server:
>> jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
>> 2008-06-11 05:31:02,490 DEBUG
>> [org.jboss.deployers.structure.spi.helpers.AbstractDeploymentContext]
>> (JBoss Shutdown Hook) Removed component
>> jboss.remoting:handler=ejb3,name=DefaultEjb3Connector,type=Connector
>> from
>> vfsfile:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/build/output/jboss-5.0.0.CR1/server/all/deploy/ejb3-connectors-service.xml
>>
>>
>> The finalize() method on SocketServerInvoker calls stop() again; hence
>> the second logging. Looking at the stop() code I don't see anything
>> that should take a long time; the first call to stop() should have
>> already called destroy() on any ServerInvokerCallbackHandler, and once
>> that's done, doing it again should be very fast, just a couple of null
>> checks.
>>
>> It's quite possible this is completely unrelated to remoting; i.e.
>> something else is going on with the Finalizer thread and it's just our
>> bad luck that remoting happened to log something and focus attention
>> on us. :-)
>>
>> - Brian
>>
>> Brian Stansberry wrote:
>>> I'll have a look. I for sure I didn't see that effect when I tested
>>> that change.
>>>
>>> Dimitris Andreadis wrote:
>>>> Hm, probably related to this:
>>>> http://jira.jboss.com/jira/browse/JBAS-5590?page=comments
>>>>
>>>> Dimitris Andreadis wrote:
>>>>> The AS testsuite fails because of a delayed shutdown of the all
>>>>> config:
>>>>> http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-Test...
>>>>>
>>>>>
>>>>> The last entry in the log is this weird ServerInvoker finalizer
>>>>> thread the seems to be holding the server for more than 42 seconds
>>>>> after the previous log message:
>>>>>
>>>>> Any idea where is this coming from?
>>>>>
>>>>> 2008-06-11 05:31:12,707 DEBUG
>>>>> [org.jboss.deployers.plugins.deployers.DeployersImpl] (JBoss
>>>>> Shutdown Hook) Fully Undeployed
>>>>> vfsfile:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/build/output/jboss-5.0.0.CR1/server/all/conf/jboss-service.xml
>>>>>
>>>>> 2008-06-11 05:31:54,744 DEBUG [org.jboss.remoting.ServerInvoker]
>>>>> (Finalizer) SocketServerInvoker[10.18.6.175:3873] stopped
>>>>>
>>>>> _______________________________________________
>>>>> jboss-development mailing list
>>>>> jboss-development(a)lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>>
>>
>
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
16 years, 6 months
jboss-metadata
by Dimitris Andreadis
I did some cleanup of the JIRA jboss-metadata project, because it's becoming impossible to
know what version contains which fixes. I had to go back re-opened released versions/closed
issues and try to guess when those were fixed based on the jira activity.
The classic mistake is to mark a bug fixed for 1.0.0.CR1, but then create a BetaN version
our of the blue and mark that as released. One way to do that is rename CR1 to BetaN. Create
a new CR1, mark BetaN released and move unresolved issues to CR1.
In any case, whoever makes a release needs to update the JIRA accordingly.
There are now 2 next versions:
Beta19 & CR1
The metadata lead together with the people that work on the project needs to decide from the
6 scheduled and the 5 unscheduled tasks what will be fixed and when. We definitely need a
Beta19 release by tomorrow, for the TCK run, and a final CR1 by early next week.
Thanks
/Dimitris
16 years, 6 months
Slowdown in AS5 startup?
by Mark Spritzler
Anyone else seeing these delays? Could it just be my war file causing
this delay? This was in both a trunk build and a build I downloaded from
Hudson
09:37:22,281 WARN [HackClassloaderMetaDataDeployer] FIXME Using loader
repository config: org.jboss.on:loader=embedded for
vfszip:/NotBackedUp/mspritzl/code/jboss-head/build/output/jboss-5.0.0.CR1/server/default/deploy/embedded-console.war
use classloading metadata not loader repository config
09:37:22,912 INFO [AspectDeployer] Deploying xml into
org.jboss.aop.AspectManager@177b093 for
BaseClassLoader@4a85c5{vfsfile:/NotBackedUp/mspritzl/code/jboss-head/build/output/jboss-5.0.0.CR1/server/default/deploy/ejb3-interceptors-aop.xml}
09:39:29,593 INFO [NativeServerConfig] JBoss Web Services - Native
Mark
16 years, 6 months
Delayed Shutdown
by Dimitris Andreadis
The AS testsuite fails because of a delayed shutdown of the all config:
http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-Test...
The last entry in the log is this weird ServerInvoker finalizer thread the seems to be
holding the server for more than 42 seconds after the previous log message:
Any idea where is this coming from?
2008-06-11 05:31:12,707 DEBUG [org.jboss.deployers.plugins.deployers.DeployersImpl] (JBoss
Shutdown Hook) Fully Undeployed
vfsfile:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/build/output/jboss-5.0.0.CR1/server/all/conf/jboss-service.xml
2008-06-11 05:31:54,744 DEBUG [org.jboss.remoting.ServerInvoker] (Finalizer)
SocketServerInvoker[10.18.6.175:3873] stopped
16 years, 6 months
jboss-vfs update breaks <service-ref> in nested jar
by Thomas Diesler
FYI,
http://jira.jboss.org/jira/browse/JBAS-5614
org.jboss:jboss-vfs:jar:2.0.0.Beta15 might cause this issue
Caused by: java.lang.RuntimeException: File not found:
/usr/share/java/workspace/jbossws-hudson-3.0.2-SNAPSHOT/hudson-home/jobs/Native-Core-AS-5.0.1/workspace/stack-native/modules/testsuite/native-tests/target/test-libs/jaxrpc-jbws1121.ear/jaxrpc-jbws1121-client.jar
at
org.jboss.virtual.plugins.context.zip.ZipFileWrapper.<init>(ZipFileWrapper.java:87)
at
org.jboss.virtual.plugins.context.zip.ZipEntryContext.init(ZipEntryContext.java:210)
at
org.jboss.virtual.plugins.context.zip.ZipEntryContext.<init>(ZipEntryContext.java:138)
at
org.jboss.virtual.plugins.context.zip.ZipEntryContext.<init>(ZipEntryContext.java:123)
at
org.jboss.virtual.plugins.context.zip.ZipEntryContextFactory.getVFS(ZipEntryContextFactory.java:100)
at
org.jboss.virtual.plugins.context.zip.ZipEntryContextFactory.getVFS(ZipEntryContextFactory.java:60)
at
org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.readObject(AbstractVirtualFileHandler.java:580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1809)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1908)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1832)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1908)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1832)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1908)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1832)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at
org.jboss.ws.core.jaxrpc.client.ServiceObjectFactoryJAXRPC.getObjectInstance(ServiceObjectFactoryJAXRPC.java:118)
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
Web Service Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
16 years, 6 months
AS5 trunk update
by Ales Justin
I've updated the following libs:
- VFS --> 2.0.0.Beta14
- windows locking issue
- better modification options
- Microcontainer --> 2.0.0.Beta15
- few minor fixes
- @Bean(Factory)
- Deployers --> 2.0.0.Beta16
- Adrian's CL fixes
- @annotation scanning improvement
- few new helper deployers
- JBossXB --> 2.0.0.CR9
- PersistenceMD fix
- EJB3 --> 0.1.2
- ...
- Persistence URL handling
Code changes:
- adding jboss-structure.xml to shutdown.sar, instructing it to do
temp copy.
- adding sys property to minimal server config, running in synch
reaper mode.
- updating Ejb3 deployer to use new PersistenceMD
- this includes hack, since I forgot to handle (vfs)zip
- going away with new ejb3 version; will fix it there
As mentioned to Dimitris, I see errors when running AS with existing
apps in deploy (they are present there before we do run.sh) - deployment
are not able to find TransactionManager.
I suspect this is effect of moving TM into deploy, and not having
explicit dependency on it (again JNDI lookup).
After my above mention changes Adrian's smoke tests produce more than
2-3 errors. :-(
I think few of them are due to TM issue.
I've commited it anyway, in order to move fwd with new MC artifacts
(which I think are not the cause of those smoke failures - perhaps only
some VFS impl detail assumption).
16 years, 6 months
Vacation Schedules for the Summer in Relation to AS 5 GA Schedule
by Andrig T Miller
Dimitris,
I think we need to look at everyone's plans for vacation this
summer as we push to get AS 5 to GA. I would hate to have someone be
missing for several weeks that is on the critical path to getting to GA,
and not no about it, or be able to schedule it so that all critical path
tasks are complete by that person before they leave. Also, I think the
critical path folks need to have a backup that can address issues while
they are away.
Can we start collecting that information, and see what plans folks have?
Thanks.
Andrig (Andy) Miller
VP of Engineering
JBoss, a division of Red Hat
16 years, 6 months
Re: [jboss-dev] Re: Big AS testsuite regression
by Dimitris Andreadis
I suppose because it's hard for individual users to make a full local testsuite run before
every commit. It takes 5-6 hours.
I'd be happy if people would monitor subsequent hudson runs to see the effects of their commits.
The AS5 testsuite is now more or less stable/predictable so it's easy to spot regression.
Sacha Labourey wrote:
> why aren't we able to catch those regression *before* we commit them to
> SVN?
>
> On 06/04/2008 12:50 PM, Alexey Loubyansky wrote:
>> Yes, it comes from metadata. The tests fail to deploy. Emanuel is
>> going to look into it.
>>
>> Caused by: java.lang.IllegalArgumentException: Can't find interface
>> null in
>> org.jboss.metadata.ejb.jboss.JBossSessionBeanMetaData@af9ed402{AuditSessionEJB}
>>
>> at
>> org.jboss.metadata.ejb.jboss.JBossEnterpriseBeanMetaData.determineResolvedJndiName(JBossEnterpriseBeanMetaData.java:702)
>>
>> at
>> org.jboss.deployment.MappedReferenceMetaDataResolverDeployer.mapEjbs(MappedReferenceMetaDataResolverDeployer.java:332)
>>
>> at
>> org.jboss.deployment.MappedReferenceMetaDataResolverDeployer.mapEndpoints(MappedReferenceMetaDataResolverDeployer.java:219)
>>
>> at
>> org.jboss.deployment.MappedReferenceMetaDataResolverDeployer.internalDeploy(MappedReferenceMetaDataResolverDeployer.java:140)
>>
>> at
>> org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
>>
>> at
>> org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:174)
>>
>>
>>
>> Dimitris Andreadis wrote:
>>> I suspect this huge regression (+1162 failures) is related to the
>>> recent jboss-metadata update:
>>>
>>> http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-Test...
>>>
>>>
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
16 years, 6 months
What is supposed to configure the JBM_ID_CACHE in the all config?
by Scott Stark
The messaging layer is reporting errors during startup of the jbossas
trunk all config. I don't see a config file that defines the
JBM_ID_CACHE table in any of the all config files. What is supposed to
do this?
[578][valkyrie: bin]$ grep -r JBM_ID_CACHE ../server/all/
../server/all/log/server.log:2008-06-03 09:47:26,876 DEBUG
[org.jboss.messaging.core.impl.JDBCSupport] (main) Failed to execute:
CREATE TABLE JBM_ID_CACHE (NODE_ID INTEGER, CNTR INTEGER, JBM_ID
VARCHAR(255), PRIMARY KEY(NODEID, CNTR))
../server/all/log/server.log:java.sql.SQLException: Column not found:
NODEID in statement [CREATE TABLE JBM_ID_CACHE (NODE_ID INTEGER, CNTR
INTEGER, JBM_ID VARCHAR(255), PRIMARY KEY(NODEID, CNTR)]
../server/all/log/server.log:java.sql.SQLException: Table not found in
statement [SELECT CNTR, ID FROM JBM_ID_CACHE WHERE NODE_ID = ?]
../server/all/log/server.log:java.sql.SQLException: Table not found in
statement [SELECT CNTR, ID FROM JBM_ID_CACHE WHERE NODE_ID = ?]
../server/all/log/server.log:java.sql.SQLException: Table not found in
statement [SELECT CNTR, ID FROM JBM_ID_CACHE WHERE NODE_ID = ?]
../server/all/log/server.log:jboss.messaging:service=ServerPeer ->
**ERROR** | java.sql.SQLException: Table not found in statement [SELECT
CNTR, ID FROM JBM_ID_CACHE WHERE NODE_ID = ?]
16 years, 6 months
