This agent itest error on travis just happened to me locally. After looking at it, I think it is a bug in inventory REST server. I'll try to explain as briefly as I can. In this particular itest [B] it creates datasources in WildFly, checks that they exist in h-inventory, removes those datasources from WildFly, and checks that they no longer exist in h-inventory. For some reason, after it is removed from WildFly and inventory, a REST query to inventory says it is still there (even though a previous, different, query said it was not there). Here's what I see: 1. "testXaDs" is an XA datasource that is created in Wildfly and the agent stored it in inventory. A "get all resources" inventory REST query "hawkular/inventory/resources?feedId=itest-rest-feed" shows it exists in inventory: testXaDs id: itest-rest-feed~Local DMR~/subsystem=datasources/xa- data-source=testXaDs parent: itest-rest-feed~Local DMR~~ type: XA Datasource 2. The test then removes that datasource from WildFly (the itest asserts via a DMR query that it is gone from WildFly). Then, making that same inventory REST query "hawkular/inventory/resources?feedId=itest-rest-feed" that "testXaDs" is not in the list - it is gone from inventory based on that "get all resources" query. This is good and expected. We see the datasource successfully removed from inventory after WildFly deleted it. 3. But then the itest queries inventory using a different REST URL (filtered on type ID): "/hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource" and this time I GET RESULTS back saying that "testXaDs" resource is still there! Even though the "get all resources" query said it was not there. This is the call here [C] which ends up at this inventory REST request [D]. Who wants to investigate the inventory service to see if I'm right that there is a bug here? If there is no bug in inventory, the only other option is it's a bug in the itest code, but I checked and everything looks good to me there. I can't see a problem in the itest code. And the fact this is intermittent (passes sometimes, fails sometimes) tells me there is something sinister going on in the inventory server (i.e. if it is a problem in the test code, you would think it would always fail). ~~~~~~~~~~ [D] https://github.com/hawkular/hawkular-agent/blob/hawkular- 1275/hawkular-agent-itest-util/src/main/java/org/ hawkular/agent/itest/util/ITestHelper.java#L93 [C] https://github.com/hawkular/hawkular-agent/blob/hawkular- 1275/hawkular-javaagent-itest-parent/hawkular-javaagent-all- itests/hawkular-javaagent-cmdgw-itest/src/test/java/org/ hawkular/agent/ws/test/DatasourceCommandITest.java#L377 [B] https://github.com/hawkular/hawkular-agent/blob/hawkular- 1275/hawkular-javaagent-itest-parent/hawkular-javaagent-all- itests/hawkular-javaagent-cmdgw-itest/src/test/java/org/ hawkular/agent/ws/test/DatasourceCommandITest.java [A] Here's the full output from the test along with the itest assertion error stack trace. There is a lot of junk, but in short, it shows the XA datasource called "testXaDs" is in inventory, then the test deletes it, we see "testXaDs" is no longer in inventory (that print out of the inventory list comes from query results of that "get all resources" REST query), but a query to inventory REST API filtering on typeId says it is still there. Doesn't happen all the time, happens half the time in travis and just happened locally to me. ===== Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=Datasource [Invoker 211968962] Invoking @AfterMethod AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] [Invoker 211968962] Invoking @BeforeMethod AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] [Invoker 211968962] Invoking org.hawkular.agent.ws.test. DatasourceCommandITest.testRemoveXaDatasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed ALL RESOURCES IN HAWKULAR INVENTORY: 67 ===== testXaDs exists and will be removed ===== --- Hawkular Java Agent id: itest-rest-feed~Local JMX~org.hawkular:type= hawkular-javaagent parent: null type: Hawkular Java Agent --- JMX [Local JMX] MemoryPool Metaspace id: itest-rest-feed~Local JMX~java.lang:name=Metaspace, type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- Processor [3] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=3 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- JMX [Remote JMX] MemoryPool PS Old Gen id: itest-rest-feed~Remote JMX~java.lang:name=PS Old Gen,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- jolokia-war.war id: itest-rest-feed~Local DMR~/deployment=jolokia-war.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- hawkular-alerts id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=hawkular-alerts parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- DLQ id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-queue=DLQ parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Queue --- InventoryServiceIspn id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=ejb3/stateless-session-bean= InventoryServiceIspn parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Stateless Session EJB --- ajp id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=ajp parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- mail-smtp id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/remote-destination-outbound-socket-binding=mail-smtp parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Remote Destination Outbound Socket Binding --- https id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=https parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- management-https id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=management-https parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- InventoryStats id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=ejb3/singleton-bean=InventoryStats parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Singleton EJB --- testXaDs id: itest-rest-feed~Local DMR~/subsystem=datasources/xa- data-source=testXaDs parent: itest-rest-feed~Local DMR~~ type: XA Datasource --- File Store [/dev/mapper/fedora_mazzwork-home] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=/dev/mapper/fedora_mazzwork-home parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- Memory id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/MEMORY=Memory parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Memory --- Processor [2] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=2 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- File Store [/dev/sda1] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=/dev/sda1 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- JMX [Remote JMX] MemoryPool Code Cache id: itest-rest-feed~Remote JMX~java.lang:name=Code Cache,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Remote JMX] MemoryPool Metaspace id: itest-rest-feed~Remote JMX~java.lang:name=Metaspace, type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Remote JMX] MemoryPool PS Survivor Space id: itest-rest-feed~Remote JMX~java.lang:name=PS Survivor Space,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- txn-recovery-environment id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=txn-recovery-environment parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- default id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default parent: itest-rest-feed~Local DMR~~ type: Messaging Server --- Infinispan id: itest-rest-feed~Local DMR~/subsystem=infinispan parent: itest-rest-feed~Local DMR~~ type: Infinispan --- jolokia-agent id: itest-rest-feed~Local DMR~/deployment=jolokia-war. war/subsystem=undertow/servlet=jolokia-agent parent: itest-rest-feed~Local DMR~/deployment=jolokia-war.war type: Servlet --- hawkular-inventory-ui.war id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-ui.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- hawkular-services id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=hawkular-services parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- web id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=web parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- management-http id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=management-http parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- JMX [Local JMX] MemoryPool PS Survivor Space id: itest-rest-feed~Local JMX~java.lang:name=PS Survivor Space,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Local JMX] MemoryPool PS Eden Space id: itest-rest-feed~Local JMX~java.lang:name=PS Eden Space,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Local JMX] MemoryPool Code Cache id: itest-rest-feed~Local JMX~java.lang:name=Code Cache,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- Processor [7] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=7 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- Processor [1] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=1 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- txn-status-manager id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=txn-status-manager parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- ejb id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=ejb parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- h2 id: itest-rest-feed~Local DMR~/subsystem=datasources/ jdbc-driver=h2 parent: itest-rest-feed~Local DMR~~ type: JDBC Driver --- File Store [tmpfs] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=tmpfs parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- Processor [0] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=0 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- Processor [6] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=6 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- JMX [Remote JMX] MemoryPool PS Eden Space id: itest-rest-feed~Remote JMX~java.lang:name=PS Eden Space,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Remote JMX][Runtime] id: itest-rest-feed~Remote JMX~java.lang:type=Runtime parent: null type: Runtime MBean --- JMX [Local JMX] MemoryPool PS Old Gen id: itest-rest-feed~Local JMX~java.lang:name=PS Old Gen,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- HawkularQueue id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularQueue parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic --- Local DMR id: itest-rest-feed~Local DMR~~ parent: null type: WildFly Server --- HawkularTopic id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularTopic parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic --- hawkular-command-gateway-war.war id: itest-rest-feed~Local DMR~/deployment=hawkular- command-gateway-war.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- hibernate id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=hibernate parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- HawkularCommandEvent id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularCommandEvent parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic --- File Store [/dev/mapper/fedora_mazzwork-root] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=/dev/mapper/fedora_mazzwork-root parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- Processor [5] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=5 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- JMX [Local JMX][Runtime] id: itest-rest-feed~Local JMX~java.lang:type=Runtime parent: null type: Runtime MBean --- JMX [Local JMX] MemoryPool Compressed Class Space id: itest-rest-feed~Local JMX~java.lang:name=Compressed Class Space,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- server id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=server parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- standard-sockets id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets parent: itest-rest-feed~Local DMR~~ type: Socket Binding Group --- hawkular-inventory-service.war id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- JMX [Remote JMX] MemoryPool Compressed Class Space id: itest-rest-feed~Remote JMX~java.lang:name=Compressed Class Space,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- File Store [/] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=~ parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- itest-rest-feed_OperatingSystem id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem parent: null type: Platform_Operating System --- Processor [4] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=4 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- ExampleDS id: itest-rest-feed~Local DMR~/subsystem=datasources/ data-source=ExampleDS parent: itest-rest-feed~Local DMR~~ type: Datasource --- ExpiryQueue id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-queue=ExpiryQueue parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Queue --- org.hawkular.inventory.handlers.InventoryApp id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=undertow/servlet=org.hawkular.inventory. handlers.InventoryApp parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Servlet --- http id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=http parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- Transaction Manager id: itest-rest-feed~Local DMR~/subsystem=transactions parent: itest-rest-feed~Local DMR~~ type: Transaction Manager --- InventoryConfig id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=ejb3/singleton-bean=InventoryConfig parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Singleton EJB --- HawkularInventoryChanges id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularInventoryChanges parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic ===== Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener onOpen FINE: WebSocket opened 20:13:29,091 INFO [org.hawkular.cmdgw.command.ws.server.UIClientWebSocket] (default task-42) HAWKFEEDCOMM000013: WebSocket session [Hgf9Zznsx_ y9U14uaHPPWIPuSarLogLgugUCweVK] opened for endpoint [/ui/ws] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Received message[0] from WebSocket: [ActualMessage [type=application/vnd.okhttp.websocket+text; charset=utf-8, body=WelcomeResponse={"sessionId":"Hgf9Zznsx_ y9U14uaHPPWIPuSarLogLgugUCweVK"}]] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Message[0]: [PASSED] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$MessageAnswer$1 run FINE: Sending over WebSocket: RemoveDatasourceRequest={" authentication":{"username":"itest-rest","password":"itest-rest-password"}, "feedId":"itest-rest-feed","resourceId":"itest-rest-feed~Local DMR~/subsystem=datasources/xa-data-source=testXaDs"} Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$MessageAnswer$1 run FINE: No sleep configured 20:13:29,102 INFO [org.hawkular.cmdgw.command.ws.server.AbstractGatewayWebSocket] (default task-43) HAWKFEEDCOMM000014: Received message [org.hawkular.cmdgw.api.RemoveDatasourceRequest] from WebSocket session [Hgf9Zznsx_y9U14uaHPPWIPuSarLogLgugUCweVK] of [/ui/ws] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Received message[1] from WebSocket: [ActualMessage [type=application/vnd.okhttp.websocket+text; charset=utf-8, body=GenericSuccessResponse={"message":"The request has been forwarded to feed [itest-rest-feed] (MessageId=ID:27f779d9-bd07- 11e7-a03b-55f35baba5f1)"}]] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Message[1]: [PASSED] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$ExpectedEvent scheduleAnswer FINE: No answer to send for message[1] 20:13:29,108 INFO [org.hawkular.agent.monitor.cmd. AbstractDMRResourcePathCommand] (OkHttp http://127.0.0.1:8080/...) HAWKAGENT010065: Received request to perform [Remove] on a [Datasource] given by inventory path [itest-rest-feed~Local DMR~/subsystem=datasources/xa-data-source=testXaDs] 20:13:29,150 INFO [org.hawkular.cmdgw.command.ws.server.AbstractGatewayWebSocket] (default task-47) HAWKFEEDCOMM000014: Received message [org.hawkular.cmdgw.api.RemoveDatasourceResponse] from WebSocket session [AQNPLng4hq-vgfGOOo53kVpLuQjVcKvzXAzwqPS3] of [/feed/{feedId}] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Received message[2] from WebSocket: [ActualMessage [type=application/vnd.okhttp.websocket+text; charset=utf-8, body=RemoveDatasourceResponse={"feedId":"itest-rest-feed"," resourceId":"itest-rest-feed~Local DMR~/subsystem=datasources/xa- data-source=testXaDs","destinationSessionId":"Hgf9Zznsx_ y9U14uaHPPWIPuSarLogLgugUCweVK","status":"OK","message":"Performed [Remove] on a [Datasource] given by Feed Id [itest-rest-feed] Resource Id [itest-rest-feed~Local DMR~/subsystem=datasources/xa- data-source=testXaDs]","serverRefreshIndicator":"RELOAD-REQUIRED"}]] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Message[2]: [PASSED] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$Answer$1 lambda$schedule$0 FINE: About to send manual close Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$Answer$1 lambda$schedule$0 FINE: Close sent 20:13:29,156 INFO [org.hawkular.cmdgw.command.ws.server.UIClientWebSocket] (default task-44) HAWKFEEDCOMM000015: WebSocket session [Hgf9Zznsx_ y9U14uaHPPWIPuSarLogLgugUCweVK] of [/ui/ws] closed. Reason=[CloseReason[1000,OK]] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Received message[3] from WebSocket: [ActualClose [code=1000, reason=OK]] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Message[3]: [PASSED] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$ExpectedEvent scheduleAnswer FINE: No answer to send for message[3] Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener handle FINE: Message[3] was the last expected message, sending close. Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener close FINE: Closing the websocket Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener shutDownExecutor FINE: Shutting down the executor Oct 29, 2017 8:13:29 PM org.hawkular.cmdgw.ws.test.TestWebSocketClient$TestListener shutDownExecutor FINE: Executor shut down Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed ALL RESOURCES IN HAWKULAR INVENTORY: 66 ===== testXaDs deleted, should be out of inventory ===== --- Hawkular Java Agent id: itest-rest-feed~Local JMX~org.hawkular:type= hawkular-javaagent parent: null type: Hawkular Java Agent --- JMX [Local JMX] MemoryPool Metaspace id: itest-rest-feed~Local JMX~java.lang:name=Metaspace, type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- Processor [3] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=3 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- JMX [Remote JMX] MemoryPool PS Old Gen id: itest-rest-feed~Remote JMX~java.lang:name=PS Old Gen,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- jolokia-war.war id: itest-rest-feed~Local DMR~/deployment=jolokia-war.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- hawkular-alerts id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=hawkular-alerts parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- DLQ id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-queue=DLQ parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Queue --- InventoryServiceIspn id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=ejb3/stateless-session-bean= InventoryServiceIspn parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Stateless Session EJB --- ajp id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=ajp parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- mail-smtp id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/remote-destination-outbound-socket-binding=mail-smtp parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Remote Destination Outbound Socket Binding --- https id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=https parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- management-https id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=management-https parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- InventoryStats id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=ejb3/singleton-bean=InventoryStats parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Singleton EJB --- File Store [/dev/mapper/fedora_mazzwork-home] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=/dev/mapper/fedora_mazzwork-home parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- Memory id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/MEMORY=Memory parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Memory --- Processor [2] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=2 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- File Store [/dev/sda1] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=/dev/sda1 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- JMX [Remote JMX] MemoryPool Code Cache id: itest-rest-feed~Remote JMX~java.lang:name=Code Cache,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Remote JMX] MemoryPool Metaspace id: itest-rest-feed~Remote JMX~java.lang:name=Metaspace, type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Remote JMX] MemoryPool PS Survivor Space id: itest-rest-feed~Remote JMX~java.lang:name=PS Survivor Space,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- txn-recovery-environment id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=txn-recovery-environment parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- default id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default parent: itest-rest-feed~Local DMR~~ type: Messaging Server --- Infinispan id: itest-rest-feed~Local DMR~/subsystem=infinispan parent: itest-rest-feed~Local DMR~~ type: Infinispan --- jolokia-agent id: itest-rest-feed~Local DMR~/deployment=jolokia-war. war/subsystem=undertow/servlet=jolokia-agent parent: itest-rest-feed~Local DMR~/deployment=jolokia-war.war type: Servlet --- hawkular-inventory-ui.war id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-ui.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- hawkular-services id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=hawkular-services parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- web id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=web parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- management-http id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=management-http parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- JMX [Local JMX] MemoryPool PS Survivor Space id: itest-rest-feed~Local JMX~java.lang:name=PS Survivor Space,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Local JMX] MemoryPool PS Eden Space id: itest-rest-feed~Local JMX~java.lang:name=PS Eden Space,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Local JMX] MemoryPool Code Cache id: itest-rest-feed~Local JMX~java.lang:name=Code Cache,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- Processor [7] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=7 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- Processor [1] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=1 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- txn-status-manager id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=txn-status-manager parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- ejb id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=ejb parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- h2 id: itest-rest-feed~Local DMR~/subsystem=datasources/ jdbc-driver=h2 parent: itest-rest-feed~Local DMR~~ type: JDBC Driver --- File Store [tmpfs] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=tmpfs parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- Processor [0] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=0 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- Processor [6] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=6 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- JMX [Remote JMX] MemoryPool PS Eden Space id: itest-rest-feed~Remote JMX~java.lang:name=PS Eden Space,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- JMX [Remote JMX][Runtime] id: itest-rest-feed~Remote JMX~java.lang:type=Runtime parent: null type: Runtime MBean --- JMX [Local JMX] MemoryPool PS Old Gen id: itest-rest-feed~Local JMX~java.lang:name=PS Old Gen,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- HawkularQueue id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularQueue parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic --- Local DMR id: itest-rest-feed~Local DMR~~ parent: null type: WildFly Server --- HawkularTopic id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularTopic parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic --- hawkular-command-gateway-war.war id: itest-rest-feed~Local DMR~/deployment=hawkular- command-gateway-war.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- hibernate id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=hibernate parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- HawkularCommandEvent id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularCommandEvent parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic --- File Store [/dev/mapper/fedora_mazzwork-root] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=/dev/mapper/fedora_mazzwork-root parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- Processor [5] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=5 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- JMX [Local JMX][Runtime] id: itest-rest-feed~Local JMX~java.lang:type=Runtime parent: null type: Runtime MBean --- JMX [Local JMX] MemoryPool Compressed Class Space id: itest-rest-feed~Local JMX~java.lang:name=Compressed Class Space,type=MemoryPool parent: itest-rest-feed~Local JMX~java.lang:type=Runtime type: Memory Pool MBean --- server id: itest-rest-feed~Local DMR~/subsystem=infinispan/ cache-container=server parent: itest-rest-feed~Local DMR~/subsystem=infinispan type: Infinispan Cache Container --- standard-sockets id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets parent: itest-rest-feed~Local DMR~~ type: Socket Binding Group --- hawkular-inventory-service.war id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war parent: itest-rest-feed~Local DMR~~ type: Deployment --- JMX [Remote JMX] MemoryPool Compressed Class Space id: itest-rest-feed~Remote JMX~java.lang:name=Compressed Class Space,type=MemoryPool parent: itest-rest-feed~Remote JMX~java.lang:type=Runtime type: Memory Pool MBean --- File Store [/] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/FILE_STORE=~ parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_File Store --- itest-rest-feed_OperatingSystem id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem parent: null type: Platform_Operating System --- Processor [4] id: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem/PROCESSOR=4 parent: itest-rest-feed~platform~/OPERATING_SYSTEM=itest-rest- feed_OperatingSystem type: Platform_Processor --- ExampleDS id: itest-rest-feed~Local DMR~/subsystem=datasources/ data-source=ExampleDS parent: itest-rest-feed~Local DMR~~ type: Datasource --- ExpiryQueue id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-queue=ExpiryQueue parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Queue --- org.hawkular.inventory.handlers.InventoryApp id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=undertow/servlet=org.hawkular.inventory. handlers.InventoryApp parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Servlet --- http id: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets/socket-binding=http parent: itest-rest-feed~Local DMR~/socket-binding-group= standard-sockets type: Socket Binding --- Transaction Manager id: itest-rest-feed~Local DMR~/subsystem=transactions parent: itest-rest-feed~Local DMR~~ type: Transaction Manager --- InventoryConfig id: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war/subsystem=ejb3/singleton-bean=InventoryConfig parent: itest-rest-feed~Local DMR~/deployment=hawkular- inventory-service.war type: Singleton EJB --- HawkularInventoryChanges id: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default/jms-topic=HawkularInventoryChanges parent: itest-rest-feed~Local DMR~/subsystem=messaging- activemq/server=default type: JMS Topic ===== 20:13:29,186 ERROR [org.hawkular.agent.monitor.storage.HawkularStorageAdapter] (OkHttp http://127.0.0.1:8080/...) HAWKAGENT010083: Failed to store notification: {"type":"RESOURCE_ADDED","properties":{"resourceId":"itest-rest-feed~Local DMR~~","feedId":"itest-rest-feed","resourceType":"WildFly Server"}}: java.io.IOException: status-code=[405], reason=[Method Not Allowed], url=[ http://127.0.0.1:8080/hawkular/api/notification] at org.hawkular.agent.monitor.storage.HawkularStorageAdapter$1. onResponse(HawkularStorageAdapter.java:116) at okhttp3.shaded.RealCall$AsyncCall.execute(RealCall.java:126) at okhttp3.shaded.internal.NamedRunnable.run( NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource Got code 200 and message [OK] retries: http://127.0.0.1:8080/ hawkular/inventory/resources?feedId=itest-rest-feed&typeId=XA%20Datasource [Invoker 211968962] Invoking @AfterMethod AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] ===== Invoked methods AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 StandaloneWildFlyITest.testAgent()[pri:0, instance:org.hawkular.agent. ws.test.StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 StandaloneWildFlyITest.datasourcesAddedToInventory()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 StandaloneWildFlyITest.operationParameters()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 StandaloneWildFlyITest.socketBindingGroupsInInventory()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 StandaloneWildFlyITest.machineId()[pri:0, instance:org.hawkular.agent. ws.test.StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 StandaloneWildFlyITest.resourceConfig()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneWildFlyITest@226a82c4] 577405636 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. ExecuteOperationCommandITest@77e4c80f] 2011482127 ExecuteOperationCommandITest.testExecuteAgentDiscoveryScan()[pri:0, instance:org.hawkular.agent.ws.test.ExecuteOperationCommandITest@77e4c80f] 2011482127 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. ExecuteOperationCommandITest@77e4c80f] 2011482127 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 StandaloneDeployApplicationITest.testAddDeployment()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneDeployApplicationITe st@351d0846] 891095110 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 StandaloneDeployApplicationITest.testDisableDeployment()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneDeployApplicationITe st@351d0846] 891095110 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 StandaloneDeployApplicationITest.testEnableDeployment()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneDeployApplicationITe st@351d0846] 891095110 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 StandaloneDeployApplicationITest.testRestartDeployment()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneDeployApplicationITe st@351d0846] 891095110 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 StandaloneDeployApplicationITest.testLocalAndRemoteJMXOperations()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneDeployApplicationITe st@351d0846] 891095110 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 StandaloneDeployApplicationITest.testUndeploy()[pri:0, instance:org.hawkular.agent.ws.test.StandaloneDeployApplicationITe st@351d0846] 891095110 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. StandaloneDeployApplicationITest@351d0846] 891095110 AbstractITest.before()[pri:0, instance:org.hawkular.agent. ws.test.ExportJdrCommandITest@176d53b2] 393040818 ExportJdrCommandITest.exportJdrCommand()[pri:0, instance:org.hawkular.agent.ws.test.ExportJdrCommandITest@176d53b2] 393040818 AbstractITest.after()[pri:0, instance:org.hawkular.agent. ws.test.ExportJdrCommandITest@176d53b2] 393040818 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 DatasourceCommandITest.testAddDatasource()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 DatasourceCommandITest.testAddXaDatasource()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 DatasourceCommandITest.testUpdateDatasource()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 DatasourceCommandITest.testUpdateXaDatasource()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 DatasourceCommandITest.testRemoveDatasource()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.before()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 DatasourceCommandITest.testRemoveXaDatasource()[pri:0, instance:org.hawkular.agent.ws.test.DatasourceCommandITest@4461c7e3] 1147258851 AbstractITest.after()[pri:0, instance:org.hawkular.agent.ws.test. DatasourceCommandITest@4461c7e3] 1147258851 ===== Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/Surefire suite/Surefire test.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/Surefire suite/Surefire test.xml PASSED: testAgent PASSED: datasourcesAddedToInventory PASSED: operationParameters PASSED: socketBindingGroupsInInventory PASSED: machineId PASSED: resourceConfig PASSED: testExecuteAgentDiscoveryScan PASSED: testAddDeployment PASSED: testDisableDeployment PASSED: testEnableDeployment PASSED: testRestartDeployment PASSED: testLocalAndRemoteJMXOperations PASSED: testUndeploy PASSED: exportJdrCommand PASSED: testAddDatasource PASSED: testAddXaDatasource PASSED: testUpdateDatasource PASSED: testUpdateXaDatasource PASSED: testRemoveDatasource FAILED: testRemoveXaDatasource java.lang.AssertionError: Resource [type=XA Datasource, containing=testXaDs] still found after 10 attempts. at org.hawkular.agent.itest.util.ITestHelper. waitForNoResourceContaining(ITestHelper.java:167) at org.hawkular.agent.ws.test.DatasourceCommandITest. testRemoveXaDatasource(DatasourceCommandITest.java:381) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod( MethodInvocationHelper.java:86) at org.testng.internal.Invoker.invokeMethod(Invoker.java:643) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820) at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1128) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) SKIPPED: testEnableStatistics java.lang.Throwable: Method StatisticsControlCommandITest.testEnableStatistics()[pri:0, instance:org.hawkular.agent.ws.test.StatisticsControlCommandITest@4450d156] depends on not successfully finished methods in group "DatasourceCommandITest" at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1037) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) SKIPPED: testDisableStatistics java.lang.Throwable: Method StatisticsControlCommandITest.testDisableStatistics()[pri:0, instance:org.hawkular.agent.ws.test.StatisticsControlCommandITest@4450d156] depends on not successfully finished methods at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1037) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) SKIPPED: testEnableStatisticsSubset java.lang.Throwable: Method StatisticsControlCommandITest. testEnableStatisticsSubset()[pri:0, instance:org.hawkular.agent.ws.test. StatisticsControlCommandITest@4450d156] depends on not successfully finished methods at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1037) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) SKIPPED: testAddJdbcDriver java.lang.Throwable: Method JdbcDriverCommandITest.testAddJdbcDriver()[pri:0, instance:org.hawkular.agent.ws.test.JdbcDriverCommandITest@731f8236] depends on not successfully finished methods in group " StatisticsControlCommandITest" at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1037) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) SKIPPED: testRemoveJdbcDriver java.lang.Throwable: Method JdbcDriverCommandITest.testRemoveJdbcDriver()[pri:0, instance:org.hawkular.agent.ws.test.JdbcDriverCommandITest@731f8236] depends on not successfully finished methods at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1037) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) SKIPPED: testImmutableUpdate java.lang.Throwable: Method ImmutableITest.testImmutableUpdate()[pri:0, instance:org.hawkular.agent.ws.test.ImmutableITest@1b68b9a4] depends on not successfully finished methods in group "JdbcDriverCommandITest" at org.testng.internal.Invoker.invokeTestMethods(Invoker. java:1037) at org.testng.internal.TestMethodWorker.invokeTestMethods( TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run( TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at org.testng.SuiteRunner.run(SuiteRunner.java:268) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker. java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at org.testng.TestNG.run(TestNG.java:1064) at org.apache.maven.surefire.testng.TestNGExecutor.run( TestNGExecutor.java:132) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. executeMulti(TestNGDirectoryTestSuite.java:193) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite. execute(TestNGDirectoryTestSuite.java:94) at org.apache.maven.surefire.testng.TestNGProvider.invoke( TestNGProvider.java:147) at org.apache.maven.surefire.booter.ForkedBooter. invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter. runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main( ForkedBooter.java:121) =============================================== Surefire test Tests run: 26, Failures: 1, Skips: 6 =============================================== =============================================== Surefire suite Total tests run: 26, Failures: 1, Skips: 6 =============================================== Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/testng-failed.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/Surefire suite/testng-failed.xml [TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 7 ms Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/toc.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/Surefire test.properties Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/index.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/main.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/groups.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/classes.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/reporter-output.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/methods-not-run.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/Surefire suite/testng.xml.html Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/old/index.html [TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@396e2f39: 9 ms Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.StandaloneDeployApplicationITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.ExportJdrCommandITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.ImmutableITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.ExecuteOperationCommandITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.DatasourceCommandITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.StatisticsControlCommandITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.StandaloneWildFlyITest.xml Creating /home/mazz/source/hawkular-agent/hawkular-javaagent- itest-parent/hawkular-javaagent-all-itests/hawkular- javaagent-cmdgw-itest/target/failsafe-reports/junitreports/ TEST-org.hawkular.agent.ws.test.JdbcDriverCommandITest.xml [TestNG] Time taken by org.testng.reporters.JUnitReportReporter@2eda0940: 7 ms [TestNG] Time taken by org.testng.reporters.jq.Main@7e0e6aa2: 61 ms [TestNG] Time taken by org.testng.reporters.XMLReporter@2e3fc542: 6 ms [TestNG] Time taken by org.testng.reporters.EmailableReporter2@704d6e83: 7 ms Tests run: 26, Failures: 1, Errors: 0, Skipped: 6, Time elapsed: 103.303 sec <<< FAILURE! - in TestSuite testRemoveXaDatasource(org.hawkular.agent.ws.test.DatasourceCommandITest) Time elapsed: 50.287 sec <<< FAILURE! java.lang.AssertionError: Resource [type=XA Datasource, containing=testXaDs] still found after 10 attempts. at org.hawkular.agent.ws.test.DatasourceCommandITest. testRemoveXaDatasource(DatasourceCommandITest.java:381) Results : Failed tests: DatasourceCommandITest.testRemoveXaDatasource:381 Resource [type=XA Datasource, containing=testXaDs] still found after 10 attempts. Tests run: 26, Failures: 1, Errors: 0, Skipped: 6 _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Some update, here is the latest logs I got while the test failed: https://paste.fedoraproject.org/paste/Yy8CuDvgqzX1RazON6Cmag I haven't time to dig further today, but see in the middle that lovely stack trace: *18:01:59,012 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-35) ISPN000255: Error while processing prepare: org.infinispan.transaction.WriteSkewException: Write skew detected on key itest-rest-feed~Local DMR~/subsystem=datasources/xa-data-source=testXaDs for transaction TransactionImple < ac, BasicAction: 0:ffffc0a8000c:-5784e282:59f75acd:b4 status: ActionStatus.RUNNING >* * at org.infinispan.transaction.impl.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:66)* * at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.clusteredCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:261)* I have no idea why it wasn't displayed before, and it is now. On Mon, Oct 30, 2017 at 1:15 PM, Joel Takvorian <jtakvori(a)redhat.com&gt; wrote: > I've seen the same error just once (out of 5 or 6 tries), and, > coincidentally, had the same error notified "Failed to store > notification: {"type":"RESOURCE_ADDED" etc." with "MethodNotAllowed" at > the very same moment, that is, just when the testRemoveXaDatasource test > runs. > That's maybe a trail to investigate, maybe it betrays some server-side > malfunction. I don't know if it's possible to see server-side logs during > agent itests, but it would probably help. I don't know why this error > happens. > > When I run tests successfully, I also see from times to times this error, > but it's distributed randomly in tests, and fortunately or not (rather > not), tests pass. > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Well, in details it means that is trying to write/delete at same key and it failed, and then probably the indexmanager was not updated. Did you get that trace also in the server.log as well ? On Mon, Oct 30, 2017 at 6:25 PM, Lucas Ponce <lponce(a)redhat.com&gt; wrote: > That's interesting. > I haven't got that trace, yes, it seems the indexmanager failed, still > trying to get the same one. > > On Mon, Oct 30, 2017 at 6:17 PM, Joel Takvorian <jtakvori(a)redhat.com&gt; > wrote: > >> Some update, here is the latest logs I got while the test failed: >> https://paste.fedoraproject.org/paste/Yy8CuDvgqzX1RazON6Cmag >> >> I haven't time to dig further today, but see in the middle that lovely >> stack trace: >> >> *18:01:59,012 ERROR [org.infinispan.transaction.im >> <http://org.infinispan.transaction.im>pl.TransactionCoordinator] (default >> task-35) ISPN000255: Error while processing prepare: >> org.infinispan.transaction.WriteSkewException: Write skew detected on key >> itest-rest-feed~Local DMR~/subsystem=datasources/xa-data-source=testXaDs >> for transaction TransactionImple < ac, BasicAction: >> 0:ffffc0a8000c:-5784e282:59f75acd:b4 status: ActionStatus.RUNNING >* >> * at >> org.infinispan.transaction.impl.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:66)* >> * at >> org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.clusteredCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:261)* >> >> I have no idea why it wasn't displayed before, and it is now. >> >> On Mon, Oct 30, 2017 at 1:15 PM, Joel Takvorian <jtakvori(a)redhat.com&gt; >> wrote: >> >>> I've seen the same error just once (out of 5 or 6 tries), and, >>> coincidentally, had the same error notified "Failed to store >>> notification: {"type":"RESOURCE_ADDED" etc." with "MethodNotAllowed" >>> at the very same moment, that is, just when the testRemoveXaDatasource test >>> runs. >>> That's maybe a trail to investigate, maybe it betrays some server-side >>> malfunction. I don't know if it's possible to see server-side logs during >>> agent itests, but it would probably help. I don't know why this error >>> happens. >>> >>> When I run tests successfully, I also see from times to times this >>> error, but it's distributed randomly in tests, and fortunately or not >>> (rather not), tests pass. >>> >> >> >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> >> >

> Who wants to investigate the inventory service to see if I'm right that > there is a bug here? If there is no bug in inventory, the only other option > is it's a bug in the itest code, but I checked and everything looks good to > me there. I can't see a problem in the itest code. And the fact this is > intermittent (passes sometimes, fails sometimes) tells me there is > something sinister going on in the inventory server (i.e. if it is a > problem in the test code, you would think it would always fail). > After a investigation, I don't think there is a problem in the Inventory. In a preliminar status, that was the assumption, and yes, the inventory has a flag that potentially could add an entry and theorically could make that an entry was not indexed if there was a query fast enough after the insertion. But after that PR in commons, the problem still exists, so I ran several itests of commons + agent and this race happens in a ratio of 10%, it took time to test, as itest are time consuming. Basically what happens according the logs in the failed test is the following: 1) The testXaDs datasource is added in a previous test. 2) Agent thread [Hawkular-Agent-Full-Discovery-Scan-Local DMR-1] starts a discovery in its scheduling period. 3) Agent thread [OkHttp http://127.0.0.1:8080/...] process a remove of the testXaDs and succesfully send to the inventory the delete command. 4) Agent thread [Hawkular-Agent-Full-Discovery-Scan-Local DMR-1] might still see the testXaDs resource and it is added into the inventory. 5) Test expects there is no testXaDs resource in inventory, but there is one, as there is a race condition between both threads, for some reason, when the datasource is deleted, it is not updated in the info used by the discovery task. This doesn't happen always, as it is a random scenario due some moving parts between the autodiscovery threads, and the architecture of the itest. I was analyzing why when the testXaDs is removed is still visibile by the autodiscovery thread. Tomorrow I am off and I can't work on this until thursday, so feel free if you want to continue from here.

Lucas