Who wants to investigate the inventory service to see if I'm right that there is a bug here? If there is no bug in inventory, the only other option is it's a bug in the itest code, but I checked and everything looks good to me there. I can't see a problem in the itest code. And the fact this is intermittent (passes sometimes, fails sometimes) tells me there is something sinister going on in the inventory server (i.e. if it is a problem in the test code, you would think it would always fail).After a investigation, I don't think there is a problem in the Inventory.In a preliminar status, that was the assumption, and yes, the inventory has a flag that potentially could add an entry and theorically could make that an entry was not indexed if there was a query fast enough after the insertion.But after that PR in commons, the problem still exists, so I ran several itests of commons + agent and this race happens in a ratio of 10%, it took time to test, as itest are time consuming.Basically what happens according the logs in the failed test is the following:1) The testXaDs datasource is added in a previous test.2) Agent thread [Hawkular-Agent-Full-Discovery-Scan-Local DMR-1] starts a discovery in its scheduling period. 3) Agent thread [OkHttp http://127.0.0.1:8080/...] process a remove of the testXaDs and succesfully send to the inventory the delete command.4) Agent thread [Hawkular-Agent-Full-Discovery-Scan-Local DMR-1] might still see the testXaDs resource and it is added into the inventory. 5) Test expects there is no testXaDs resource in inventory, but there is one, as there is a race condition between both threads, for some reason, when the datasource is deleted, it is not updated in the info used by the discovery task.This doesn't happen always, as it is a random scenario due some moving parts between the autodiscovery threads, and the architecture of the itest.I was analyzing why when the testXaDs is removed is still visibile by the autodiscovery thread.Tomorrow I am off and I can't work on this until thursday, so feel free if you want to continue from here.
Lucas