1:35 p.m.
Who wants to investigate the inventory service to see if I'm right that
there is a bug here? If there is no bug in inventory, the only other option
is it's a bug in the itest code, but I checked and everything looks good to
me there. I can't see a problem in the itest code. And the fact this is
intermittent (passes sometimes, fails sometimes) tells me there is
something sinister going on in the inventory server (i.e. if it is a
problem in the test code, you would think it would always fail).
After a investigation, I don't think there is a problem in the Inventory.
In a preliminar status, that was the assumption, and yes, the inventory has
a flag that potentially could add an entry and theorically could make that
an entry was not indexed if there was a query fast enough after the
insertion.
But after that PR in commons, the problem still exists, so I ran several
itests of commons + agent and this race happens in a ratio of 10%, it took
time to test, as itest are time consuming.
Basically what happens according the logs in the failed test is the
following:
1) The testXaDs datasource is added in a previous test.
2) Agent thread [Hawkular-Agent-Full-Discovery-Scan-Local DMR-1] starts a
discovery in its scheduling period.
3) Agent thread [OkHttp http://127.0.0.1:8080/...] process a remove of the
testXaDs and succesfully send to the inventory the delete command.
4) Agent thread [Hawkular-Agent-Full-Discovery-Scan-Local DMR-1] might
still see the testXaDs resource and it is added into the inventory.
5) Test expects there is no testXaDs resource in inventory, but there is
one, as there is a race condition between both threads, for some reason,
when the datasource is deleted, it is not updated in the info used by the
discovery task.
This doesn't happen always, as it is a random scenario due some moving
parts between the autodiscovery threads, and the architecture of the itest.
I was analyzing why when the testXaDs is removed is still visibile by the
autodiscovery thread.
Tomorrow I am off and I can't work on this until thursday, so feel free if
you want to continue from here.
Lucas