what could be the issue is that task service will load known groups and "enhance" result from your callback. I believe this is a bug as there is a property of usergroupcallback manager that should limit this behavior but unfortunately manager will not return this property, see here.
Could you open a jira for it and would be perfect if you could provide a test case for that too.
Cheers