Hi,
let me tell you first that this is not the optimal way but you asked for API solution. I think you could achieve what you described in following steps:
1. find tasks for given process definition
taskService.createTaskQuery().processDefinitionId(defId).list()
2. Iterate over returned list of tasks and get partcipations for each
taskService.getTaskParticipations(taskId)
3. Check returned list of participations based on
participation.getGroupId()
you can filter what groups you trying to find.
That probably should do what you want but not in the best way. You could find related jira issue or create new one to extend API if it comes to querying for tasks.
HTH
Maciej