You should provide implementation of UserGroupCallback interface, dedicated to your custom user system.
Once you have the implementation ready you should provide it to the jvm that is starting the task server with parameter "jbpm.usergroup.callback", for instance -Djbpm.usergroup.callback=com.test.ldap.LdapUserGroupCallback
HTH