I figured it out from the mvel file in the human task jar file. This is the command:
Operation.Release
: [ new OperationCommand().{
status = [ Status.Reserved, Status.InProgress ],
allowed = [Allowed.Owner, Allowed.BusinessAdministrator ],
setNewOwnerToNull = true,
newStatus = Status.Ready
} ],
So if the current status is reserved or inprogress and the user is the owner or a business administrator the owner is set to null and the status is set to ready so it can be claimed.