After some research I'm not certain I can get the behavior I need.
I was able to isolate transactions for a specific use case where I know the invoking client does not have a transaction:
invoke Task [SSB with @TransactionAttribute(NOT_SUPPORTED)]
invoke Search [SSB with @TransactionAttribute(REQUIRED)], look up entities to operate on
for each entity
invoke Operation [SSB with @TransactionAttribute(REQUIRED)], perform operation
This does execute the Search and Operation methods in separate transactions and allow run times greater than the default timeout for the Task method.
However, if the client invoking Task is not an MBean, but rather another business component, it's transaction will be suspended while the Task runs and will timeout after 5 minutes.
So am I understanding this correctly?
You can gain some control over transaction timeouts, but only if you know your clients are not using a transaction.