That topic is more related with hibernate and the database that you are using:
Some of the first hits in google give you this: http://docs.jboss.org/hibernate/core/3.3/reference/en/html/transactions.html#transactions-optimistic
If the support is not in jBPM5 we can definitely help to add that.
Before going to pesimistic locking I recommend you guys to check if you really need that. We usually see that the best approach is to implement a retry mechanism for multiple reasons:
1) JMS will do it for you
2) to handle low level problems like: server is down, database is down, network failure, etc a retrying mechanism will help your administrators to easily solve the problems
3) You will not add low lever error handling in your processes.
Cheers