Exception handling changed in 5.2 when hibernate-entitymanager was merged into hibernate-core. This change is documented in the 5.2 migration guide. [1]

Here is the relevant text:

"org.hibernate.HibernateException now extends javax.persistence.PersistenceExceptions. Hibernate methods that "override" methods from their JPA counterparts now will also throw various JDK defined RuntimeExceptions (such as IllegalArgumentException, IllegalStateException, etc) as required by the JPA contract."

While digging into this, I see that a HibernateException thrown when using 5.1 may, in 5.3, be unwrapped, or may be wrapped by a PersistenceException (or a subclass), IllegalArgumentException, or IllegalStateException, depending on the particular operation being performed when the HibernateException is thrown.

The reason why the exceptions may be wrapped or unwrapped is because Hibernate converts exceptions (via AbstractSharedSessionContract#exceptionConverter) for JPA operations which may wrap the HibernateException or throw a different exception. Hibernate does not convert exceptions from strictly "native" operations, so those exceptions remain unwrapped.

Here are a couple of examples to illustrate:

1) A HibernateException (org.hibernate.TransientObjectException) was thrown in 5.1. In 5.3, the same condition can result in org.hibernate.TransientObjectException that is either unwrapped, or wrapped by IllegalStateException. I've pushed a test to my fork to illustrate. [2]

Thrown during Session#save, #saveOrUpdate
In 5.1: org.hibernate.TransientObjectException (unwrapped)
In 5.3: org.hibernate.TransientObjectException (unwrapped)

Thrown during Session#persist, #merge, #flush
In 5.1, org.hibernate.TransientObjectException (unwrapped)
In 5.3, org.hibernate.TransientObjectException wrapped by IllegalStateException

2) A HibernateException thrown when using 5.1 may be wrapped by a PersistenceException, even though HibernateException already extends PersistenceException.

For example, see TransactionTimeoutTest#testTransactionTimeoutFailure:
https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/test/java/org/hibernate/test/tm/TransactionTimeoutTest.java#L60-L82

The exception thrown by the test indicates that the transaction timed out. This exception is important enough that an application might retry the operation, or at least log for future investigation.

Thrown during Session#persist (or when changed to use Session#merge or Session#flush):
In 5.1: org.hibernate.TransactionException (unwrapped)
In 5.3: org.hibernate.TransactionException wrapped by javax.persistence.PersistenceException

Thrown if the test is changed to use Session#save or #saveOrUpdate instead:
In 5.1: org.hibernate.TransactionException (unwrapped)
In 5.3: org.hibernate.TransactionException (unwrapped)

Similarly, by adding some logging, I see that HibernateException objects can be wrapped by PersistenceException when running the 5.3 hibernate-core unit tests. Depending on the context, I see that some of the following exceptions can be wrapped or unwrapped.

org.hibernate.exception.ConstraintViolationException
org.hibernate.exception.DataException
org.hibernate.exception.GenericJDBCException
org.hibernate.exception.SQLGrammarException
org.hibernate.id.IdentifierGenerationException
org.hibernate.loader.custom.NonUniqueDiscoveredSqlAliasException
org.hibernate.PersistentObjectException
org.hibernate.PropertyAccessException
org.hibernate.PropertyValueException
org.hibernate.TransactionException

You can see an example using org.hibernate.exception.ConstraintViolationException at [3].

In order to deal with these differences, an application could change the following (which was appropriate for 5.1):

try

catch (PersistenceException | IllegalStateException | IllegalArgumentException ex) {
if ( HibernateException.class.isInstance( ex ) )

else if ( HibernateException.class.isInstance( ex.getCause() ) )

}

IMO, it's a little clumsy having to deal with both wrapped and unwrapped exceptions. It would be better if exceptions were consistently wrapped, or consistently unwrapped.

To deal with this, hibernate.exception_converter could be added with the following values:

jpa - default for JPA applications (org.hibernate.internal.ExceptionConverterImpl)
native - default for native applications that does not wrap HibernateException

Should there actually be 2 "native" implementations:
1) native-5.2, conform to 5.2 exceptions (wrapped and unwrapped);
2) native-5.1, conform to 5.1 exceptions (unwrapped)?

[1] https://github.com/hibernate/hibernate-orm/blob/5.2/migration-guide.adoc
[2] https://github.com/gbadner/hibernate-core/blob/exception-compatibility/orm/hibernate-orm-5/src/test/java/org/hibernate/bugs/ORMTransientObjectExceptionUnitTestCase.java
[3] https://github.com/gbadner/hibernate-core/blob/exception-compatibility/orm/hibernate-orm-5/src/test/java/org/hibernate/bugs/ORMConstraintViolationExceptionUnitTestCase.java