Starting from hibernate 5.2, method

SharedSessionContractImplementor#setFlushMode()

is deprecated in favor of

#setHibernateFlushMode()

(to avoid naming collision with JPA..).

While setFlushMode() is allowed to be called without an active transaction (an open session in enough), method setHibernateFlushMode() is not allowed to do so.
This is due to

ThreadLocalSessionContext 

proxy InvocationHandler, that allows setFlushMode() and do not allow setHibernateFlushMode() (line 334).