The memory usage is growing beyond control bringing server to it's knees.

Could you share some metrics, heap and logs from GC? Maybe "beyond control" just means your heap is not large enough?

Either way, I'm quite confident this is not the problem: actually wiping this specific cache would cause you to have more memory issues than not wiping it - unless it really leaks but we have seen no signs of that in this specific report.

I realized today that we are technically not even using Query cache because of the above configuration

The query cache doesn't use Ehcache, as you can se in your heap dump.

Also remember: we no longer maintain Hibernate ORM 5.1.x so you should probably upgrade first.