The Travis cache behaves this way:
- try to get a cache specific to the PR
- then try to get a cache for master
- if none found, start without cache
The ORM Travis build was configured to not build the branches so master was
never build thus each PR starts with an empty cache (but subsequent pushes
to the same PR will use the PR cache).
I enabled the build for branches (Settings > Build pushed branches) so that
you have a cache for master. This should speed up the builds significantly.