I think you're right. In our use cases we never need an "aggressive close", as for example a restarting SearchFactory is only meant to be incremental (so to support adding new entities and indexes, but not removing them), so this implies that if a Service is ever started, it's probably going to be needed again.
So it would make sense when a closing Service reaches the "0 users" counter to be set in a state "available for close" but only actually close it on shutdown of the SearchFactory.

The exception would be Services which are only temporarily used during boostrap; these would not free their memory but I can't think of a practical example, if any the memory usage would be negligible.