It turns out we have a nasty race condition between "wrapping up flush operations" and "shutdown and close IndexWriter" operations.
Updating the title of this issue.