The new IndexWriter in Lucene 4 uses a limited set of affinity tokens to hand to each thread.
The default is only 8, which gets exhausted quite easily in the way we use the IndexWriter; also there seems to be no high penalty in proposing an higher value other than a slight memory cost.

I'm proposing a default of 32 and to make this a configurable setting.

writerConfig.setMaxThreadStates( 32 );