From your description it's not clear if the performance issue is related mostly to high CPU usage (because of writer threads) or because of high memory pressure (which will trigger longer and frequent garbage collector pauses).
For example, you might try adding to your File Appender configuration:
<param name="ImmediateFlush" value="false" />
This will avoid flusing every log statement, thus reducing the stress on CPU, but will engage more memory.
However the problem appears to me quite complex, so you will need a full CPU-Memory analisys.
(By the way, I also am not aware of any issue related to log4j file Appender like the one you have mentioned).
Hope it helps
Francesco