[infinispan-issues] [JBoss JIRA] (ISPN-10174) Adjusting thread pools for container environments
Galder Zamarreño (Jira)
issues at jboss.org
Fri May 10 08:14:00 EDT 2019
[ https://issues.jboss.org/browse/ISPN-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731915#comment-13731915 ]
Galder Zamarreño commented on ISPN-10174:
-----------------------------------------
I've done some testing but the results are not very conclusive:
In a server with 6 off-heap caches and default thread pool sizes, a container uses between 332mb-340mb out of the box. After 10-20 mins this goes up to 410-420mb and stays stable. That's without any data or any incoming requests and with a limit of 512mb. Native memory tracking shows:
{code}
Native Memory Tracking:
Total: reserved=1953934KB -630KB, committed=493618KB +678KB
- Java Heap (reserved=262144KB, committed=68096KB -1024KB)
(mmap: reserved=262144KB, committed=68096KB -1024KB)
- Class (reserved=1131296KB +18KB, committed=93344KB +530KB)
(classes #14440 +3)
(malloc=4896KB +18KB #23987 +364)
(mmap: reserved=1126400KB, committed=88448KB +512KB)
- Thread (reserved=116682KB -1033KB, committed=116682KB -1033KB)
(thread #113 -1)
(stack: reserved=116136KB -1028KB, committed=116136KB -1028KB)
(malloc=381KB -3KB #563 -5)
(arena=164KB -1 #223 -2)
- Code (reserved=254153KB +374KB, committed=26461KB +2194KB)
(malloc=4553KB +374KB #7814 +517)
(mmap: reserved=249600KB, committed=21908KB +1820KB)
- GC (reserved=11909KB, committed=11285KB)
(malloc=2325KB #525 +1)
(mmap: reserved=9584KB, committed=8960KB)
- Compiler (reserved=257KB -4KB, committed=257KB -4KB)
(malloc=126KB -4KB #675 +6)
(arena=131KB #5)
- Internal (reserved=154058KB, committed=154058KB)
(malloc=154026KB #27664 -2)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=19609KB, committed=19609KB)
(malloc=16788KB #170432)
(arena=2821KB #1)
- Native Memory Tracking (reserved=3646KB +14KB, committed=3646KB +14KB)
(malloc=18KB #205 -1)
(tracking overhead=3628KB +14KB)
- Arena Chunk (reserved=180KB +1KB, committed=180KB +1KB)
(malloc=180KB +1KB)
{code}
The interesting thing here is that the number of threads for which memory is allocated in an unused server is 113. So, not all threads are taking space, and as Dan suggested in a separate chat, only a small fraction of that might be in use, but it's committed anyway.
Then I tried to reduce the number of threads with these changes:
* HotRod-hotrod-internal-ServerHandler: core=16 max=16
* remote-thread: core=4 max=4
* transport-thread: core=4 max=4
* async-thread: core=4 max=4
With still the same off-heap caches, the container took 320-330mb to start with and then grew to 330-410mb. Looking at the native memory tracking:
{code}
Total: reserved=1930181KB +514KB, committed=468493KB +3894KB
- Java Heap (reserved=262144KB, committed=67072KB -512KB)
(mmap: reserved=262144KB, committed=67072KB -512KB)
- Class (reserved=1131354KB +69KB, committed=94170KB +581KB)
(classes #14684 +8)
(malloc=4954KB +69KB #24153 +866)
(mmap: reserved=1126400KB, committed=89216KB +512KB)
- Thread (reserved=76411KB, committed=76411KB)
(thread #74)
(stack: reserved=76044KB, committed=76044KB)
(malloc=248KB #368)
(arena=119KB #145)
- Code (reserved=253888KB +727KB, committed=25088KB +4107KB)
(malloc=4288KB +727KB #7277 +986)
(mmap: reserved=249600KB, committed=20800KB +3380KB)
- GC (reserved=11909KB, committed=11277KB)
(malloc=2325KB #525 +6)
(mmap: reserved=9584KB, committed=8952KB)
- Compiler (reserved=233KB -9KB, committed=233KB -9KB)
(malloc=102KB -9KB #630 -11)
(arena=131KB #5)
- Internal (reserved=170643KB +1KB, committed=170643KB +1KB)
(malloc=170611KB +1KB #28177 +8)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=19750KB +16KB, committed=19750KB +16KB)
(malloc=16929KB +16KB #172444 +9)
(arena=2821KB #1)
- Native Memory Tracking (reserved=3670KB +29KB, committed=3670KB +29KB)
(malloc=14KB #166)
(tracking overhead=3656KB +29KB)
- Arena Chunk (reserved=179KB -319KB, committed=179KB -319KB)
(malloc=179KB -319KB)
{code}
We see the number of threads has gone down to 74, but that didn't make much difference overall.
Next I removed the 6 off heap caches and that made the containers start with 260-280mb, and after 20 minutes it settled in 350-360. That would seem each off heap caches takes ~10mb of memory empty. There is certain allocation happening there for buckets, so that's expected AFAIK.
The full effect of reducing the thread pools can't been seen like this. A more realistic scenario would be a soak test where we define the number of concurrent requests a server should be able to handle and see if with a constant load, the server can handle that. For example, say that with 3 node cluster of 160 Hot Rod worker threads, you can (hyphotetically) handle 3*160 client threads, each doing a put/get on a given key. Assuming the that data stored does not increase over time (each thread read/writes same key), the container should be able to handle the load without being killed by Kubernetes.
> Adjusting thread pools for container environments
> -------------------------------------------------
>
> Key: ISPN-10174
> URL: https://issues.jboss.org/browse/ISPN-10174
> Project: Infinispan
> Issue Type: Enhancement
> Components: Cloud
> Affects Versions: 10.0.0.Beta3, 9.4.13.Final
> Reporter: Galder Zamarreño
> Priority: Major
> Labels: rhdemo-2019
>
> Default thread pool values in Infinispan Server cloud.xml can make containers be killed if all of them are in use. The main defaults are:
> * HotRod-hotrod-internal-ServerHandler: core=160 max=160
> * remote-thread: core=25 max=25
> * transport-thread: core=25 max=25
> * async-thread: core=25 max=25
> * jgroups: core=0 max=200
> * jgroups-int: core=0 max=16
> * stateTransferExecutor-thread: core=1 max=60
> * add-listener-thread: core=0 max=10
> * REST-rest-ServerHandler: core=1 max=1
> * DefaultExecutorService: core=1 max=1
> * notification-thread: core=1 max=1
> The total number of core threads is 239, and if the system is under load, the threads alone can take ~239mb of native memory. That's before the heap and other parameters are counted. Our defaults are 0.5 CPU and 512mb.
> This thread pools should be trimmed, since if used at full capacity, 0.5 CPU won't be able to do much with ~200+ threads.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
More information about the infinispan-issues
mailing list