[infinispan-issues] [JBoss JIRA] (ISPN-8141) Docker container with memory limits dies after adding RocksDB
Sebastian Łaskawiec (JIRA)
issues at jboss.org
Tue Aug 1 06:28:00 EDT 2017
Sebastian Łaskawiec created ISPN-8141:
-----------------------------------------
Summary: Docker container with memory limits dies after adding RocksDB
Key: ISPN-8141
URL: https://issues.jboss.org/browse/ISPN-8141
Project: Infinispan
Issue Type: Bug
Components: Cloud Integrations, Loaders and Stores
Affects Versions: 9.1.0.Final
Reporter: Sebastian Łaskawiec
Assignee: Sebastian Łaskawiec
When testing Infinispan Persistent Docker container (with RocksDB) we encountered very interesting problem. The container went up to 100% of memory limit and then was killed by the OOM killer. After removing RocksDB Cache Store the problem went away.
After looking at {{dmsg}} it turned out that the container was killed because of the OOM. The interesting thing is that the OOM allows to process to keep using 100% of the memory but only for a short amount of time.
The second thing we investigated are GC logs:
{code}
2017-08-01T10:06:33.421+0000: [GC (Allocation Failure)
Desired survivor size 35651584 bytes, new threshold 2 (max 15)
[PSYoungGen: 38624K->3328K(70144K)] 95950K->63878K(280576K), 0.0089506 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2017-08-01T10:06:33.431+0000: Total time for which application threads were stopped: 0.0117935 seconds, Stopping threads took: 0.0002661 seconds
2017-08-01T10:06:36.797+0000: Total time for which application threads were stopped: 0.1213646 seconds, Stopping threads took: 0.0153583 seconds
2017-08-01T10:06:37.425+0000: Total time for which application threads were stopped: 0.3444540 seconds, Stopping threads took: 0.0092094 seconds
2017-08-01T10:06:40.719+0000: Total time for which application threads were stopped: 0.2126636 seconds, Stopping threads took: 0.0643458 seconds
2017-08-01T10:06:42.663+0000: Total time for which application threads were stopped: 0.9234535 seconds, Stopping threads took: 0.0554174 seconds
TLAB: gc thread: 0x00007fc6b402c000 [id: 1038] desired_size: 4KB slow allocs: 6 refill waste: 64B alloc: 0.00609 139KB refills: 12 waste 3.9% gc: 1464B slow: 528B fast: 72B
TLAB: gc thread: 0x00007fc6b4021800 [id: 1037] desired_size: 4KB slow allocs: 6 refill waste: 64B alloc: 0.00609 139KB refills: 13 waste 7.5% gc: 3656B slow: 560B fast: 72B
TLAB: gc thread: 0x00007fc6f4088000 [id: 975] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00118 27KB refills: 14 waste 4.5% gc: 928B slow: 312B fast: 56B
TLAB: gc thread: 0x00007fc6ec019800 [id: 788] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00431 98KB refills: 32 waste 2.0% gc: 960B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6d0016800 [id: 787] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00432 98KB refills: 30 waste 2.4% gc: 1160B slow: 1112B fast: 0B
TLAB: gc thread: 0x00007fc70c039800 [id: 786] desired_size: 2KB slow allocs: 5 refill waste: 40B alloc: 0.00419 95KB refills: 35 waste 2.1% gc: 1024B slow: 1192B fast: 0B
TLAB: gc thread: 0x00007fc6b4016800 [id: 785] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00427 97KB refills: 31 waste 1.5% gc: 264B slow: 1144B fast: 0B
TLAB: gc thread: 0x00007fc69c072000 [id: 784] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00432 98KB refills: 33 waste 1.3% gc: 256B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc6c0022000 [id: 783] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00449 102KB refills: 23 waste 2.4% gc: 1000B slow: 808B fast: 0B
TLAB: gc thread: 0x00007fc6f4079800 [id: 782] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00427 97KB refills: 28 waste 2.9% gc: 1496B slow: 968B fast: 0B
TLAB: gc thread: 0x00007fc6bc047800 [id: 781] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00426 97KB refills: 30 waste 3.5% gc: 2192B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc6c800d000 [id: 780] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00442 101KB refills: 26 waste 2.2% gc: 1064B slow: 792B fast: 0B
TLAB: gc thread: 0x00007fc6b0018000 [id: 779] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00426 97KB refills: 29 waste 3.2% gc: 1832B slow: 1024B fast: 0B
TLAB: gc thread: 0x00007fc6a402e800 [id: 778] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00440 100KB refills: 34 waste 3.1% gc: 2328B slow: 976B fast: 0B
TLAB: gc thread: 0x00007fc6b8062000 [id: 777] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00435 99KB refills: 31 waste 2.9% gc: 1744B slow: 1056B fast: 0B
TLAB: gc thread: 0x00007fc694016800 [id: 776] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00427 97KB refills: 32 waste 4.2% gc: 2992B slow: 1160B fast: 0B
TLAB: gc thread: 0x00007fc6cc016000 [id: 775] desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00421 96KB refills: 30 waste 2.5% gc: 1312B slow: 984B fast: 0B
TLAB: gc thread: 0x00007fc6c4069800 [id: 774] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00431 98KB refills: 28 waste 1.6% gc: 480B slow: 952B fast: 0B
TLAB: gc thread: 0x0000000001823800 [id: 773] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00447 102KB refills: 31 waste 3.1% gc: 2152B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b4014800 [id: 772] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00444 101KB refills: 33 waste 2.5% gc: 1472B slow: 1152B fast: 0B
TLAB: gc thread: 0x00007fc70c0ab800 [id: 771] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00451 103KB refills: 31 waste 1.1% gc: 40B slow: 1096B fast: 0B
TLAB: gc thread: 0x00007fc694014800 [id: 770] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00436 99KB refills: 30 waste 3.0% gc: 1808B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6d0014800 [id: 769] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00466 106KB refills: 29 waste 3.4% gc: 2456B slow: 904B fast: 0B
TLAB: gc thread: 0x00007fc69c070000 [id: 768] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00430 98KB refills: 32 waste 2.2% gc: 920B slow: 1272B fast: 0B
TLAB: gc thread: 0x00007fc6c4068000 [id: 767] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00426 97KB refills: 31 waste 1.3% gc: 136B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6ec017000 [id: 766] desired_size: 2KB slow allocs: 8 refill waste: 40B alloc: 0.00414 94KB refills: 32 waste 3.5% gc: 2336B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc694012800 [id: 765] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00429 98KB refills: 28 waste 1.3% gc: 160B slow: 936B fast: 0B
TLAB: gc thread: 0x00007fc6bc046000 [id: 764] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00466 106KB refills: 31 waste 3.4% gc: 2504B slow: 1088B fast: 0B
TLAB: gc thread: 0x00007fc6a402d000 [id: 763] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00442 101KB refills: 30 waste 3.7% gc: 2592B slow: 920B fast: 0B
TLAB: gc thread: 0x00007fc6b8060000 [id: 762] desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00410 93KB refills: 36 waste 1.2% gc: 32B slow: 1272B fast: 0B
TLAB: gc thread: 0x00007fc6f4077000 [id: 761] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00446 101KB refills: 28 waste 1.2% gc: 192B slow: 872B fast: 0B
TLAB: gc thread: 0x00007fc6d0012800 [id: 760] desired_size: 3KB slow allocs: 3 refill waste: 48B alloc: 0.00435 99KB refills: 34 waste 2.7% gc: 1856B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc694011000 [id: 759] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00441 100KB refills: 30 waste 3.2% gc: 2120B slow: 944B fast: 0B
TLAB: gc thread: 0x00007fc70c0aa000 [id: 758] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00426 97KB refills: 30 waste 1.2% gc: 136B slow: 960B fast: 0B
TLAB: gc thread: 0x00007fc6c4066000 [id: 757] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00439 100KB refills: 27 waste 2.0% gc: 928B slow: 816B fast: 0B
TLAB: gc thread: 0x00007fc69c06e000 [id: 756] desired_size: 3KB slow allocs: 13 refill waste: 48B alloc: 0.00434 99KB refills: 31 waste 1.2% gc: 120B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6c001f800 [id: 755] desired_size: 2KB slow allocs: 4 refill waste: 40B alloc: 0.00410 93KB refills: 32 waste 1.3% gc: 280B slow: 992B fast: 0B
TLAB: gc thread: 0x00007fc6c800b000 [id: 754] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00442 101KB refills: 32 waste 2.1% gc: 992B slow: 1192B fast: 0B
TLAB: gc thread: 0x00007fc6b4012800 [id: 753] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00433 98KB refills: 29 waste 1.6% gc: 504B slow: 984B fast: 0B
TLAB: gc thread: 0x00007fc6ec03c000 [id: 752] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00445 101KB refills: 30 waste 1.5% gc: 520B slow: 920B fast: 0B
TLAB: gc thread: 0x00007fc6cc013800 [id: 751] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00425 97KB refills: 33 waste 1.4% gc: 376B slow: 1024B fast: 0B
TLAB: gc thread: 0x00007fc6b0016000 [id: 750] desired_size: 3KB slow allocs: 12 refill waste: 48B alloc: 0.00431 98KB refills: 35 waste 1.1% gc: 24B slow: 1152B fast: 0B
TLAB: gc thread: 0x00007fc70c0a8000 [id: 749] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00442 101KB refills: 30 waste 1.5% gc: 448B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6c001d800 [id: 748] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00449 102KB refills: 28 waste 2.4% gc: 1120B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6c8009000 [id: 747] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00455 104KB refills: 30 waste 1.5% gc: 648B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc69c06c000 [id: 746] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00427 97KB refills: 37 waste 1.9% gc: 928B slow: 1288B fast: 0B
TLAB: gc thread: 0x00007fc69400e800 [id: 745] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00429 98KB refills: 36 waste 3.0% gc: 2088B slow: 1288B fast: 0B
TLAB: gc thread: 0x00007fc6c4077800 [id: 744] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00432 98KB refills: 31 waste 2.3% gc: 1336B slow: 904B fast: 0B
TLAB: gc thread: 0x0000000001821800 [id: 743] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00435 99KB refills: 26 waste 1.5% gc: 96B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc6d0010800 [id: 742] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00446 101KB refills: 34 waste 3.4% gc: 2544B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc69400d000 [id: 741] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00428 97KB refills: 32 waste 1.5% gc: 344B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc6c001b800 [id: 740] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00459 104KB refills: 32 waste 1.4% gc: 152B slow: 1304B fast: 0B
TLAB: gc thread: 0x00007fc70c0a6000 [id: 739] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00454 103KB refills: 31 waste 1.2% gc: 248B slow: 960B fast: 0B
TLAB: gc thread: 0x00007fc6b0013800 [id: 738] desired_size: 2KB slow allocs: 9 refill waste: 40B alloc: 0.00422 96KB refills: 34 waste 4.0% gc: 3016B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc69c06a000 [id: 737] desired_size: 2KB slow allocs: 12 refill waste: 40B alloc: 0.00423 96KB refills: 33 waste 1.2% gc: 208B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc6cc011800 [id: 736] desired_size: 2KB slow allocs: 2 refill waste: 40B alloc: 0.00407 93KB refills: 29 waste 3.3% gc: 1920B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6a402a800 [id: 735] desired_size: 2KB slow allocs: 8 refill waste: 40B alloc: 0.00415 94KB refills: 31 waste 3.7% gc: 2400B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc6c8007000 [id: 734] desired_size: 3KB slow allocs: 3 refill waste: 48B alloc: 0.00451 103KB refills: 31 waste 2.7% gc: 1640B slow: 1056B fast: 0B
TLAB: gc thread: 0x000000000181f800 [id: 733] desired_size: 3KB slow allocs: 2 refill waste: 48B alloc: 0.00426 97KB refills: 24 waste 2.1% gc: 696B slow: 848B fast: 0B
TLAB: gc thread: 0x00007fc6d000e800 [id: 732] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00442 101KB refills: 33 waste 2.6% gc: 1544B slow: 1184B fast: 0B
TLAB: gc thread: 0x00007fc6bc043800 [id: 731] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00434 99KB refills: 34 waste 1.2% gc: 200B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc6c4075800 [id: 730] desired_size: 3KB slow allocs: 3 refill waste: 48B alloc: 0.00449 102KB refills: 32 waste 2.5% gc: 1592B slow: 976B fast: 0B
TLAB: gc thread: 0x00007fc6f4075000 [id: 729] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00437 99KB refills: 31 waste 4.3% gc: 3064B slow: 1184B fast: 0B
TLAB: gc thread: 0x00007fc69400b000 [id: 728] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00443 101KB refills: 27 waste 2.9% gc: 1552B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6b805d800 [id: 727] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00461 105KB refills: 28 waste 1.7% gc: 632B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6b0012000 [id: 726] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00435 99KB refills: 24 waste 3.9% gc: 2192B slow: 760B fast: 0B
TLAB: gc thread: 0x00007fc6c8070800 [id: 725] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00436 99KB refills: 34 waste 4.1% gc: 3152B slow: 1272B fast: 0B
TLAB: gc thread: 0x00007fc6c4073800 [id: 724] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00441 100KB refills: 27 waste 2.4% gc: 1216B slow: 888B fast: 0B
TLAB: gc thread: 0x00007fc6cc00f800 [id: 723] desired_size: 2KB slow allocs: 3 refill waste: 40B alloc: 0.00424 96KB refills: 30 waste 2.4% gc: 1240B slow: 952B fast: 0B
TLAB: gc thread: 0x00007fc6bc041800 [id: 722] desired_size: 2KB slow allocs: 5 refill waste: 40B alloc: 0.00402 91KB refills: 37 waste 3.8% gc: 2808B slow: 1256B fast: 0B
TLAB: gc thread: 0x00007fc6ec039800 [id: 721] desired_size: 3KB slow allocs: 3 refill waste: 48B alloc: 0.00449 102KB refills: 31 waste 2.0% gc: 1184B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6c001a000 [id: 720] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00429 98KB refills: 28 waste 3.6% gc: 2208B slow: 912B fast: 0B
TLAB: gc thread: 0x00007fc6cc00d800 [id: 719] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00438 100KB refills: 28 waste 1.7% gc: 656B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc6f4084800 [id: 718] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00442 101KB refills: 29 waste 3.5% gc: 2320B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6c806e800 [id: 717] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00450 102KB refills: 30 waste 2.9% gc: 1936B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6b0010000 [id: 716] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00425 97KB refills: 37 waste 2.4% gc: 1368B slow: 1408B fast: 0B
TLAB: gc thread: 0x00007fc6b4010800 [id: 715] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00463 105KB refills: 28 waste 2.3% gc: 1104B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6d000d000 [id: 714] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00460 105KB refills: 34 waste 1.8% gc: 864B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6cc00b800 [id: 713] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00435 99KB refills: 26 waste 4.3% gc: 2488B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b805b800 [id: 712] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00448 102KB refills: 34 waste 1.6% gc: 424B slow: 1320B fast: 0B
TLAB: gc thread: 0x00007fc6c806c800 [id: 711] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00458 104KB refills: 25 waste 3.0% gc: 1600B slow: 920B fast: 0B
TLAB: gc thread: 0x00007fc6a4028800 [id: 710] desired_size: 3KB slow allocs: 13 refill waste: 48B alloc: 0.00453 103KB refills: 37 waste 2.9% gc: 2256B slow: 1256B fast: 0B
TLAB: gc thread: 0x00007fc69c068000 [id: 709] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00451 103KB refills: 29 waste 2.1% gc: 944B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6ec038000 [id: 708] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00447 102KB refills: 27 waste 3.5% gc: 2208B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6c806b000 [id: 707] desired_size: 2KB slow allocs: 6 refill waste: 40B alloc: 0.00419 95KB refills: 27 waste 1.5% gc: 384B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc6bc03f800 [id: 706] desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00410 93KB refills: 31 waste 2.3% gc: 1136B slow: 976B fast: 0B
TLAB: gc thread: 0x000000000181d800 [id: 705] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00457 104KB refills: 31 waste 2.0% gc: 984B slow: 1024B fast: 0B
TLAB: gc thread: 0x00007fc6b000d800 [id: 704] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00444 101KB refills: 27 waste 3.7% gc: 2160B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc70c065000 [id: 703] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00430 98KB refills: 30 waste 2.0% gc: 816B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc6f4082800 [id: 702] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00435 99KB refills: 29 waste 3.0% gc: 1608B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6d000b000 [id: 701] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00441 100KB refills: 31 waste 4.3% gc: 3160B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc694009000 [id: 700] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00436 99KB refills: 32 waste 4.0% gc: 2944B slow: 1048B fast: 0B
TLAB: gc thread: 0x000000000181c000 [id: 699] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00440 100KB refills: 30 waste 4.2% gc: 3048B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6ec035000 [id: 698] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00435 99KB refills: 31 waste 3.3% gc: 2208B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b000c000 [id: 697] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00460 105KB refills: 30 waste 4.1% gc: 2952B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6b400e800 [id: 696] desired_size: 2KB slow allocs: 4 refill waste: 40B alloc: 0.00419 95KB refills: 27 waste 4.3% gc: 2712B slow: 768B fast: 0B
TLAB: gc thread: 0x00007fc6bc03d000 [id: 695] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00445 101KB refills: 32 waste 3.2% gc: 2200B slow: 1080B fast: 0B
TLAB: gc thread: 0x00007fc6c0018000 [id: 694] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00439 100KB refills: 34 waste 1.8% gc: 824B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6b8059800 [id: 693] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00441 100KB refills: 30 waste 1.9% gc: 808B slow: 1056B fast: 0B
TLAB: gc thread: 0x00007fc6c4071000 [id: 692] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00431 98KB refills: 32 waste 2.4% gc: 1288B slow: 1112B fast: 0B
TLAB: gc thread: 0x000000000181a000 [id: 691] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00429 98KB refills: 31 waste 1.9% gc: 712B slow: 1096B fast: 0B
TLAB: gc thread: 0x00007fc6b400c000 [id: 690] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00445 101KB refills: 27 waste 1.3% gc: 216B slow: 936B fast: 0B
TLAB: gc thread: 0x00007fc694007000 [id: 689] desired_size: 2KB slow allocs: 4 refill waste: 40B alloc: 0.00413 94KB refills: 30 waste 2.5% gc: 1368B slow: 864B fast: 0B
TLAB: gc thread: 0x00007fc6a4027000 [id: 688] desired_size: 2KB slow allocs: 4 refill waste: 40B alloc: 0.00421 96KB refills: 33 waste 2.6% gc: 1616B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc69c066000 [id: 687] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00445 101KB refills: 32 waste 2.5% gc: 1568B slow: 1032B fast: 0B
TLAB: gc thread: 0x00007fc70c063000 [id: 686] desired_size: 2KB slow allocs: 6 refill waste: 40B alloc: 0.00422 96KB refills: 32 waste 2.9% gc: 1736B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6d0008800 [id: 685] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00433 98KB refills: 29 waste 2.5% gc: 1296B slow: 960B fast: 0B
TLAB: gc thread: 0x00007fc6bc03b000 [id: 684] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00429 98KB refills: 33 waste 2.6% gc: 1456B slow: 1168B fast: 0B
TLAB: gc thread: 0x00007fc6c8060800 [id: 683] desired_size: 2KB slow allocs: 5 refill waste: 40B alloc: 0.00418 95KB refills: 35 waste 2.1% gc: 1144B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc6b0009800 [id: 682] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00427 97KB refills: 30 waste 4.3% gc: 3056B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6cc009000 [id: 681] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00441 100KB refills: 32 waste 1.5% gc: 640B slow: 912B fast: 0B
TLAB: gc thread: 0x00007fc6b8057000 [id: 680] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00440 100KB refills: 31 waste 1.7% gc: 656B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6bc039000 [id: 679] desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00419 95KB refills: 33 waste 2.0% gc: 1024B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6c0015800 [id: 678] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00427 97KB refills: 31 waste 3.6% gc: 2360B slow: 1056B fast: 0B
TLAB: gc thread: 0x00007fc6ec02e000 [id: 677] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00454 103KB refills: 28 waste 3.0% gc: 1856B slow: 936B fast: 0B
TLAB: gc thread: 0x00007fc6c406f800 [id: 676] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00435 99KB refills: 36 waste 3.7% gc: 3080B slow: 1160B fast: 0B
TLAB: gc thread: 0x00007fc6d0006800 [id: 675] desired_size: 3KB slow allocs: 11 refill waste: 48B alloc: 0.00454 103KB refills: 33 waste 3.9% gc: 3088B slow: 1144B fast: 0B
TLAB: gc thread: 0x00007fc6c0014000 [id: 674] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00442 101KB refills: 28 waste 1.1% gc: 136B slow: 872B fast: 0B
TLAB: gc thread: 0x00007fc6bc037800 [id: 673] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00455 103KB refills: 34 waste 2.4% gc: 1520B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6b8055800 [id: 672] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00427 97KB refills: 31 waste 1.5% gc: 296B slow: 1096B fast: 0B
TLAB: gc thread: 0x0000000001818000 [id: 671] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00435 99KB refills: 34 waste 1.8% gc: 672B slow: 1296B fast: 0B
TLAB: gc thread: 0x00007fc6b0007800 [id: 670] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00455 103KB refills: 27 waste 2.6% gc: 1488B slow: 856B fast: 0B
TLAB: gc thread: 0x00007fc6a4024800 [id: 669] desired_size: 3KB slow allocs: 10 refill waste: 48B alloc: 0.00449 102KB refills: 31 waste 1.1% gc: 64B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6cc007000 [id: 668] desired_size: 2KB slow allocs: 3 refill waste: 40B alloc: 0.00404 92KB refills: 31 waste 0.9% gc: 16B slow: 816B fast: 0B
TLAB: gc thread: 0x00007fc6d0004800 [id: 667] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00450 102KB refills: 29 waste 2.6% gc: 1456B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc694004800 [id: 666] desired_size: 2KB slow allocs: 5 refill waste: 40B alloc: 0.00413 94KB refills: 32 waste 3.4% gc: 2264B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6c0011800 [id: 665] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00429 97KB refills: 31 waste 1.7% gc: 568B slow: 1088B fast: 0B
TLAB: gc thread: 0x00007fc70c061000 [id: 664] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00435 99KB refills: 38 waste 2.2% gc: 1480B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6bc035800 [id: 663] desired_size: 3KB slow allocs: 13 refill waste: 48B alloc: 0.00434 99KB refills: 32 waste 4.1% gc: 2960B slow: 1168B fast: 0B
TLAB: gc thread: 0x00007fc69c063800 [id: 662] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00437 99KB refills: 30 waste 3.2% gc: 2144B slow: 888B fast: 0B
TLAB: gc thread: 0x00007fc6b400a000 [id: 661] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00454 103KB refills: 32 waste 1.5% gc: 552B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6ec02c800 [id: 660] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00437 99KB refills: 34 waste 2.0% gc: 1040B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6bc034800 [id: 659] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00466 106KB refills: 32 waste 1.7% gc: 672B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6b4008000 [id: 658] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00441 100KB refills: 30 waste 1.7% gc: 656B slow: 944B fast: 0B
TLAB: gc thread: 0x00007fc6f4080800 [id: 657] desired_size: 3KB slow allocs: 3 refill waste: 48B alloc: 0.00449 102KB refills: 25 waste 4.2% gc: 2576B slow: 848B fast: 0B
TLAB: gc thread: 0x00007fc6b0005800 [id: 656] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00426 97KB refills: 32 waste 1.1% gc: 128B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b8053800 [id: 655] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00430 98KB refills: 29 waste 1.5% gc: 432B slow: 888B fast: 0B
TLAB: gc thread: 0x00007fc6ec02a800 [id: 654] desired_size: 2KB slow allocs: 6 refill waste: 40B alloc: 0.00421 96KB refills: 29 waste 1.7% gc: 696B slow: 832B fast: 0B
TLAB: gc thread: 0x00007fc6c8060000 [id: 653] desired_size: 2KB slow allocs: 6 refill waste: 40B alloc: 0.00418 95KB refills: 36 waste 2.3% gc: 1384B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6a4023000 [id: 652] desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00420 96KB refills: 37 waste 3.6% gc: 2720B slow: 1328B fast: 0B
TLAB: gc thread: 0x0000000001815800 [id: 651] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00431 98KB refills: 35 waste 3.4% gc: 2392B slow: 1344B fast: 0B
TLAB: gc thread: 0x00007fc6c0010800 [id: 650] desired_size: 2KB slow allocs: 9 refill waste: 40B alloc: 0.00424 97KB refills: 34 waste 3.8% gc: 2896B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc70c05f000 [id: 649] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00446 102KB refills: 27 waste 1.1% gc: 112B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6cc005000 [id: 648] desired_size: 2KB slow allocs: 5 refill waste: 40B alloc: 0.00420 96KB refills: 30 waste 0.9% gc: 64B slow: 800B fast: 0B
TLAB: gc thread: 0x00007fc694003000 [id: 647] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00439 100KB refills: 33 waste 1.7% gc: 736B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc69c062000 [id: 646] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00426 97KB refills: 31 waste 4.1% gc: 2944B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6f407e000 [id: 645] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00446 102KB refills: 30 waste 2.2% gc: 1040B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc6ec028000 [id: 644] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00434 99KB refills: 30 waste 2.4% gc: 1256B slow: 1032B fast: 0B
TLAB: gc thread: 0x00007fc6cc003000 [id: 643] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00431 98KB refills: 30 waste 1.9% gc: 728B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc6d0001800 [id: 642] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00442 101KB refills: 29 waste 1.5% gc: 552B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6c4091000 [id: 641] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00427 97KB refills: 35 waste 2.2% gc: 1200B slow: 1120B fast: 0B
TLAB: gc thread: 0x00007fc6c80b8000 [id: 639] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00442 100KB refills: 35 waste 3.6% gc: 2816B slow: 1224B fast: 0B
TLAB: gc thread: 0x00007fc6b0004000 [id: 637] desired_size: 2KB slow allocs: 8 refill waste: 40B alloc: 0.00424 97KB refills: 34 waste 1.5% gc: 392B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6b8052000 [id: 636] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00454 103KB refills: 31 waste 1.6% gc: 584B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6bc028800 [id: 635] desired_size: 3KB slow allocs: 6 refill waste: 48B alloc: 0.00439 100KB refills: 27 waste 2.1% gc: 904B slow: 872B fast: 0B
TLAB: gc thread: 0x00007fc6b4006800 [id: 634] desired_size: 2KB slow allocs: 6 refill waste: 40B alloc: 0.00417 95KB refills: 31 waste 4.3% gc: 2976B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6a4021800 [id: 633] desired_size: 2KB slow allocs: 5 refill waste: 40B alloc: 0.00424 96KB refills: 30 waste 2.5% gc: 1472B slow: 816B fast: 0B
TLAB: gc thread: 0x00007fc6c000d800 [id: 632] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00457 104KB refills: 25 waste 3.3% gc: 1880B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc69c061000 [id: 631] desired_size: 3KB slow allocs: 9 refill waste: 48B alloc: 0.00428 97KB refills: 34 waste 2.2% gc: 1160B slow: 1120B fast: 0B
TLAB: gc thread: 0x00007fc764018800 [id: 624] desired_size: 12KB slow allocs: 4 refill waste: 200B alloc: 0.01775 404KB refills: 25 waste 3.5% gc: 10360B slow: 816B fast: 48B
TLAB: gc thread: 0x00007fc764017000 [id: 623] desired_size: 47KB slow allocs: 0 refill waste: 752B alloc: 0.06673 1518KB refills: 33 waste 1.6% gc: 25240B slow: 1032B fast: 0B
TLAB: gc thread: 0x00007fc694001000 [id: 622] desired_size: 3KB slow allocs: 7 refill waste: 48B alloc: 0.00425 97KB refills: 32 waste 1.1% gc: 88B slow: 992B fast: 0B
TLAB: gc thread: 0x00007fc764015800 [id: 621] desired_size: 47KB slow allocs: 0 refill waste: 752B alloc: 0.06686 1521KB refills: 34 waste 2.8% gc: 44816B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc764015000 [id: 620] desired_size: 47KB slow allocs: 0 refill waste: 752B alloc: 0.06696 1523KB refills: 34 waste 3.0% gc: 47240B slow: 1464B fast: 0B
TLAB: gc thread: 0x0000000001814800 [id: 619] desired_size: 2KB slow allocs: 9 refill waste: 40B alloc: 0.00415 94KB refills: 33 waste 4.1% gc: 2936B slow: 1112B fast: 0B
TLAB: gc thread: 0x00007fc76400e800 [id: 618] desired_size: 47KB slow allocs: 0 refill waste: 752B alloc: 0.06704 1525KB refills: 33 waste 0.4% gc: 5272B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc70c054000 [id: 617] desired_size: 3KB slow allocs: 8 refill waste: 48B alloc: 0.00438 100KB refills: 27 waste 2.3% gc: 976B slow: 952B fast: 0B
TLAB: gc thread: 0x00007fc70c050800 [id: 582] desired_size: 3KB slow allocs: 5 refill waste: 48B alloc: 0.00436 99KB refills: 28 waste 1.4% gc: 176B slow: 1080B fast: 0B
TLAB: gc thread: 0x00007fc6f805f000 [id: 443] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00014 3KB refills: 5 waste 4.3% gc: 352B slow: 96B fast: 0B
TLAB: gc thread: 0x00007fc728241800 [id: 328] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00263 60KB refills: 1 waste 44.2% gc: 912B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73c56a800 [id: 282] desired_size: 9KB slow allocs: 0 refill waste: 152B alloc: 0.01413 321KB refills: 1 waste 88.2% gc: 9008B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73c567800 [id: 281] desired_size: 4KB slow allocs: 0 refill waste: 64B alloc: 0.00597 136KB refills: 1 waste 98.9% gc: 4264B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73c4a4000 [id: 279] desired_size: 13KB slow allocs: 0 refill waste: 208B alloc: 0.01852 421KB refills: 1 waste 99.3% gc: 13304B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73803f000 [id: 271] desired_size: 2KB slow allocs: 0 refill waste: 40B alloc: 0.00372 85KB refills: 1 waste 90.5% gc: 2432B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73803a800 [id: 270] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00031 7KB refills: 9 waste 2.1% gc: 136B slow: 232B fast: 16B
TLAB: gc thread: 0x00007fc72c11a000 [id: 267] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00013 3KB refills: 7 waste 0.6% gc: 48B slow: 16B fast: 16B
TLAB: gc thread: 0x00007fc73007d800 [id: 266] desired_size: 2KB slow allocs: 4 refill waste: 32B alloc: 0.00163 37KB refills: 31 waste 4.2% gc: 1840B slow: 728B fast: 88B
TLAB: gc thread: 0x00007fc7300c1000 [id: 225] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00269 61KB refills: 1 waste 100.0% gc: 2064B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc70c14b800 [id: 224] desired_size: 2KB slow allocs: 6 refill waste: 32B alloc: 0.00112 25KB refills: 31 waste 4.2% gc: 1904B slow: 688B fast: 120B
TLAB: gc thread: 0x00007fc728007000 [id: 209] desired_size: 26KB slow allocs: 0 refill waste: 424B alloc: 0.03771 858KB refills: 1 waste 99.6% gc: 27160B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc7b8227000 [id: 196] desired_size: 40KB slow allocs: 0 refill waste: 648B alloc: 0.05801 1320KB refills: 1 waste 99.7% gc: 41856B slow: 0B fast: 0B
TLAB totals: thrds: 181 refills: 5230 max: 38 slow allocs: 1093 max 13 waste: 2.8% gc: 461904B max: 47240B slow: 173544B max: 1464B fast: 488B max: 120B
2017-08-01T10:06:50.679+0000: [GC (System.gc())
Desired survivor size 33554432 bytes, new threshold 3 (max 15)
[PSYoungGen: 26076K->5312K(72192K)] 86627K->65870K(282624K), 0.0141440 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
2017-08-01T10:06:50.694+0000: [Full GC (System.gc()) [PSYoungGen: 5312K->0K(72192K)] [ParOldGen: 60558K->50324K(210432K)] 65870K->50324K(282624K), [Metaspace: 65689K->65689K(1110016K)], 0.1561113 secs] [Times: user=0.52 sys=0.05, real=0.16 secs]
2017-08-01T10:06:50.863+0000: Total time for which application threads were stopped: 0.1985033 seconds, Stopping threads took: 0.0037365 seconds
2017-08-01T10:07:04.502+0000: Total time for which application threads were stopped: 2.5434673 seconds, Stopping threads took: 0.0128751 seconds
TLAB: gc thread: 0x00007fc6f4088000 [id: 975] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00120 0KB refills: 10 waste 4.9% gc: 784B slow: 168B fast: 56B
TLAB: gc thread: 0x00007fc6b8059800 [id: 693] desired_size: 3KB slow allocs: 3 refill waste: 48B alloc: 0.00432 1KB refills: 4 waste 10.4% gc: 1240B slow: 40B fast: 88B
TLAB: gc thread: 0x00007fc6f805f000 [id: 443] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00022 0KB refills: 10 waste 3.5% gc: 496B slow: 168B fast: 56B
TLAB: gc thread: 0x00007fc6e002e800 [id: 396] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00107 0KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc720002000 [id: 332] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00252 1KB refills: 1 waste 100.0% gc: 2064B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73803a800 [id: 270] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00045 0KB refills: 8 waste 10.0% gc: 1376B slow: 280B fast: 0B
TLAB: gc thread: 0x00007fc72c11a000 [id: 267] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00029 0KB refills: 7 waste 6.6% gc: 944B slow: 16B fast: 0B
TLAB: gc thread: 0x00007fc73007d800 [id: 266] desired_size: 2KB slow allocs: 4 refill waste: 32B alloc: 0.00200 1KB refills: 31 waste 4.1% gc: 1808B slow: 728B fast: 88B
TLAB: gc thread: 0x00007fc7300c1000 [id: 225] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00175 0KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc70c14b800 [id: 224] desired_size: 2KB slow allocs: 8 refill waste: 32B alloc: 0.00167 0KB refills: 61 waste 2.8% gc: 1904B slow: 1416B fast: 152B
TLAB totals: thrds: 10 refills: 134 max: 61 slow allocs: 16 max 8 waste: 6.4% gc: 14680B max: 2064B slow: 2816B max: 1416B fast: 440B max: 152B
2017-08-01T10:07:07.702+0000: [GC (System.gc())
Desired survivor size 33030144 bytes, new threshold 4 (max 15)
[PSYoungGen: 282K->224K(70144K)] 50606K->50548K(280576K), 2.6367725 secs] [Times: user=0.08 sys=9.35, real=2.65 secs]
2017-08-01T10:07:10.509+0000: [Full GC (System.gc()) *** JBossAS process (179) received KILL signal ***
{code}
It became clear that the JVM was killed while it was reclaiming space. Unfortunately it took a little while and OOM killer kicked in.
The only missing bit is why it happened with Rocks DB? The answer is in {{System.gc()}} called while reinitializing the database. The test harness used {{cache.clear()}} method which cased that.
The bottom line: if you are using persistent Docker image, please make sure you're not calling {{cache.clear()}} too often or (this is suggested solution) increase memory limits.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
More information about the infinispan-issues
mailing list