[jboss-user] [Performance Tuning] - Full GC not releasing Memory

[vons1234]

Hi,

I am using Jboss 4.2.0 GA version in clustered mode with JDK 1.5.0 update 12 on Enterprise Linux 4 32 bit OS with 4GB of RAM. 

The JVM parameter that i am using is

  | -Xms1000m -Xmx1000m -Xss128k -XX:NewSize=400M -XX:MaxNewSize=400M -verbose:gc -Xloggc:/opt/jboss-4.2.0.GA/logs/GC.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseLargePages -XX:PermSize=512m -XX:MaxPermSize=512m -XX:ThreadStackSize=512 -XX:+UseParallelGC -XX:CompileThreshold=100 -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 
  | 

After a certain period of time, i noticed that even after full GC JVM is not releasing memory and hence doing Full GC every minute after that,
and server goes in hung mode.

During load test which we ran for almost 24-48 hrs with 200 users and i didn't find any problem at all. It was doing Full GC and releasing memory, and GC graph was coming as a perfect saw tooth. Not rising one.

I am not sure what the problem is? Is it memory leak or is it problem in JVM?

Here is the excerpt from GC.log

  | 638334.244: [GC [PSYoungGen: 382909K->3534K(389568K)] 827233K->452848K(1003968K), 0.0115900 secs]
  | 638574.303: [GC [PSYoungGen: 373070K->19442K(381056K)] 822384K->472103K(995456K), 0.0438910 secs]
  | 638793.372: [GC [PSYoungGen: 381042K->16736K(378368K)] 833703K->470908K(992768K), 0.0331950 secs]
  | 638983.090: [GC [PSYoungGen: 378336K->7879K(385472K)] 832508K->469857K(999872K), 0.0309380 secs]
  | 639219.439: [GC [PSYoungGen: 367943K->7233K(384192K)] 829921K->475628K(998592K), 0.0251040 secs]
  | 639285.521: [GC [PSYoungGen: 367297K->2232K(386752K)] 835692K->476113K(1001152K), 0.0109590 secs]
  | 639396.991: [GC [PSYoungGen: 365944K->4784K(386560K)] 839825K->479720K(1000960K), 0.0121520 secs]
  | 639646.787: [GC [PSYoungGen: 368496K->7073K(388736K)] 843432K->484642K(1003136K), 0.0146230 secs]
  | 639703.924: [GC [PSYoungGen: 373729K->2147K(387520K)] 851298K->485016K(1001920K), 0.0114690 secs]
  | 640049.236: [GC [PSYoungGen: 368803K->20607K(364544K)] 851672K->526870K(978944K), 0.5512520 secs]
  | 640716.851: [GC [PSYoungGen: 364543K->7604K(376768K)] 870806K->520935K(991168K), 0.0178060 secs]
  | 640732.582: [GC [PSYoungGen: 351540K->2071K(378944K)] 864871K->522468K(993344K), 0.0152810 secs]
  | 640769.185: [GC [PSYoungGen: 349783K->2404K(378368K)] 870180K->523542K(992768K), 0.0107200 secs]
  | 640857.226: [GC [PSYoungGen: 350116K->2250K(380608K)] 871254K->524553K(995008K), 0.0107310 secs]
  | 641318.940: [GC [PSYoungGen: 352970K->10829K(379712K)] 875273K->533978K(994112K), 0.0150990 secs]
  | 641372.431: [GC [PSYoungGen: 361549K->8612K(383680K)] 884698K->538463K(998080K), 0.0204710 secs]
  | 641443.211: [GC [PSYoungGen: 364516K->21173K(377088K)] 894367K->558436K(991488K), 0.1472660 secs]
  | 641532.918: [GC [PSYoungGen: 377077K->3495K(381568K)] 914340K->542338K(995968K), 0.0122830 secs]
  | 641705.573: [GC [PSYoungGen: 356263K->9448K(380800K)] 895106K->549300K(995200K), 0.0246750 secs]
  | 641735.698: [GC [PSYoungGen: 362216K->4786K(384320K)] 902068K->552784K(998720K), 0.0243410 secs]
  | 641765.077: [GC [PSYoungGen: 362930K->3291K(383424K)] 910928K->553293K(997824K), 0.0120260 secs]
  | 641831.710: [GC [PSYoungGen: 361435K->12727K(385024K)] 911437K->563796K(999424K), 0.0266910 secs]
  | 641896.881: [GC [PSYoungGen: 373175K->2640K(385024K)] 924244K->563498K(999424K), 0.0728110 secs]
  | 641923.804: [GC [PSYoungGen: 363088K->2042K(386112K)] 923946K->564596K(1000512K), 0.0127510 secs]
  | 641968.256: [GC [PSYoungGen: 364090K->3377K(385536K)] 926644K->566768K(999936K), 0.0125380 secs]
  | 641992.504: [GC [PSYoungGen: 365425K->2542K(387776K)] 928816K->567285K(1002176K), 0.0129380 secs]
  | 642101.209: [GC [PSYoungGen: 367726K->2513K(387008K)] 932469K->568066K(1001408K), 0.0119600 secs]
  | 642195.011: [GC [PSYoungGen: 367697K->11696K(387520K)] 933250K->578423K(1001920K), 0.0240670 secs]
  | 642326.913: [GC [PSYoungGen: 378160K->5957K(388032K)] 944887K->573604K(1002432K), 0.0112410 secs]
  | 642423.973: [GC [PSYoungGen: 372421K->2983K(390080K)] 940068K->574423K(1004480K), 0.0119510 secs]
  | 642478.466: [GC [PSYoungGen: 372839K->2920K(389376K)] 944279K->575639K(1003776K), 0.0121660 secs]
  | 642514.758: [GC [PSYoungGen: 372776K->3071K(391616K)] 945495K->576989K(1006016K), 0.0120900 secs]
  | 642521.929: [GC [PSYoungGen: 375935K->2197K(390848K)] 949853K->577433K(1005248K), 0.0127390 secs]
  | 642700.727: [GC [PSYoungGen: 374946K->2422K(392960K)] 950182K->578456K(1007360K), 0.0127970 secs]
  | 643020.504: [GC [PSYoungGen: 378030K->9907K(392256K)] 954064K->586987K(1006656K), 0.0144830 secs]
  | 643198.235: [GC [PSYoungGen: 385502K->6367K(393152K)] 962582K->591327K(1007552K), 0.0174430 secs]
  | 643741.029: [GC [PSYoungGen: 382239K->9497K(392320K)] 967199K->599539K(1006720K), 0.0223850 secs]
  | 643753.433: [GC [PSYoungGen: 385369K->4596K(393664K)] 975411K->602849K(1008064K), 0.0185820 secs]
  | 643895.464: [GC [PSYoungGen: 381492K->3688K(392832K)] 979745K->604220K(1007232K), 0.0117230 secs]
  | 644173.048: [GC [PSYoungGen: 380584K->8471K(394048K)] 981116K->610693K(1008448K), 0.0136230 secs]
  | 645598.528: [GC [PSYoungGen: 387049K->15150K(393920K)] 989272K->622584K(1008320K), 0.0301660 secs]
  | 645598.558: [Full GC [PSYoungGen: 15150K->0K(393920K)] [PSOldGen: 607433K->214127K(614400K)] 622584K->214127K(1008320K) [PSPermGen: 94656K->94656K(524288K)], 1.3451750 secs]
  | 646665.970: [GC [PSYoungGen: 378752K->14655K(385984K)] 592879K->228783K(1000384K), 0.0228620 secs]
  | 646855.357: [GC [PSYoungGen: 384959K->16931K(387264K)] 599087K->243328K(1001664K), 0.0433420 secs]
  | 647161.471: [GC [PSYoungGen: 387235K->8731K(388992K)] 613632K->245455K(1003392K), 0.0322590 secs]
  | 648269.211: [GC [PSYoungGen: 375835K->17468K(384576K)] 612559K->261305K(998976K), 0.0373880 secs]
  | 648275.976: [GC [PSYoungGen: 384572K->6858K(387776K)] 628409K->266318K(1002176K), 0.0277720 secs]
  | 648551.571: [GC [PSYoungGen: 371914K->9432K(386880K)] 631374K->275601K(1001280K), 0.0285320 secs]
  | 649470.985: [GC [PSYoungGen: 374488K->14102K(387648K)] 640657K->287654K(1002048K), 0.0324130 secs]
  | 649687.132: [GC [PSYoungGen: 381078K->4902K(388288K)] 654630K->290314K(1002688K), 0.0201850 secs]
  | 649861.212: [GC [PSYoungGen: 371878K->20979K(367104K)] 657290K->326846K(981504K), 0.5436490 secs]
  | 650061.614: [GC [PSYoungGen: 367091K->31740K(369920K)] 672958K->342541K(984320K), 0.2747920 secs]
  | 650674.027: [GC [PSYoungGen: 369916K->8463K(366464K)] 680717K->329343K(980864K), 0.0272090 secs]
  | 650821.195: [GC [PSYoungGen: 343183K->7110K(372160K)] 664063K->334410K(986560K), 0.0162060 secs]
  | 650849.520: [GC [PSYoungGen: 341830K->5526K(374144K)] 669130K->339345K(988544K), 0.0152460 secs]
  | 650935.945: [GC [PSYoungGen: 343318K->7576K(373248K)] 677137K->343814K(987648K), 0.0603020 secs]
  | 651081.184: [GC [PSYoungGen: 345368K->4032K(376320K)] 681606K->342073K(990720K), 0.0127370 secs]
  | 651118.991: [GC [PSYoungGen: 346291K->3550K(375552K)] 684332K->344691K(989952K), 0.0137720 secs]
  | 651302.587: [GC [PSYoungGen: 345822K->8030K(378944K)] 686963K->350148K(993344K), 0.0126160 secs]
  | 651397.803: [GC [PSYoungGen: 354590K->8456K(377216K)] 696708K->356798K(991616K), 0.0281420 secs]
  | 651528.420: [GC [PSYoungGen: 355016K->9448K(382400K)] 703358K->359280K(996800K), 0.0243760 secs]
  | 651694.350: [GC [PSYoungGen: 362856K->9198K(380608K)] 712688K->360630K(995008K), 0.0246090 secs]
  | 651708.556: [GC [PSYoungGen: 362606K->1861K(384128K)] 714038K->361109K(998528K), 0.0176420 secs]
  | 651923.619: [GC [PSYoungGen: 360325K->11393K(383936K)] 719573K->371187K(998336K), 0.0245150 secs]
  | 652271.782: [GC [PSYoungGen: 369857K->12653K(385280K)] 729651K->382658K(999680K), 0.0296320 secs]
  | 652666.528: [GC [PSYoungGen: 373228K->6021K(384960K)] 743234K->386480K(999360K), 0.0205290 secs]
  | 652699.157: [GC [PSYoungGen: 366661K->2842K(386368K)] 747120K->388509K(1000768K), 0.0127320 secs]
  | 652823.786: [GC [PSYoungGen: 365722K->3050K(386112K)] 751389K->389606K(1000512K), 0.0102480 secs]
  | 653038.640: [GC [PSYoungGen: 365930K->9727K(387776K)] 752486K->397544K(1002176K), 0.0230760 secs]
  | 653420.192: [GC [PSYoungGen: 374783K->6694K(386880K)] 762600K->401599K(1001280K), 0.0237390 secs]
  | 653800.634: [GC [PSYoungGen: 371750K->9360K(389632K)] 766655K->409695K(1004032K), 0.0187360 secs]
  | 654213.270: [GC [PSYoungGen: 378192K->6818K(388800K)] 778527K->414260K(1003200K), 0.0157230 secs]
  | 655005.242: [GC [PSYoungGen: 375650K->14517K(387968K)] 783092K->425968K(1002368K), 0.0251050 secs]
  | 655485.907: [GC [PSYoungGen: 383413K->7400K(389248K)] 794864K->428250K(1003648K), 0.0205270 secs]
  | 655939.178: [GC [PSYoungGen: 376296K->10991K(390400K)] 797146K->437312K(1004800K), 0.0266040 secs]
  | 656004.313: [GC [PSYoungGen: 381999K->19198K(381376K)] 808320K->470283K(995776K), 0.6070120 secs]
  | 656121.593: [GC [PSYoungGen: 381374K->9371K(369344K)] 832459K->466870K(983744K), 0.0285030 secs]
  | 656173.650: [GC [PSYoungGen: 359323K->29819K(371712K)] 816822K->540154K(986112K), 0.9276420 secs]
  | 656181.565: [GC [PSYoungGen: 371707K->29808K(305536K)] 882042K->573493K(919936K), 0.2259960 secs]
  | 656221.947: [GC [PSYoungGen: 305520K->12017K(342656K)] 849205K->582577K(957056K), 0.0762400 secs]
  | 656294.990: [GC [PSYoungGen: 287729K->12821K(347648K)] 858289K->595248K(962048K), 0.0351610 secs]
  | 656295.025: [Full GC [PSYoungGen: 12821K->0K(347648K)] [PSOldGen: 582427K->260589K(614400K)] 595248K->260589K(962048K) [PSPermGen: 96631K->96631K(524288K)], 1.3458950 secs]
  | 656322.631: [GC [PSYoungGen: 283072K->6426K(345024K)] 543661K->267016K(959424K), 0.0147990 secs]
  | 656328.315: [GC [PSYoungGen: 289498K->5615K(350784K)] 550088K->272389K(965184K), 0.0171960 secs]
  | 656381.131: [GC [PSYoungGen: 295919K->10135K(349120K)] 562693K->281948K(963520K), 0.0234060 secs]
  | 656438.871: [GC [PSYoungGen: 300439K->10362K(355968K)] 572252K->291720K(970368K), 0.0310620 secs]
  | 656498.509: [GC [PSYoungGen: 310074K->9785K(353344K)] 591432K->300709K(967744K), 0.0242930 secs]
  | 656556.480: [GC [PSYoungGen: 309497K->9012K(360704K)] 600421K->309165K(975104K), 0.0258050 secs]
  | 656619.807: [GC [PSYoungGen: 318516K->9349K(358400K)] 618669K->317962K(972800K), 0.0255210 secs]
  | 656682.572: [GC [PSYoungGen: 318853K->11538K(365568K)] 627466K->328892K(979968K), 0.0313280 secs]
  | 656747.387: [GC [PSYoungGen: 330514K->9051K(363008K)] 647868K->337694K(977408K), 0.0271890 secs]
  | 656799.384: [GC [PSYoungGen: 328027K->7916K(369344K)] 656670K->345474K(983744K), 0.0257200 secs]
  | 656860.578: [GC [PSYoungGen: 335276K->8543K(367616K)] 672834K->353534K(982016K), 0.0232020 secs]
  | 656921.072: [GC [PSYoungGen: 335903K->8501K(372992K)] 680894K->361799K(987392K), 0.0239130 secs]
  | 656984.540: [GC [PSYoungGen: 343093K->8707K(371200K)] 696391K->370146K(985600K), 0.0277250 secs]
  | 657021.279: [GC [PSYoungGen: 343299K->13068K(376704K)] 704738K->382930K(991104K), 0.0387500 secs]
  | 657084.842: [GC [PSYoungGen: 354956K->8471K(374784K)] 724818K->389620K(989184K), 0.0294630 secs]
  | 657149.816: [GC [PSYoungGen: 350359K->11553K(380032K)] 731508K->400999K(994432K), 0.0296180 secs]
  | 657211.720: [GC [PSYoungGen: 360097K->9123K(378112K)] 749543K->409913K(992512K), 0.0304270 secs]
  | 657261.603: [GC [PSYoungGen: 357667K->7744K(382400K)] 758457K->417458K(996800K), 0.0267330 secs]
  | 657321.056: [GC [PSYoungGen: 361920K->7820K(381376K)] 771634K->424927K(995776K), 0.0281760 secs]
  | 657381.760: [GC [PSYoungGen: 361996K->7829K(384448K)] 779103K->432652K(998848K), 0.0219280 secs]
  | 657438.901: [GC [PSYoungGen: 366101K->7258K(383424K)] 790924K->439703K(997824K), 0.0220020 secs]
  | 657477.779: [GC [PSYoungGen: 365530K->5508K(385984K)] 797975K->445069K(1000384K), 0.0196690 secs]
  | 657536.879: [GC [PSYoungGen: 367236K->7360K(385344K)] 806797K->451737K(999744K), 0.0213670 secs]
  | 657589.892: [GC [PSYoungGen: 369088K->11766K(387456K)] 813465K->463258K(1001856K), 0.0260220 secs]
  | 657641.039: [GC [PSYoungGen: 376566K->6365K(386944K)] 828058K->469402K(1001344K), 0.0269390 secs]
  | 657689.229: [GC [PSYoungGen: 371165K->5649K(388672K)] 834202K->474787K(1003072K), 0.0242300 secs]
  | 657739.169: [GC [PSYoungGen: 372817K->5904K(388096K)] 841955K->480375K(1002496K), 0.0229120 secs]
  | 657789.283: [GC [PSYoungGen: 373072K->5476K(389952K)] 847543K->485595K(1004352K), 0.0204630 secs]
  | 657835.959: [GC [PSYoungGen: 375140K->5259K(389312K)] 855259K->490630K(1003712K), 0.0202190 secs]
  | 657880.940: [GC [PSYoungGen: 374923K->5166K(391168K)] 860294K->495574K(1005568K), 0.0230860 secs]
  | 657935.851: [GC [PSYoungGen: 377262K->5395K(390528K)] 867670K->500663K(1004928K), 0.0204960 secs]
  | 657980.161: [GC [PSYoungGen: 377491K->4841K(392384K)] 872759K->505164K(1006784K), 0.0198990 secs]
  | 658022.434: [GC [PSYoungGen: 379433K->7578K(391808K)] 879756K->512495K(1006208K), 0.0201340 secs]
  | 658069.678: [GC [PSYoungGen: 382170K->5128K(393792K)] 887087K->517399K(1008192K), 0.0259460 secs]
  | 658111.339: [GC [PSYoungGen: 382536K->4299K(393216K)] 894807K->521332K(1007616K), 0.0195400 secs]
  | 658153.222: [GC [PSYoungGen: 381707K->4148K(394560K)] 898740K->525341K(1008960K), 0.0189940 secs]
  | 658195.446: [GC [PSYoungGen: 383284K->4427K(394176K)] 904477K->529587K(1008576K), 0.0188760 secs]
  | 658234.968: [GC [PSYoungGen: 383563K->4543K(395584K)] 908723K->533812K(1009984K), 0.0228260 secs]
  | 658268.894: [GC [PSYoungGen: 385599K->4637K(395072K)] 914868K->537905K(1009472K), 0.0218180 secs]
  | 658308.930: [GC [PSYoungGen: 385693K->4423K(396608K)] 918961K->542177K(1011008K), 0.0191590 secs]
  | 658350.354: [GC [PSYoungGen: 387527K->4205K(396096K)] 925281K->546222K(1010496K), 0.0176900 secs]
  | 658388.702: [GC [PSYoungGen: 387309K->4229K(397440K)] 929326K->550272K(1011840K), 0.0192770 secs]
  | 658419.488: [GC [PSYoungGen: 389125K->4228K(397056K)] 935168K->554298K(1011456K), 0.0204370 secs]
  | 658456.421: [GC [PSYoungGen: 389124K->6514K(398144K)] 939194K->560262K(1012544K), 0.0216680 secs]
  | 658478.320: [GC [PSYoungGen: 392946K->6280K(397888K)] 946694K->566233K(1012288K), 0.0237350 secs]
  | 658516.210: [GC [PSYoungGen: 392712K->4394K(398784K)] 952665K->570201K(1013184K), 0.0208810 secs]
  | 658540.904: [GC [PSYoungGen: 391977K->10232K(397824K)] 957784K->580025K(1012224K), 0.0325470 secs]
  | 658582.314: [GC [PSYoungGen: 397816K->4311K(397888K)] 967609K->582310K(1012288K), 0.0271830 secs]
  | 658622.909: [GC [PSYoungGen: 390423K->3965K(397824K)] 968422K->585844K(1012224K), 0.0210630 secs]
  | 658661.443: [GC [PSYoungGen: 390077K->3979K(398464K)] 971956K->589492K(1012864K), 0.0180430 secs]
  | 658691.992: [GC [PSYoungGen: 390987K->3710K(398144K)] 976500K->593007K(1012544K), 0.0205180 secs]
  | 658727.253: [GC [PSYoungGen: 390718K->3781K(399040K)] 980015K->596430K(1013440K), 0.0176180 secs]
  | 658762.386: [GC [PSYoungGen: 391941K->4601K(398720K)] 984590K->600829K(1013120K), 0.0205710 secs]
  | 658800.867: [GC [PSYoungGen: 392761K->3772K(399808K)] 988989K->603481K(1014208K), 0.0184240 secs]
  | 658840.721: [GC [PSYoungGen: 393468K->4324K(399488K)] 993177K->607656K(1013888K), 0.0188700 secs]
  | 658877.949: [GC [PSYoungGen: 394020K->4349K(400512K)] 997352K->611709K(1014912K), 0.0223020 secs]
  | 658877.971: [Full GC [PSYoungGen: 4349K->0K(400512K)] [PSOldGen: 607359K->470861K(614400K)] 611709K->470861K(1014912K) [PSPermGen: 96954K->96954K(524288K)], 1.9780070 secs]
  | 658912.455: [GC [PSYoungGen: 391040K->3248K(400128K)] 861901K->474109K(1014528K), 0.0160360 secs]
  | 658948.104: [GC [PSYoungGen: 394288K->8895K(395968K)] 865149K->483027K(1010368K), 0.0211960 secs]
  | 658984.865: [GC [PSYoungGen: 395967K->3661K(398336K)] 870099K->486127K(1012736K), 0.0214470 secs]
  | 659020.755: [GC [PSYoungGen: 390733K->3677K(398912K)] 873199K->489382K(1013312K), 0.0178900 secs]
  | 659058.534: [GC [PSYoungGen: 391581K->3583K(398592K)] 877286K->492784K(1012992K), 0.0174240 secs]
  | 659091.954: [GC [PSYoungGen: 391487K->3264K(399488K)] 880688K->495810K(1013888K), 0.0201100 secs]
  | 659138.325: [GC [PSYoungGen: 392384K->3908K(399232K)] 884930K->499397K(1013632K), 0.0185000 secs]
  | 659175.115: [GC [PSYoungGen: 393028K->3473K(400128K)] 888517K->502622K(1014528K), 0.0189690 secs]
  | 659209.316: [GC [PSYoungGen: 393873K->3313K(399872K)] 893022K->505693K(1014272K), 0.0183470 secs]
  | 659247.605: [GC [PSYoungGen: 393713K->3408K(400704K)] 896093K->508910K(1015104K), 0.0177630 secs]
  | 659287.687: [GC [PSYoungGen: 394960K->3373K(400448K)] 900462K->512135K(1014848K), 0.0177930 secs]
  | 659320.079: [GC [PSYoungGen: 394925K->3130K(401280K)] 903687K->515128K(1015680K), 0.0189960 secs]
  | 659354.840: [GC [PSYoungGen: 395834K->3199K(401024K)] 907832K->518071K(1015424K), 0.0173630 secs]
  | 659387.659: [GC [PSYoungGen: 395903K->6516K(400512K)] 910775K->524398K(1014912K), 0.0207780 secs]
  | 659423.273: [GC [PSYoungGen: 398964K->3233K(401024K)] 916846K->527301K(1015424K), 0.0213660 secs]
  | 659455.192: [GC [PSYoungGen: 395681K->2821K(401408K)] 919749K->529878K(1015808K), 0.0218030 secs]
  | 659487.094: [GC [PSYoungGen: 395909K->3156K(401280K)] 922966K->532836K(1015680K), 0.0207430 secs]
  | 659518.663: [GC [PSYoungGen: 396244K->3049K(401920K)] 925924K->535546K(1016320K), 0.0191650 secs]
  | 659552.725: [GC [PSYoungGen: 397033K->3021K(401664K)] 929530K->538307K(1016064K), 0.0177230 secs]
  | 659584.437: [GC [PSYoungGen: 397005K->3143K(402432K)] 932291K->541233K(1016832K), 0.0180700 secs]
  | 659619.945: [GC [PSYoungGen: 398151K->2953K(402176K)] 936241K->543839K(1016576K), 0.0199450 secs]
  | 659654.351: [GC [PSYoungGen: 397961K->2587K(402816K)] 938847K->546277K(1017216K), 0.0221100 secs]
  | 659688.893: [GC [PSYoungGen: 398427K->3042K(402624K)] 942117K->549098K(1017024K), 0.0196260 secs]
  | 659719.930: [GC [PSYoungGen: 398882K->2948K(403264K)] 944938K->551293K(1017664K), 0.0180750 secs]
  | 659750.607: [GC [PSYoungGen: 399620K->2773K(403008K)] 947965K->553420K(1017408K), 0.0178410 secs]
  | 659784.822: [GC [PSYoungGen: 399445K->2759K(403584K)] 950092K->555862K(1017984K), 0.0185910 secs]
  | 659817.732: [GC [PSYoungGen: 400135K->2633K(403392K)] 953238K->558136K(1017792K), 0.0179920 secs]
  | 659854.273: [GC [PSYoungGen: 400009K->5298K(402752K)] 955512K->563355K(1017152K), 0.0200440 secs]
  | 659882.229: [GC [PSYoungGen: 402226K->2293K(403264K)] 960283K->565533K(1017664K), 0.0201230 secs]
  | 659904.669: [GC [PSYoungGen: 399221K->3413K(403584K)] 962461K->568630K(1017984K), 0.0182620 secs]
  | 659935.825: [GC [PSYoungGen: 400725K->2379K(403328K)] 965942K->570614K(1017728K), 0.0206350 secs]
  | 659958.433: [GC [PSYoungGen: 399691K->2031K(403776K)] 967926K->572227K(1018176K), 0.0189340 secs]
  | 659986.697: [GC [PSYoungGen: 399919K->2379K(403712K)] 970115K->574243K(1018112K), 0.0176210 secs]
  | 660014.881: [GC [PSYoungGen: 400267K->2263K(404032K)] 972131K->576253K(1018432K), 0.0174980 secs]
  | 660044.485: [GC [PSYoungGen: 400599K->2261K(403904K)] 974589K->578350K(1018304K), 0.0179230 secs]
  | 660076.791: [GC [PSYoungGen: 400597K->2385K(404352K)] 976686K->580560K(1018752K), 0.0189120 secs]
  | 660105.838: [GC [PSYoungGen: 401297K->2196K(404160K)] 979472K->582566K(1018560K), 0.0211820 secs]
  | 660137.756: [GC [PSYoungGen: 401108K->2483K(404608K)] 981478K->584739K(1019008K), 0.0196710 secs]
  | 660175.364: [GC [PSYoungGen: 401971K->2573K(404480K)] 984227K->587083K(1018880K), 0.0182130 secs]
  | 660206.927: [GC [PSYoungGen: 402061K->1968K(404928K)] 986571K->588972K(1019328K), 0.0189490 secs]
  | 660238.717: [GC [PSYoungGen: 402088K->2933K(404800K)] 989092K->591836K(1019200K), 0.0174710 secs]
  | 660264.231: [GC [PSYoungGen: 403061K->4011K(404352K)] 991964K->595502K(1018752K), 0.0197100 secs]
  | 660286.738: [GC [PSYoungGen: 403755K->4922K(404672K)] 995246K->603984K(1019072K), 0.0253990 secs]
  | 660326.883: [GC [PSYoungGen: 404666K->2734K(398528K)] 1003728K->606472K(1012928K), 0.0201120 secs]
  | 660365.392: [GC [PSYoungGen: 396334K->2461K(401600K)] 1000072K->608782K(1016000K), 0.0215800 secs]
  | 660400.515: [GC [PSYoungGen: 396061K->2351K(402048K)] 1002382K->610987K(1016448K), 0.0200040 secs]
  | 660400.535: [Full GC [PSYoungGen: 2351K->0K(402048K)] [PSOldGen: 608635K->514108K(614400K)] 610987K->514108K(1016448K) [PSPermGen: 96983K->92339K(524288K)], 2.3080190 secs]
  | 660435.569: [GC [PSYoungGen: 394240K->2316K(401792K)] 908348K->516424K(1016192K), 0.0191240 secs]
  | 660468.551: [GC [PSYoungGen: 396556K->2274K(402496K)] 910664K->518290K(1016896K), 0.0163310 secs]
  | 660598.713: [GC [PSYoungGen: 397410K->3857K(402240K)] 913426K->521855K(1016640K), 0.0271960 secs]
  | 662029.122: [GC [PSYoungGen: 398993K->6965K(391360K)] 916991K->537555K(1005760K), 0.0384350 secs]
  | 664136.745: [GC [PSYoungGen: 391349K->12605K(387264K)] 921939K->553901K(1001664K), 0.0518620 secs]
  | 667799.172: [GC [PSYoungGen: 387261K->12603K(364480K)] 928557K->578758K(978880K), 0.0993300 secs]
  | 671396.905: [GC [PSYoungGen: 364475K->27586K(371008K)] 930630K->602946K(985408K), 0.0946000 secs]
  | 674773.489: [GC [PSYoungGen: 370922K->30072K(359424K)] 946282K->610756K(973824K), 0.1068250 secs]
  | 676297.884: [GC [PSYoungGen: 359416K->15205K(369472K)] 940100K->610196K(983872K), 0.0460100 secs]
  | 676319.996: [GC [PSYoungGen: 344549K->15603K(373440K)] 939540K->610595K(987840K), 0.0215120 secs]
  | 678366.101: [GC [PSYoungGen: 351091K->31172K(366720K)] 946083K->626163K(981120K), 0.0842090 secs]
  | 680890.747: [GC [PSYoungGen: 366621K->33085K(361536K)] 961613K->628077K(975936K), 0.0834900 secs]
  | 684312.155: [GC [PSYoungGen: 354365K->41491K(355392K)] 949357K->636482K(969792K), 0.1123060 secs]
  | 688035.391: [GC [PSYoungGen: 355347K->46783K(341568K)] 950338K->642422K(955968K), 0.1270880 secs]
  | 689423.024: [GC [PSYoungGen: 341567K->36214K(331008K)] 937206K->636244K(945408K), 0.0760550 secs]
  | 689423.100: [Full GC [PSYoungGen: 36214K->0K(331008K)] [PSOldGen: 600029K->165272K(544768K)] 636244K->165272K(875776K) [PSPermGen: 92559K->92559K(524288K)], 0.9996560 secs]
  | 689453.358: [GC [PSYoungGen: 294784K->5889K(351040K)] 460056K->171161K(895808K), 0.0122170 secs]
  | 689488.241: [GC [PSYoungGen: 297985K->11160K(350656K)] 463257K->176432K(895424K), 0.0163980 secs]
  | 689522.139: [GC [PSYoungGen: 303256K->16278K(355520K)] 468528K->181550K(900288K), 0.0225640 secs]
  | 689544.876: [GC [PSYoungGen: 314966K->21059K(352768K)] 480238K->186331K(897536K), 0.0276110 secs]
  | 689550.463: [GC [PSYoungGen: 319747K->29424K(358336K)] 485019K->194696K(903104K), 0.0337270 secs]
  | 689556.323: [GC [PSYoungGen: 336880K->32229K(339712K)] 502152K->197501K(884480K), 0.0381010 secs]
  | 689561.439: [GC [PSYoungGen: 339685K->29522K(357888K)] 504957K->199257K(902656K), 0.0394270 secs]
  | 689566.917: [GC [PSYoungGen: 335314K->23753K(357504K)] 505049K->201617K(902272K), 0.0355240 secs]
  | 689572.532: [GC [PSYoungGen: 329545K->17858K(362240K)] 507409K->204139K(907008K), 0.0271000 secs]
  | 689578.673: [GC [PSYoungGen: 331074K->16449K(360576K)] 517355K->207057K(905344K), 0.0216010 secs]
  | 689584.498: [GC [PSYoungGen: 329665K->13663K(364480K)] 520273K->208883K(909248K), 0.0188230 secs]
  | 689590.511: [GC [PSYoungGen: 332127K->11910K(363584K)] 527347K->211759K(908352K), 0.0180270 secs]
  | 689596.603: [GC [PSYoungGen: 330374K->11622K(366144K)] 530223K->213837K(910912K), 0.0182930 secs]
  | 689602.879: [GC [PSYoungGen: 333414K->11971K(365248K)] 535629K->216561K(910016K), 0.0192430 secs]
  | 689609.500: [GC [PSYoungGen: 333763K->11595K(368384K)] 538353K->218534K(913152K), 0.0180680 secs]
  | 689615.706: [GC [PSYoungGen: 337675K->11726K(367296K)] 544614K->221128K(912064K), 0.0163100 secs]
  | 689622.037: [GC [PSYoungGen: 337806K->12218K(371008K)] 547208K->224106K(915776K), 0.0195490 secs]
  | 689628.750: [GC [PSYoungGen: 343290K->12182K(369664K)] 555178K->226512K(914432K), 0.0188480 secs]
  | 689635.638: [GC [PSYoungGen: 343254K->12142K(373696K)] 557584K->229061K(918464K), 0.0220370 secs]
  | 689642.062: [GC [PSYoungGen: 348590K->12115K(372352K)] 565509K->231579K(917120K), 0.0197570 secs]
  | 689648.219: [GC [PSYoungGen: 348563K->12451K(376384K)] 568027K->234459K(921152K), 0.0199030 secs]
  | 692011.194: [GC [PSYoungGen: 354275K->22144K(375040K)] 576283K->245355K(919808K), 0.0540150 secs]
  | 695861.098: [GC [PSYoungGen: 363968K->29421K(369280K)] 587179K->261249K(914048K), 0.1022380 secs]
  | 699350.230: [GC [PSYoungGen: 364781K->24802K(360192K)] 596609K->266626K(904960K), 0.0790430 secs]
  | 700065.228: [GC [PSYoungGen: 360162K->17674K(373568K)] 601986K->270145K(918336K), 0.0361150 secs]
  | 701020.954: [GC [PSYoungGen: 353354K->12096K(371712K)] 605825K->277004K(916480K), 0.0348170 secs]
  | 701638.611: [GC [PSYoungGen: 347776K->7543K(374464K)] 612684K->282401K(919232K), 0.0229670 secs]
  | 702841.223: [GC [PSYoungGen: 346807K->18094K(374400K)] 621665K->298272K(919168K), 0.0326790 secs]
  | 705073.544: [GC [PSYoungGen: 357358K->19719K(376320K)] 637536K->309824K(921088K), 0.0640360 secs]
  | 707458.666: [GC [PSYoungGen: 362183K->26685K(369152K)] 652288K->316791K(913920K), 0.0685300 secs]
  | 710010.791: [GC [PSYoungGen: 369149K->32502K(367040K)] 659255K->322608K(911808K), 0.0860740 secs]
  | 711944.586: [GC [PSYoungGen: 364278K->33695K(365504K)] 654384K->323801K(910272K), 0.0804850 secs]
  | 713282.786: [GC [PSYoungGen: 365471K->33423K(362304K)] 655577K->323529K(907072K), 0.0657360 secs]
  | 713361.801: [GC [PSYoungGen: 353423K->27723K(364800K)] 643529K->317828K(909568K), 0.0362610 secs]
  | 713713.995: [GC [PSYoungGen: 347723K->29345K(364416K)] 637828K->319451K(909184K), 0.0459590 secs]
  | 714183.489: [GC [PSYoungGen: 349089K->32159K(351936K)] 639195K->322264K(896704K), 0.0515730 secs]
  | 714428.775: [GC [PSYoungGen: 351903K->39303K(356736K)] 642008K->329408K(901504K), 0.0733710 secs]
  | 714529.527: [GC [PSYoungGen: 349946K->37747K(348416K)] 640052K->327853K(893184K), 0.0580190 secs]
  | 715035.377: [GC [PSYoungGen: 348403K->41939K(351808K)] 638509K->332045K(896576K), 0.0720650 secs]
  | 716528.907: [GC [PSYoungGen: 342227K->48892K(342400K)] 632333K->338998K(887168K), 0.0927830 secs]
  | 716800.103: [GC [PSYoungGen: 342396K->44941K(345024K)] 632502K->335047K(889792K), 0.0710370 secs]
  | 716805.919: [GC [PSYoungGen: 330381K->44692K(330176K)] 620487K->334797K(874944K), 0.0758470 secs]
  | 717159.532: [GC [PSYoungGen: 330132K->43132K(344640K)] 620237K->339761K(889408K), 0.0825480 secs]
  | 717398.904: [GC [PSYoungGen: 323964K->35702K(345216K)] 620593K->338908K(889984K), 0.0675400 secs]
  | 717473.795: [GC [PSYoungGen: 316534K->34810K(350208K)] 619740K->341551K(894976K), 0.0650400 secs]
  | 717508.513: [GC [PSYoungGen: 323066K->30086K(347648K)] 629807K->340379K(892416K), 0.0509280 secs]
  | 717639.494: [GC [PSYoungGen: 318342K->25301K(352896K)] 628635K->342000K(897664K), 0.0469360 secs]
  | 717708.747: [GC [PSYoungGen: 320597K->20767K(352000K)] 637296K->343156K(896768K), 0.0425130 secs]
  | 717722.782: [GC [PSYoungGen: 316063K->22596K(354048K)] 638452K->348165K(898816K), 0.0339900 secs]
  | 717775.763: [GC [PSYoungGen: 320195K->11343K(353152K)] 645765K->345895K(897920K), 0.0292330 secs]
  | 717799.423: [GC [PSYoungGen: 308943K->8416K(351744K)] 643495K->347548K(896512K), 0.0203280 secs]
  | 717827.483: [GC [PSYoungGen: 303712K->6291K(301632K)] 642844K->347890K(846400K), 0.0153400 secs]
  | 717936.707: [GC [PSYoungGen: 301587K->13489K(353280K)] 643186K->358165K(898048K), 0.0304790 secs]
  | 718039.406: [GC [PSYoungGen: 309105K->12640K(351936K)] 653781K->359084K(896704K), 0.0212950 secs]
  | 718072.347: [GC [PSYoungGen: 308256K->8574K(355520K)] 654700K->360512K(900288K), 0.0292450 secs]
  | 718177.203: [GC [PSYoungGen: 309157K->3317K(354688K)] 661095K->361769K(899456K), 0.0184110 secs]
  | 718350.993: [GC [PSYoungGen: 303925K->11191K(357760K)] 662377K->371255K(902528K), 0.0220080 secs]
  | 718363.294: [GC [PSYoungGen: 315127K->1382K(355776K)] 675191K->371102K(900544K), 0.0140040 secs]
  | 718483.950: [GC [PSYoungGen: 305318K->3839K(359424K)] 675038K->374208K(904192K), 0.0151990 secs]
  | 718553.008: [GC [PSYoungGen: 311849K->2396K(358208K)] 682217K->374755K(902976K), 0.0135580 secs]
  | 718669.028: [GC [PSYoungGen: 310428K->4116K(362368K)] 682787K->377452K(907136K), 0.0153090 secs]
  | 718707.921: [GC [PSYoungGen: 317524K->6256K(360640K)] 690860K->379592K(905408K), 0.0148920 secs]
  | 718836.436: [GC [PSYoungGen: 319664K->9967K(367424K)] 693000K->383302K(912192K), 0.0161940 secs]
  | 718953.885: [GC [PSYoungGen: 332399K->23863K(364608K)] 705734K->397198K(909376K), 0.0442910 secs]
  | 719019.147: [GC [PSYoungGen: 346295K->19184K(366976K)] 719630K->392520K(911744K), 0.0284550 secs]
  | 719251.144: [GC [PSYoungGen: 342640K->23788K(366080K)] 715976K->397123K(910848K), 0.0318370 secs]
  | 719295.214: [GC [PSYoungGen: 347244K->25518K(364160K)] 720579K->398853K(908928K), 0.0340830 secs]
  | 719344.967: [GC [PSYoungGen: 346286K->24169K(344960K)] 719621K->399763K(889728K), 0.0711610 secs]
  | 719369.425: [GC [PSYoungGen: 344937K->19155K(366912K)] 720531K->400765K(911680K), 0.0340840 secs]
  | 719444.825: [GC [PSYoungGen: 341459K->14125K(364992K)] 723069K->404108K(909760K), 0.0324540 secs]
  | 719614.669: [GC [PSYoungGen: 336429K->8752K(369984K)] 726412K->406000K(914752K), 0.0244680 secs]
  | 719621.093: [GC [PSYoungGen: 338096K->7872K(368960K)] 735344K->406806K(913728K), 0.0754750 secs]
  | 719734.091: [GC [PSYoungGen: 337216K->9580K(372352K)] 736150K->413532K(917120K), 0.0190550 secs]
  | 719749.328: [GC [PSYoungGen: 343276K->9656K(370944K)] 747228K->422175K(915712K), 0.0331100 secs]
  | 719781.530: [GC [PSYoungGen: 343352K->5162K(374400K)] 755871K->418837K(919168K), 0.0163050 secs]
  | 719913.859: [GC [PSYoungGen: 343722K->7110K(373760K)] 757397K->422736K(918528K), 0.0162460 secs]
  | 719968.442: [GC [PSYoungGen: 345670K->5365K(376448K)] 761296K->425715K(921216K), 0.0171940 secs]
  | 720125.008: [GC [PSYoungGen: 347765K->5541K(375552K)] 768115K->428469K(920320K), 0.0166090 secs]
  | 720204.806: [GC [PSYoungGen: 347885K->6708K(378816K)] 770813K->432900K(923584K), 0.0178070 secs]
  | 720226.854: [GC [PSYoungGen: 353456K->10331K(377536K)] 779647K->441425K(922304K), 0.0303880 secs]
  | 720349.945: [GC [PSYoungGen: 356969K->5080K(381568K)] 788063K->444089K(926336K), 0.0215430 secs]
  | 720432.770: [GC [PSYoungGen: 357784K->3099K(380736K)] 796793K->444807K(925504K), 0.0165510 secs]
  | 720581.413: [GC [PSYoungGen: 355803K->5091K(383232K)] 797511K->448070K(928000K), 0.0138520 secs]
  | 720605.228: [GC [PSYoungGen: 360867K->5577K(382144K)] 803846K->452456K(926912K), 0.0182840 secs]
  | 720777.591: [GC [PSYoungGen: 361353K->6601K(385664K)] 808232K->457316K(930432K), 0.0187640 secs]
  | 720995.250: [GC [PSYoungGen: 366985K->4605K(384320K)] 817700K->459399K(929088K), 0.0248910 secs]
  | 721074.870: [GC [PSYoungGen: 364989K->12642K(386176K)] 819783K->467436K(930944K), 0.0242300 secs]
  | 721704.366: [GC [PSYoungGen: 375778K->19646K(382784K)] 830572K->474440K(927552K), 0.0312670 secs]
  | 721835.815: [GC [PSYoungGen: 382782K->22808K(377280K)] 837576K->477602K(922048K), 0.0347600 secs]
  | 721920.402: [GC [PSYoungGen: 374168K->22485K(373888K)] 828962K->477279K(918656K), 0.0294390 secs]
  | 722033.210: [GC [PSYoungGen: 373845K->25294K(372224K)] 828639K->480088K(916992K), 0.0374430 secs]
  | 722246.532: [GC [PSYoungGen: 365902K->27879K(368512K)] 820696K->482673K(913280K), 0.0386700 secs]
  | 722384.184: [GC [PSYoungGen: 368487K->31552K(364032K)] 823281K->486347K(908800K), 0.0541890 secs]
  | 722652.234: [GC [PSYoungGen: 357806K->29198K(355520K)] 812601K->483993K(900288K), 0.0412730 secs]
  | 722753.783: [GC [PSYoungGen: 355470K->31177K(361152K)] 810265K->485972K(905920K), 0.0434400 secs]
  | 722767.445: [GC [PSYoungGen: 348489K->31737K(349056K)] 803284K->486531K(893824K), 0.0428510 secs]
  | 722890.032: [GC [PSYoungGen: 349049K->37531K(354304K)] 803843K->494879K(899072K), 0.0612990 secs]
  | 723012.030: [GC [PSYoungGen: 343707K->30878K(337088K)] 801055K->495953K(881856K), 0.0499690 secs]
  | 723086.092: [GC [PSYoungGen: 337054K->21900K(360320K)] 802129K->489656K(905088K), 0.0289090 secs]
  | 723099.015: [GC [PSYoungGen: 330252K->25665K(357632K)] 798008K->502118K(902400K), 0.0480880 secs]
  | 723148.879: [GC [PSYoungGen: 334017K->16866K(363328K)] 810470K->497404K(908096K), 0.0297370 secs]
  | 723319.246: [GC [PSYoungGen: 332514K->15691K(361920K)] 813052K->500750K(906688K), 0.0282040 secs]
  | 723440.373: [GC [PSYoungGen: 331339K->15873K(365824K)] 816398K->503180K(910592K), 0.0254350 secs]
  | 723493.820: [GC [PSYoungGen: 336633K->9285K(364544K)] 823940K->504183K(909312K), 0.0247040 secs]
  | 723560.631: [GC [PSYoungGen: 330053K->4302K(365376K)] 824951K->505219K(910144K), 0.0172160 secs]
  | 723651.802: [GC [PSYoungGen: 326094K->9060K(365696K)] 827011K->512253K(910464K), 0.0194620 secs]
  | 723679.932: [GC [PSYoungGen: 330852K->4226K(366912K)] 834045K->508948K(911680K), 0.0156860 secs]
  | 723732.259: [GC [PSYoungGen: 328194K->11940K(366656K)] 832916K->517363K(911424K), 0.0190840 secs]
  | 723815.159: [GC [PSYoungGen: 335908K->3346K(369216K)] 841331K->513974K(913984K), 0.0167600 secs]
  | 724309.119: [GC [PSYoungGen: 331794K->6575K(368832K)] 842422K->518963K(913600K), 0.0176670 secs]
  | 724424.439: [GC [PSYoungGen: 335023K->14426K(372736K)] 847411K->526815K(917504K), 0.0320940 secs]
  | 724710.095: [GC [PSYoungGen: 347994K->17865K(370432K)] 860383K->530254K(915200K), 0.0298030 secs]
  | 724841.867: [GC [PSYoungGen: 351433K->19412K(373824K)] 863822K->531800K(918592K), 0.0324520 secs]
  | 724970.019: [GC [PSYoungGen: 357140K->23179K(360960K)] 869528K->535568K(905728K), 0.0331110 secs]
  | 725108.066: [GC [PSYoungGen: 360907K->31563K(365120K)] 873296K->543951K(909888K), 0.0548900 secs]
  | 725420.214: [GC [PSYoungGen: 359883K->26714K(355072K)] 872271K->539103K(899840K), 0.0404870 secs]
  | 725526.356: [GC [PSYoungGen: 355034K->28427K(364864K)] 867423K->540815K(909632K), 0.0349080 secs]
  | 725665.274: [GC [PSYoungGen: 351499K->30833K(353920K)] 863887K->543222K(898688K), 0.0426310 secs]
  | 725744.423: [GC [PSYoungGen: 353905K->29974K(362240K)] 866294K->542363K(907008K), 0.0393140 secs]
  | 725845.027: [GC [PSYoungGen: 347030K->26052K(363328K)] 859419K->538440K(908096K), 0.0356110 secs]
  | 725988.888: [GC [PSYoungGen: 343095K->28950K(363456K)] 855483K->541339K(908224K), 0.0446460 secs]
  | 726366.280: [GC [PSYoungGen: 346902K->28987K(346944K)] 859291K->541376K(891712K), 0.0422150 secs]
  | 726423.526: [GC [PSYoungGen: 346939K->29693K(363456K)] 859328K->542082K(908224K), 0.0428230 secs]
  | 726484.035: [GC [PSYoungGen: 347261K->46001K(356224K)] 859650K->584352K(900992K), 0.9967360 secs]
  | 726485.032: [Full GC [PSYoungGen: 46001K->0K(356224K)] [PSOldGen: 538350K->261428K(614400K)] 584352K->261428K(970624K) [PSPermGen: 103172K->103172K(524288K)], 1.3802850 secs]
  | 726520.482: [GC [PSYoungGen: 310208K->17830K(335296K)] 571636K->279258K(949696K), 0.0405200 secs]
  | 726549.807: [GC [PSYoungGen: 307110K->12696K(349440K)] 568538K->274124K(963840K), 0.0221810 secs]
  | 726665.639: [GC [PSYoungGen: 301976K->14638K(351552K)] 563404K->276066K(965952K), 0.0203700 secs]
  | 726715.059: [GC [PSYoungGen: 306734K->13990K(350144K)] 568162K->275418K(964544K), 0.0201680 secs]
  | 726823.928: [GC [PSYoungGen: 306086K->15832K(354944K)] 567514K->277261K(969344K), 0.0227490 secs]
  | 726981.850: [GC [PSYoungGen: 314200K->18973K(353024K)] 575629K->280401K(967424K), 0.0253610 secs]
  | 727123.743: [GC [PSYoungGen: 317341K->21522K(360192K)] 578769K->282950K(974592K), 0.0297800 secs]
  | 727179.398: [GC [PSYoungGen: 329490K->15099K(357376K)] 590918K->284088K(971776K), 0.0328880 secs]
  | 727331.399: [GC [PSYoungGen: 323051K->12644K(362688K)] 592041K->285567K(977088K), 0.0233190 secs]
  | 727462.751: [GC [PSYoungGen: 327332K->17594K(361600K)] 600255K->292900K(976000K), 0.0262760 secs]
  | 727475.684: [GC [PSYoungGen: 332282K->12239K(365760K)] 607588K->292784K(980160K), 0.0249070 secs]
  | 727494.195: [GC [PSYoungGen: 333135K->9268K(364736K)] 613680K->293405K(979136K), 0.0173800 secs]
  | 727501.242: [GC [PSYoungGen: 330164K->3708K(366144K)] 614301K->294706K(980544K), 0.0140750 secs]
  | 727581.526: [GC [PSYoungGen: 326652K->4937K(366272K)] 617650K->297328K(980672K), 0.0120750 secs]
  | 727597.015: [GC [PSYoungGen: 327881K->6145K(367872K)] 620272K->302067K(982272K), 0.0142040 secs]
  | 727623.067: [GC [PSYoungGen: 331265K->2651K(366848K)] 627187K->301026K(981248K), 0.0120040 secs]
  | 727652.971: [GC [PSYoungGen: 327771K->3180K(369408K)] 626146K->302589K(983808K), 0.0100680 secs]
  | 727781.496: [GC [PSYoungGen: 331436K->28857K(357120K)] 630845K->329128K(971520K), 0.1631970 secs]
  | 727913.951: [GC [PSYoungGen: 357113K->8854K(368128K)] 657384K->316254K(982528K), 0.0219140 secs]
  | 728271.871: [GC [PSYoungGen: 333526K->41471K(358400K)] 640926K->358192K(972800K), 0.6131290 secs]
  | 728292.904: [GC [PSYoungGen: 358399K->43444K(305216K)] 675120K->401776K(919616K), 1.2681580 secs]
  | 728297.960: [GC [PSYoungGen: 305204K->40972K(302784K)] 663536K->428664K(917184K), 0.1713090 secs]
  | 728323.287: [GC [PSYoungGen: 302732K->5826K(335936K)] 690424K->425406K(950336K), 0.0811750 secs]
  | 728382.527: [GC [PSYoungGen: 266566K->11393K(334464K)] 686146K->436686K(948864K), 0.0270390 secs]
  | 728411.555: [GC [PSYoungGen: 272193K->7008K(340992K)] 697486K->441802K(955392K), 0.0213800 secs]
  | 728463.851: [GC [PSYoungGen: 277216K->10142K(338816K)] 712010K->450254K(953216K), 0.0224220 secs]
  | 728499.645: [GC [PSYoungGen: 280350K->6938K(346240K)] 720462K->455561K(960640K), 0.0224080 secs]
  | 728554.476: [GC [PSYoungGen: 287450K->9700K(343872K)] 736073K->464020K(958272K), 0.0288500 secs]
  | 728584.005: [GC [PSYoungGen: 290212K->6420K(351488K)] 744532K->469177K(965888K), 0.0215800 secs]
  | 728610.463: [GC [PSYoungGen: 297492K->4588K(349184K)] 760249K->472637K(963584K), 0.0156550 secs]
  | 728660.097: [GC [PSYoungGen: 295660K->9622K(356672K)] 763709K->482011K(971072K), 0.0242390 secs]
  | 728711.517: [GC [PSYoungGen: 310294K->11038K(353600K)] 782683K->491650K(968000K), 0.0272260 secs]
  | 728719.928: [GC [PSYoungGen: 311710K->3471K(361472K)] 792322K->494856K(975872K), 0.0207250 secs]
  | 728730.520: [GC [PSYoungGen: 315151K->3259K(359808K)] 806536K->496610K(974208K), 0.0138120 secs]
  | 728788.802: [GC [PSYoungGen: 314939K->9514K(366144K)] 808290K->505069K(980544K), 0.0215790 secs]
  | 728845.645: [GC [PSYoungGen: 329258K->10739K(363200K)] 824813K->515077K(977600K), 0.0311230 secs]
  | 728865.986: [GC [PSYoungGen: 330483K->40575K(334336K)] 834821K->570924K(948736K), 1.1462860 secs]
  | 728871.754: [GC [PSYoungGen: 334335K->54108K(347904K)] 864684K->601289K(962304K), 0.1483570 secs]
  | 728878.963: [GC [PSYoungGen: 347868K->21753K(345216K)] 895049K->617581K(959616K), 0.1718590 secs]
  | 728879.135: [Full GC [PSYoungGen: 21753K->0K(345216K)] [PSOldGen: 595827K->354829K(614400K)] 617581K->354829K(959616K) [PSPermGen: 104992K->104992K(524288K)], 1.5983270 secs]
  | 728899.653: [GC [PSYoungGen: 283520K->37215K(320768K)] 638349K->392045K(935168K), 0.3492760 secs]
  | 728918.256: [GC [PSYoungGen: 320735K->10217K(346880K)] 675565K->402442K(961280K), 0.5435910 secs]
  | 728945.803: [GC [PSYoungGen: 292265K->9302K(344768K)] 684490K->411274K(959168K), 0.0314040 secs]
  | 728975.764: [GC [PSYoungGen: 291350K->8595K(350976K)] 693322K->419715K(965376K), 0.0244910 secs]
  | 729006.600: [GC [PSYoungGen: 298899K->9393K(348928K)] 710019K->428928K(963328K), 0.0301520 secs]
  | 729038.876: [GC [PSYoungGen: 299697K->8508K(355456K)] 719232K->436625K(969856K), 0.0280820 secs]
  | 729070.820: [GC [PSYoungGen: 307644K->9566K(353280K)] 735761K->446041K(967680K), 0.0246880 secs]
  | 729101.295: [GC [PSYoungGen: 308702K->8772K(360064K)] 745177K->453452K(974464K), 0.0247740 secs]
  | 729133.955: [GC [PSYoungGen: 317060K->14892K(357824K)] 761740K->468256K(972224K), 0.0284770 secs]
  | 729166.636: [GC [PSYoungGen: 323180K->9091K(365312K)] 776544K->477280K(979712K), 0.0321650 secs]
  | 729198.603: [GC [PSYoungGen: 328131K->10566K(363328K)] 796320K->487398K(977728K), 0.0288240 secs]
  | 729212.347: [GC [PSYoungGen: 329606K->12273K(369856K)] 806438K->497779K(984256K), 0.0377500 secs]
  | 729218.738: [GC [PSYoungGen: 339958K->2569K(367488K)] 825464K->497843K(981888K), 0.0213970 secs]
  | 729251.948: [GC [PSYoungGen: 330313K->9147K(372352K)] 825587K->506723K(986752K), 0.0215630 secs]
  | 729274.095: [GC [PSYoungGen: 342331K->7121K(370432K)] 839907K->513335K(984832K), 0.0237330 secs]
  | 729310.117: [GC [PSYoungGen: 340305K->8840K(375680K)] 846519K->521039K(990080K), 0.0242150 secs]
  | 729342.766: [GC [PSYoungGen: 348744K->9512K(373824K)] 860943K->530218K(988224K), 0.0252700 secs]
  | 729375.846: [GC [PSYoungGen: 349416K->8688K(379008K)] 870122K->538446K(993408K), 0.0244650 secs]
  | 729407.217: [GC [PSYoungGen: 355568K->8108K(377472K)] 885326K->546186K(991872K), 0.0237820 secs]
  | 729442.302: [GC [PSYoungGen: 354988K->8684K(381824K)] 893066K->554395K(996224K), 0.0265120 secs]
  | 729477.033: [GC [PSYoungGen: 361324K->8271K(380416K)] 907035K->562338K(994816K), 0.0256910 secs]
  | 729513.394: [GC [PSYoungGen: 360911K->8329K(384320K)] 914978K->570616K(998720K), 0.0299910 secs]
  | 729524.400: [GC [PSYoungGen: 366153K->4685K(383104K)] 928440K->575210K(997504K), 0.0234440 secs]
  | 729544.452: [GC [PSYoungGen: 362509K->6128K(385792K)] 933034K->579836K(1000192K), 0.0192980 secs]
  | 729577.415: [GC [PSYoungGen: 367216K->8346K(384896K)] 940924K->588066K(999296K), 0.0249440 secs]
  | 729608.632: [GC [PSYoungGen: 369434K->9989K(388096K)] 949154K->597785K(1002496K), 0.0287180 secs]
  | 729641.388: [GC [PSYoungGen: 375557K->8164K(387072K)] 963353K->605783K(1001472K), 0.0281490 secs]
  | 729667.893: [GC [PSYoungGen: 373732K->7726K(389952K)] 971351K->612764K(1004352K), 0.0265880 secs]
  | 729667.920: [Full GC [PSYoungGen: 7726K->0K(389952K)] [PSOldGen: 605038K->487372K(614400K)] 612764K->487372K(1004352K) [PSPermGen: 105419K->92827K(524288K)], 2.0597030 secs]
  | 729701.832: [GC [PSYoungGen: 369472K->7979K(389120K)] 856844K->495351K(1003520K), 0.0167560 secs]
  | 729734.351: [GC [PSYoungGen: 377451K->17073K(386752K)] 864823K->504445K(1001152K), 0.0254100 secs]
  | 729745.047: [GC [PSYoungGen: 385073K->20796K(379328K)] 872445K->513560K(993728K), 0.0431050 secs]
  | 729764.560: [GC [PSYoungGen: 379324K->11087K(381632K)] 872088K->517136K(996032K), 0.0329690 secs]
  | 729796.895: [GC [PSYoungGen: 371919K->6935K(385216K)] 877968K->523956K(999616K), 0.0280190 secs]
  | 729828.378: [GC [PSYoungGen: 367767K->6810K(386368K)] 884788K->530610K(1000768K), 0.0235260 secs]
  | 729858.540: [GC [PSYoungGen: 369364K->7053K(385792K)] 893164K->537469K(1000192K), 0.0238060 secs]
  | 729883.735: [GC [PSYoungGen: 369613K->6508K(387648K)] 900029K->543595K(1002048K), 0.0228510 secs]
  | 729908.560: [GC [PSYoungGen: 371564K->6932K(387008K)] 908651K->549384K(1001408K), 0.0232160 secs]
  | 729932.640: [GC [PSYoungGen: 371988K->8250K(389312K)] 914440K->556940K(1003712K), 0.0255300 secs]
  | 729962.432: [GC [PSYoungGen: 376314K->6391K(388352K)] 925004K->561673K(1002752K), 0.0231310 secs]
  | 729990.551: [GC [PSYoungGen: 374455K->7013K(390592K)] 929737K->568495K(1004992K), 0.0262600 secs]
  | 730016.136: [GC [PSYoungGen: 377893K->6246K(389888K)] 939375K->574290K(1004288K), 0.0252860 secs]
  | 730040.180: [GC [PSYoungGen: 377126K->9016K(392000K)] 945170K->582512K(1006400K), 0.0260720 secs]
  | 730063.399: [GC [PSYoungGen: 382584K->6326K(391168K)] 956080K->587466K(1005568K), 0.0273610 secs]
  | 730090.776: [GC [PSYoungGen: 379894K->5905K(392896K)] 961034K->592703K(1007296K), 0.0237110 secs]
  | 730121.300: [GC [PSYoungGen: 381713K->6079K(392512K)] 968511K->598442K(1006912K), 0.0246060 secs]
  | 730149.074: [GC [PSYoungGen: 381887K->6104K(393920K)] 974250K->604387K(1008320K), 0.0239800 secs]
  | 730177.402: [GC [PSYoungGen: 383832K->5882K(393408K)] 982115K->610083K(1007808K), 0.0249770 secs]
  | 730177.427: [Full GC [PSYoungGen: 5882K->0K(393408K)] [PSOldGen: 604201K->588573K(614400K)] 610083K->588573K(1007808K) [PSPermGen: 93435K->93435K(524288K)], 1.9472700 secs]
  | 730205.970: [GC [PSYoungGen: 377728K->5760K(394816K)] 966301K->594333K(1009216K), 0.0186180 secs]
  | 730234.811: [GC [PSYoungGen: 385344K->11428K(391040K)] 973917K->600002K(1005440K), 0.0248660 secs]
  | 730264.847: [GC [PSYoungGen: 391012K->15551K(388544K)] 979586K->605987K(1002944K), 0.0324950 secs]
  | 730289.488: [GC [PSYoungGen: 388543K->10954K(391296K)] 978979K->611358K(1005696K), 0.0369580 secs]
  | 730314.527: [GC [PSYoungGen: 383946K->16325K(387840K)] 984350K->616730K(1002240K), 0.0301980 secs]
  | 730338.002: [GC [PSYoungGen: 385989K->19959K(389632K)] 986394K->620870K(1004032K), 0.0342600 secs]
  | 730362.366: [GC [PSYoungGen: 389623K->15156K(382976K)] 990534K->626128K(997376K), 0.0442420 secs]
  | 730362.410: [Full GC [PSYoungGen: 15156K->0K(382976K)] [PSOldGen: 610971K->608688K(614400K)] 626128K->608688K(997376K) [PSPermGen: 93441K->93441K(524288K)], 2.0412260 secs]
  | 730387.080: [Full GC [PSYoungGen: 363008K->0K(382976K)] [PSOldGen: 608688K->612204K(614400K)] 971696K->612204K(997376K) [PSPermGen: 93507K->93507K(524288K)], 2.1644880 secs]
  | 730413.733: [Full GC [PSYoungGen: 363008K->0K(382976K)] [PSOldGen: 612204K->584597K(614400K)] 975212K->584597K(997376K) [PSPermGen: 93507K->92598K(524288K)], 2.3508870 secs]
  | 730439.155: [GC [PSYoungGen: 363008K->4576K(386304K)] 947605K->589174K(1000704K), 0.0187860 secs]
  | 730462.083: [GC [PSYoungGen: 367584K->9218K(387392K)] 952182K->593816K(1001792K), 0.0257920 secs]
  | 730483.717: [GC [PSYoungGen: 373378K->19690K(383872K)] 957976K->604288K(998272K), 0.0350250 secs]
  | 730508.112: [GC [PSYoungGen: 383850K->24441K(377984K)] 968448K->609094K(992384K), 0.0498640 secs]
  | 730528.148: [GC [PSYoungGen: 377977K->28025K(381568K)] 962630K->614992K(995968K), 0.0468880 secs]
  | 730534.685: [GC [PSYoungGen: 381561K->28022K(359680K)] 968528K->622438K(974080K), 0.0560120 secs]
  | 730553.204: [GC [PSYoungGen: 359670K->17515K(370624K)] 954086K->625111K(985024K), 0.0427130 secs]
  | 730553.246: [Full GC [PSYoungGen: 17515K->9715K(370624K)] [PSOldGen: 607596K->614399K(614400K)] 625111K->624115K(985024K) [PSPermGen: 92602K->92602K(524288K)], 1.9814840 secs]
  | 730575.494: [Full GC [PSYoungGen: 331648K->29870K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->644270K(985024K) [PSPermGen: 92602K->92602K(524288K)], 2.0432500 secs]
  | 730596.119: [Full GC [PSYoungGen: 331648K->33310K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->647710K(985024K) [PSPermGen: 92602K->92602K(524288K)], 2.0400970 secs]
  | 730617.445: [Full GC [PSYoungGen: 331648K->3270K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->617670K(985024K) [PSPermGen: 92602K->92587K(524288K)], 2.4190660 secs]
  | 730641.482: [Full GC [PSYoungGen: 331648K->23622K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->638022K(985024K) [PSPermGen: 92588K->92588K(524288K)], 2.0699690 secs]
  | 730661.055: [Full GC [PSYoungGen: 331648K->26973K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->641373K(985024K) [PSPermGen: 92588K->92588K(524288K)], 2.0769410 secs]
  | 730681.612: [Full GC [PSYoungGen: 331648K->32121K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->646520K(985024K) [PSPermGen: 92588K->92588K(524288K)], 2.0867450 secs]
  | 730701.270: [Full GC [PSYoungGen: 331648K->14539K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->628939K(985024K) [PSPermGen: 92588K->92587K(524288K)], 2.4397800 secs]
  | 730722.327: [Full GC [PSYoungGen: 331648K->34468K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->648868K(985024K) [PSPermGen: 92588K->92588K(524288K)], 2.1772900 secs]
  | 730742.334: [Full GC [PSYoungGen: 331648K->37665K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->652065K(985024K) [PSPermGen: 92590K->92590K(524288K)], 2.1053450 secs]
  | 730763.200: [Full GC [PSYoungGen: 331648K->40905K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->655305K(985024K) [PSPermGen: 92590K->92590K(524288K)], 2.1708110 secs]
  | 730784.122: [Full GC [PSYoungGen: 331648K->23081K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->637481K(985024K) [PSPermGen: 92590K->92503K(524288K)], 2.5207070 secs]
  | 730804.270: [Full GC [PSYoungGen: 331648K->42751K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->657151K(985024K) [PSPermGen: 92504K->92504K(524288K)], 2.1371330 secs]
  | 730822.544: [Full GC [PSYoungGen: 331648K->45375K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->659775K(985024K) [PSPermGen: 92504K->92504K(524288K)], 2.1339340 secs]
  | 730841.894: [Full GC [PSYoungGen: 331648K->48099K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->662499K(985024K) [PSPermGen: 92504K->92504K(524288K)], 2.1960780 secs]
  | 730859.098: [Full GC [PSYoungGen: 331648K->31318K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->645718K(985024K) [PSPermGen: 92504K->92498K(524288K)], 2.5426460 secs]
  | 730880.495: [Full GC [PSYoungGen: 331648K->50813K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->665213K(985024K) [PSPermGen: 92498K->92498K(524288K)], 2.1668250 secs]
  | 730899.238: [Full GC [PSYoungGen: 331648K->54095K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->668495K(985024K) [PSPermGen: 92500K->92500K(524288K)], 2.1441620 secs]
  | 730908.894: [Full GC [PSYoungGen: 331648K->60234K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->674634K(985024K) [PSPermGen: 92502K->92502K(524288K)], 2.2139840 secs]
  | 730926.868: [Full GC [PSYoungGen: 331648K->36132K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->650532K(985024K) [PSPermGen: 92506K->92493K(524288K)], 2.5211620 secs]
  | 730947.268: [Full GC [PSYoungGen: 331648K->57596K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->671996K(985024K) [PSPermGen: 92495K->92495K(524288K)], 2.1779790 secs]
  | 730964.208: [Full GC [PSYoungGen: 331648K->60031K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->674431K(985024K) [PSPermGen: 92495K->92495K(524288K)], 2.1885250 secs]
  | 730981.252: [Full GC [PSYoungGen: 331648K->62468K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->676868K(985024K) [PSPermGen: 92496K->92496K(524288K)], 2.1941640 secs]
  | 730998.537: [Full GC [PSYoungGen: 331648K->43822K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->658222K(985024K) [PSPermGen: 92496K->92489K(524288K)], 2.6008400 secs]
  | 731014.412: [Full GC [PSYoungGen: 331648K->63244K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->677644K(985024K) [PSPermGen: 92489K->92489K(524288K)], 2.2107950 secs]
  | 731028.901: [Full GC [PSYoungGen: 331648K->65198K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->679598K(985024K) [PSPermGen: 92539K->92539K(524288K)], 2.2016210 secs]
  | 731045.370: [Full GC [PSYoungGen: 331648K->67733K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->682133K(985024K) [PSPermGen: 92542K->92542K(524288K)], 2.2147930 secs]
  | 731062.587: [Full GC [PSYoungGen: 331648K->51662K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->666062K(985024K) [PSPermGen: 92542K->92542K(524288K)], 2.5754560 secs]
  | 731080.837: [Full GC [PSYoungGen: 331648K->70711K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->685111K(985024K) [PSPermGen: 92542K->92542K(524288K)], 2.2338750 secs]
  | 731096.662: [Full GC [PSYoungGen: 331648K->72815K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->687215K(985024K) [PSPermGen: 92542K->92542K(524288K)], 2.2346160 secs]
  | 731113.877: [Full GC [PSYoungGen: 331648K->58109K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->672509K(985024K) [PSPermGen: 92548K->92548K(524288K)], 2.2678900 secs]
  | 731130.582: [Full GC [PSYoungGen: 331648K->59625K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->674025K(985024K) [PSPermGen: 92548K->92547K(524288K)], 2.6058470 secs]
  | 731146.169: [Full GC [PSYoungGen: 331648K->107124K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->721524K(985024K) [PSPermGen: 92550K->92550K(524288K)], 2.3349050 secs]
  | 731157.200: [Full GC [PSYoungGen: 331648K->95857K(370624K)] [PSOldGen: 614399K->614399K(614400K)] 946047K->710257K(985024K) [PSPermGen: 92562K->92562K(524288K)], 2.2864950 secs]
  | 
  | 

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4240165#4240165

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4240165