Re: [infinispan-dev] Infinispan embedded off-heap cache

Ah, I think I see it. In my old System V days, I think we used to call this "your process is eating up all my 'SWAP' ..." ... which of course was *precious* and the entire basis for demand-paged Virtual Memory process internals. So when you say my SHM can grow to 137GB virtual capacity (even on a 7.7GB physical capacity) it is because you are consuming ‘SWAP’ … correct? I see I’ve got 2 new from you in my Inbox. Probably the answers. (Even if I’ve in fact solved this puzzle, I think I should spin again! Or buy a vowel?) :-) From: Ben Cotton [mailto:bendcotton@gmail.com] On Behalf Of ben.cotton(a)alumni.rutgers.edu Sent: Saturday, March 8, 2014 7:53 PM To: 'Peter Lawrey' Cc: 'Ben Cotton'; 'Justin P Dildy'; 'Dmitry Gordeev'; ml-node+s980875n4028967h94(a)n3.nabble.com; ben.cotton(a)alumni.rutgers.edu Subject: RE: [infinispan-dev] Infinispan embedded off-heap cache Ø How you might wonder? You got that right! Let me study this … Truth is my first, second nor third reads set off the “light bulb” in my head. Ø This works because even though I only have 7.7 GB after the OS, I can create a SHM of 137 GB because this only uses 21 MB of actual disk space/memory. Still wondering ‘How?’, Peter. LOL. I’m mean I’m sure it is true …. But I have no insight to how it is true. Again, Let me study the code for this one. From: Peter Lawrey [mailto:peter.lawrey@higherfrequencytrading.com] Sent: Saturday, March 8, 2014 7:41 PM To: ben.cotton(a)alumni.rutgers.edu Cc: Ben Cotton; Justin P Dildy; Dmitry Gordeev; ml-node+s980875n4028967h94(a)n3.nabble.com Subject: Re: [infinispan-dev] Infinispan embedded off-heap cache You might find this example interesting. While SHM is not expandable, this is not as much of a problem as it might seem. SHM uses virtual memory and leave the OS to map it to real memory as required. This means you can over allocate extents with a very low cost on Linux. https://github.com/OpenHFT/HugeCollections/blob/master/collections/src/test/ java/net/openhft/collections/OSResizesMain.java In the example above I create extents for an SHM which much larger than main memory and it takes a fraction of a second to do this. How you might wonder? It prints System memory= 7.7 GB, Size of map is 137.5 GB, disk used= 21MB This works because even though I only have 7.7 GB after the OS, I can create a SHM of 137 GB because this only uses 21 MB of actual disk space/memory. You can freely over allocate the size on the basis that the system only uses the resources it needs. On 8 March 2014 04:06, Ben Cotton <ben.cotton(a)alumni.rutgers.edu&gt; wrote: Thanks Peter. The plan is for Dmitry and I to at first extend VanillaSharedHashMap and groom it into an ISPN7 join via their DataContainer API bridge. That ExtendedVSHM will be morphed into a fully inter-operable JCACHE operand will - initially be brokered by the ISPN 7 config (JSR107 <----> VSHMExtendedDataContainer <----> VSHM) - eventually, possibly, be rendered with ExtendedVSHM directly implementing javax.cache.Cache (in addititon to DataContainer) On 03/07/2014 11:43 AM, Peter Lawrey wrote: In the medium term I would see SHM supporting a DataContainer. If a Cache were supported I would do it as a layered class so those who don't need the functionality of a Cache don't incur an overhead. On 8 Mar 2014 03:35, "Ben Cotton" <ben.cotton(a)alumni.rutgers.edu&gt; wrote: Thank you for this insight Mircea ... Ultimately ... I want the OpenHFT SHM off-heap operand to behave *exactly* like a JCACHE ... Amenable to being soundly/completely operated upon by any/all parts of ISPN7's Impl of the JSR-107 API . Musing openly: Won't that (eventually) necessitate me physically implementing javax.cache.Cache ? only, and then configure Infinispan's JCache implementation to use that custom DataContainer. I see what you mean. OK, for sure, this sounds much simpler than what I have put on my initial TODO list. Question: Will doing it this way in any manner suggest that my JSR-107 specific operators are being transitively "brokered" by the ISPN config onto my OpenHFT SHM operand? If possible, I want everything to be direct -- no API bridge. Thanks again, Mircea. -Ben and Dmitry Sent from my iPhone On 03/07/2014 09:45 AM, Mircea Markus-2 [via Infinispan Developer List] wrote: Hi Ben, In the diagram provided, the CacheImpl (your class) extends both from javax.cache.Cache and org.infinispan.container.DataContainer. The Cache and DataContainer interfaces are quite different and I anticipate an single class implementing both to be hard to follow and potentially not very efficient. Another way to do it is to have CacheImpl implement the DataContainer only, and then configure Infinispan's JCache implementation to use that custom DataContainer. On Mar 3, 2014, at 3:46 PM, cotton-ben <[hidden email] <http://user/SendEmail.jtp?type=node&node=4028967&i=0> > wrote: of a off-heap https://raw.github.com/Cotton-Ben/OpenHFT/master/doc/AdaptingOpenHFT-SHM-as- JCACHE-Impl.jpg http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infinis pan-embedded-off-heap-cache-tp4026102p4028931.html Nabble.com. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) _______________________________________________ infinispan-dev mailing list [hidden email] <http://user/SendEmail.jtp?type=node&node=4028967&i=2> https://lists.jboss.org/mailman/listinfo/infinispan-dev _____ If you reply to this email, your message will be added to the discussion below: http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infinis pan-embedded-off-heap-cache-tp4026102p4028967.html To start a new topic under Infinispan Developer List, email ml-node+s980875n2085493h0(a)n3.nabble.com To unsubscribe from [infinispan-dev] Infinispan embedded off-heap cache, click here <http://infinispan-developer-list.980875.n3.nabble.com/template/NamlServlet. jtp?macro=unsubscribe_by_code&node=4026102&code=YmVuLmNvdHRvbkBBTFVNTkkuUlVU R0VSUy5FRFV8NDAyNjEwMnwtMTU2ODA0NTc1OA==> . <http://infinispan-developer-list.980875.n3.nabble.com/template/NamlServlet. jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.nam l.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble. view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aem ail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3A email.naml> NAML -- View this message in context: http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infi... Sent from the Infinispan Developer List mailing list archive at Nabble.com.