Ah, I think I see it.  

In my old System V days, I think we used to call this "your process is eating up all my 'SWAP' ..."  ... which of course was *precious* and the entire basis for  demand-paged Virtual Memory process internals.

So when you say my SHM can grow to 137GB virtual capacity (even on a 7.7GB physical capacity) it is because you are consuming ‘SWAP’ … correct?

 

I see I’ve got 2 new from you in my Inbox.  Probably the answers.

 

(Even if I’ve in fact solved this puzzle, I think I should spin again!  Or buy a vowel?)   :-)

 

From: Ben Cotton [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Saturday, March 8, 2014 7:53 PM
To: 'Peter Lawrey'
Cc: 'Ben Cotton'; 'Justin P Dildy'; 'Dmitry Gordeev'; [hidden email]; [hidden email]
Subject: RE: [infinispan-dev] Infinispan embedded off-heap cache

 

Ø  How you might wonder?

 

You got that right!  Let me study this …  Truth is my first, second nor third reads set off the “light bulb” in my head.

 

Ø  This works because even though I only have 7.7 GB after the OS, I can create a SHM of 137 GB because this only uses 21 MB of actual disk space/memory.

 

Still wondering ‘How?’, Peter.  LOL.   I’m mean I’m sure it is true …. But I have no insight to how it is true.

 

Again, Let me study the code for this one.

 

 

From: Peter Lawrey [[hidden email]]
Sent: Saturday, March 8, 2014 7:41 PM
To: [hidden email]
Cc: Ben Cotton; Justin P Dildy; Dmitry Gordeev; [hidden email]
Subject: Re: [infinispan-dev] Infinispan embedded off-heap cache

 

You might find this example interesting.

 

While SHM is not expandable, this is not as much of a problem as it might seem.  SHM uses virtual memory and leave the OS to map it to real memory as required.  This means you can over allocate extents with a very low cost on Linux.

 

https://github.com/OpenHFT/HugeCollections/blob/master/collections/src/test/java/net/openhft/collections/OSResizesMain.java

 

In the example above I create extents for an SHM which much larger than main memory and it takes a fraction of a second to do this.  How you might wonder? It prints

 

System memory= 7.7 GB, Size of map is 137.5 GB, disk used= 21MB


This works because even though I only have 7.7 GB after the OS, I can create a SHM of 137 GB because this only uses 21 MB of actual disk space/memory.

 

You can freely over allocate the size on the basis that the system only uses the resources it needs.

 

On 8 March 2014 04:06, Ben Cotton <[hidden email]> wrote:


Thanks Peter.  The plan is for Dmitry and I to at first extend VanillaSharedHashMap and groom it into an ISPN7 join via their DataContainer API bridge.

That ExtendedVSHM will be morphed into a fully inter-operable JCACHE operand will

- initially be brokered by the ISPN 7 config  (JSR107 <----> VSHMExtendedDataContainer <----> VSHM)
- eventually, possibly, be rendered with ExtendedVSHM directly implementing javax.cache.Cache (in addititon to DataContainer)

 

On 03/07/2014 11:43 AM, Peter Lawrey wrote:

In the medium term I would see SHM supporting a DataContainer. If a Cache were supported I would do it as a layered class so those who don't need the functionality of a Cache don't incur an overhead.

On 8 Mar 2014 03:35, "Ben Cotton" <[hidden email]> wrote:

Thank you for this insight Mircea ...

Ultimately ... I want the OpenHFT SHM off-heap operand to behave *exactly* like a JCACHE ... Amenable to being soundly/completely operated upon by any/all parts of ISPN7's Impl of the JSR-107 API .

 

Musing openly:  Won't that (eventually) necessitate me physically implementing javax.cache.Cache ?

>  Another way to do it is to have CacheImpl implement the DataContainer only, and then configure Infinispan's JCache implementation to use that custom DataContainer.

I see what you mean.  OK, for sure, this sounds much simpler than what I have put on my initial TODO list.

Question:  Will doing it this way  in any manner suggest that my JSR-107 specific operators are  being transitively "brokered" by the ISPN config onto my OpenHFT SHM operand?  If possible, I want everything to be direct -- no API bridge.

Thanks again, Mircea.

-Ben and Dmitry


Sent from my iPhone

On 03/07/2014 09:45 AM, Mircea Markus-2 [via Infinispan Developer List] wrote:

Hi Ben,

In the diagram provided, the CacheImpl (your class) extends both from javax.cache.Cache and org.infinispan.container.DataContainer.
The Cache and DataContainer interfaces are quite different and I anticipate an single class implementing both to be hard to follow and potentially not very efficient.
Another way to do it is to have CacheImpl implement the DataContainer only, and then configure Infinispan's JCache implementation to use that custom DataContainer.

On Mar 3, 2014, at 3:46 PM, cotton-ben <[hidden email]> wrote:


> Quick Update:
>
> It is my understandng that Peter Lawrey will make available  an OpenHFT HC
> Alpha Release in Maven Central next weekend. At that time, Dmitry Gordeev
> and I will take the OpenHFT dependency tag and proceed to build a branch of
> Red Hat's ISPN 7 that will treat net.openhft.collections.SharedHashMap as a
> Red Hat Infinispan 7 default impl of a fully JSR-107 interoperable off-heap
> javax.cache.Cache ...
>
> A diagram of this build effort can be found here:
> https://raw.github.com/Cotton-Ben/OpenHFT/master/doc/AdaptingOpenHFT-SHM-as-JCACHE-Impl.jpg
> ...
>
> The Red Hat view of his effort will be tracked here:
> https://issues.jboss.org/browse/ISPN-871  ...
>
> The code that defines the Impl will be here
> https://github.com/Cotton-Ben/infinispan ...
>
>
>
>
>
> --
> View this message in context: http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infinispan-embedded-off-heap-cache-tp4026102p4028931.html
> Sent from the Infinispan Developer List mailing list archive at Nabble.com.
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


To start a new topic under Infinispan Developer List, email [hidden email]
To unsubscribe from [infinispan-dev] Infinispan embedded off-heap cache, click here.
NAML

 

 

 



View this message in context: RE: [infinispan-dev] Infinispan embedded off-heap cache
Sent from the Infinispan Developer List mailing list archive at Nabble.com.