[infinispan-dev] Multi tenancy support for Infinispan

Sebastian Laskawiec slaskawi at redhat.com
Mon May 16 01:02:48 EDT 2016 

Hey Sanne!

Comments inlined.

Thanks
Sebastian

On Sun, May 15, 2016 at 11:27 PM, Sanne Grinovero <sanne at infinispan.org>
wrote:

> Hi Sebastian,
>
> the design seems to assume that what people want is to have multiple
> cache containers, one per tenant.
> Did you consider the tradeoffs comparing to a solution in which you
> have a single cache container to manage all caches, but isolate
> tenants by having each one access only the subset of caches it is
> owning?
>

This approach was the first I crossed out from my list mainly due to
isolation, name clashes (but the they are easy to solve - just prefix cache
name with tenant) and configuration on Cache Manager level (some tenants
might want to use different authentication settings, marshallers etc).


> I haven't thought about all implications, but it seems desirable that
> all caches - from all tenants - could be managed as a whole. For
> example in future one might want to know how the memory consumption is
> being balanced across different tenants and caches, and have some
> smart policies around such concepts. Were would such logic live? It
> seems like there is a need for a global coordination of resources
> across all caches, and so far this has been the CacheManager.
>

Our roadmap contains a health check endpoint. We might aggregate them and
create some scaling policies based on aggregated data from all
CacheManagers.

Regarding to the memory consumption I've seen it implemented the other way
around (you can measure how much memory you have by `cat
/sys/fs/cgroup/memory/memory.limit_in_bytes` and using it for constructing
-Xmx). This way your container will never go beyond the memory limit. I
believe this is not an ideal way but definitely the easiest.


> You could change this, but then an higher level component will be
> needed to orchestrate the various CacheManager instances at server
> level.
>

Yes, the Router could do that. But I would assume that's the health check
feature and not multi tenancy.


> Similarly, different Caches will need to share some resources; I would
> expect for example that when you want to run "Infinispan as a
> Service", you'd want to also provide the option of enabling some
> popular CacheStores in an easy way for the end user (like a checkbox,
> as simple as "enable JDBC cachestore" or even higher level "enable
> persistent backup").

Taking for example the JDBC CacheStore, I think you'd not want to
> create a new database instance dynamically for each instance but
> rather have them all share the same, so adding a tenant-id to the key,
> but also having the JDBC connection pool to this database shared
> across all tenants.
>

Those points seem to be valid but again we assume similar configuration for
all clients in hosted Infinispan service. This may not always be true (as I
pointed out - settings on CacheManager level) and I would prefer to have a
configuration flexibility here. We may also address some of the resource
consumption/performance issues on the Cloud layer e.g. add a MySQL DB to
each Infinispan pod - this way all DB connection will be local to the
machine which runs the containers.

I realize that this alternative approach will have you face some other
> issues - like adding tenant-aware capabilities to some CacheStore
> implementations - but sharing and managing the resources is crucial to
> implement multi-tenancy: if we don't, why would you not rather start
> separate instances of the Infinispan server?
>

I think Tristan had similar question about starting Infinispan server
instances and maybe I didn't emphasize it enough in the design.

The goal of adding a router is to allow configuring and starting a
CacheManager without starting Netty server. The Netty server will be
started only in the Router and it will "borrow" handlers from given
Protocol Server.

However point granted for Cache Store resources utilization - if all our
tenants want to have a JDBC Cache Store than we might create lots of
connections to the database. But please note that we will have some
problems on Cache Store level this way or another (if we decided to
implement multi tenancy as sharing the same CacheManager than we would need
to solve data isolation problem). The router approach at least guarantees
us isolation which is a #1 priority for me in multi tenancy. I'm just
thinking - maybe we should enhance Cloud Cache Store (the name fits ideally
here) to deal with such situation and recommend it for our clients as the
best tool for storing multi tenant data?


>
> Thanks,
> Sanne
>
>
>
> On 13 May 2016 at 14:51, Sebastian Laskawiec <slaskawi at redhat.com> wrote:
> > Hey guys!
> >
> > Any last call on this? I'm going to start the implementation on Monday.
> >
> > Thanks
> > Sebastian
> >
> > On Wed, May 11, 2016 at 10:38 AM, Sebastian Laskawiec <
> slaskawi at redhat.com>
> > wrote:
> >>
> >> Hey Tristan!
> >>
> >> If I understood you correctly, you're suggesting to enhance the
> >> ProtocolServer to support multiple EmbeddedCacheManagers (probably with
> >> shared transport and by that I mean started on the same Netty server).
> >>
> >> Yes, that also could work but I'm not convinced if we won't loose some
> >> configuration flexibility.
> >>
> >> Let's consider a configuration file -
> >> https://gist.github.com/slaskawi/c85105df571eeb56b12752d7f5777ce9, how
> for
> >> example use authentication for CacheContainer cc1 (and not for cc2) and
> >> encryption for cc1 (and not for cc1)? Both are tied to
> hotrod-connector. I
> >> think using this kind of different options makes sense in terms of multi
> >> tenancy. And please note that if we start a new Netty server for each
> >> CacheContainer - we almost ended up with the router I proposed.
> >>
> >> The second argument for using a router is extracting the routing logic
> >> into a separate module. Otherwise we would probably end up with several
> >> if(isMultiTenent()) statements in Hotrod as well as REST server.
> Extracting
> >> this has also additional advantage that we limit changes in those
> modules
> >> (actually there will be probably 2 changes #1 we should be able to
> start a
> >> ProtocolServer without starting a Netty server (the Router will do it in
> >> multi tenant configuration) and #2 collect Netty handlers from
> >> ProtocolServer).
> >>
> >> To sum it up - the router's implementation seems to be more complicated
> >> but in the long run I think it might be worth it.
> >>
> >> I also wrote the summary of the above here:
> >>
> https://github.com/infinispan/infinispan/wiki/Multi-tenancy-for-Hotrod-Server#alternative-approach
> >>
> >> @Galder - you wrote a huge part of the Hot Rod server - I would love to
> >> hear your opinion as well.
> >>
> >> Thanks
> >> Sebastian
> >>
> >>
> >>
> >> On Tue, May 10, 2016 at 10:59 AM, Tristan Tarrant <ttarrant at redhat.com>
> >> wrote:
> >>>
> >>> Not sure I like the introduction of another component at the front.
> >>>
> >>> My original idea for allowing the client to choose the container was:
> >>>
> >>> - with TLS: use SNI to choose the container
> >>> - without TLS: enhance the PING operation of the Hot Rod protocol to
> >>> also take the server name. This would need to be a requirement when
> >>> exposing multiple containers over the same endpoint.
> >>>
> >>>  From a client API perspective, there would be no difference between
> the
> >>> above two approaches: just specify the server name and depending on the
> >>> transport, select the right one.
> >>>
> >>> Tristan
> >>>
> >>> On 29/04/2016 17:29, Sebastian Laskawiec wrote:
> >>> > Dear Community,
> >>> >
> >>> > Please have a look at the design of Multi tenancy support for
> >>> > Infinispan
> >>> > [1]. I would be more than happy to get some feedback from you.
> >>> >
> >>> > Highlights:
> >>> >
> >>> >   * The implementation will be based on a Router (which will be built
> >>> >     based on Netty)
> >>> >   * Multiple Hot Rod and REST servers will be attached to the router
> >>> >     which in turn will be attached to the endpoint
> >>> >   * The router will operate on a binary protocol when using Hot Rod
> >>> >     clients and path-based routing when using REST
> >>> >   * Memcached will be out of scope
> >>> >   * The router will support SSL+SNI
> >>> >
> >>> > Thanks
> >>> > Sebastian
> >>> >
> >>> > [1]
> >>> >
> >>> >
> https://github.com/infinispan/infinispan/wiki/Multi-tenancy-for-Hotrod-Server
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > infinispan-dev mailing list
> >>> > infinispan-dev at lists.jboss.org
> >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >
> >>>
> >>> --
> >>> Tristan Tarrant
> >>> Infinispan Lead
> >>> JBoss, a division of Red Hat
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160516/3e739188/attachment-0001.html

More information about the infinispan-dev mailing list