protostream library
by Adrian Nistor
Hi devs,
the protostream library that was mentioned in previous emails on this
list regarding "protobuf as a marshaling protocol for remote query" now
has a home, here: https://github.com/infinispan/protostream.
It is by no means closed and complete, although it already works fine
for current purposes. The approach is still discussed and you are
welcome to have a look at it and join the discussion.
Adrian
10 years, 9 months
Something wrong in the build order?
by Sanne Grinovero
I was doing a clean build on an empty system, so I suspect the Maven
order is having some weirdness causing this:
[INFO] Reactor Summary:
[INFO]
[INFO] Infinispan BOM .................................... SUCCESS [0.485s]
[INFO] Infinispan Common Parent .......................... SUCCESS [3.339s]
[INFO] Infinispan Commons ................................ SUCCESS [12.924s]
[INFO] Infinispan Core ................................... SUCCESS [51.556s]
[INFO] Infinispan Extended Statistics .................... SUCCESS [11.652s]
[INFO] Parent pom for server modules ..................... SUCCESS [3.136s]
[INFO] Infinispan Server - Core Components ............... SUCCESS [50.031s]
[INFO] Parent pom for cachestore modules ................. SUCCESS [1.068s]
[INFO] Infinispan JDBC CacheStore ........................ SUCCESS [3.142s]
[INFO] Parent pom for the Lucene integration modules ..... SUCCESS [0.766s]
[INFO] Infinispan integration with Lucene v.3.x .......... SUCCESS [2.477s]
[INFO] Infinispan integration with Lucene v.4.x .......... SUCCESS [1.646s]
[INFO] Infinispan Lucene Directory Implementation ........ SUCCESS [1.896s]
[INFO] Infinispan Query API .............................. SUCCESS [5.505s]
[INFO] Infinispan Tools .................................. SUCCESS [2.873s]
[INFO] Infinispan JDBM CacheStore ........................ SUCCESS [3.151s]
[INFO] Infinispan Tree API ............................... SUCCESS [3.035s]
[INFO] Infinispan BDBJE CacheStore ....................... SUCCESS [2.281s]
[INFO] Infinispan CloudCacheStore ........................ SUCCESS [4.656s]
[INFO] Infinispan Hot Rod Server ......................... SUCCESS [58.436s]
[INFO] Infinispan Hot Rod Client ......................... FAILURE [6.072s]
[INFO] Infinispan remote CacheStore ...................... SKIPPED
[INFO] Infinispan CassandraCacheStore .................... SKIPPED
[INFO] Infinispan HBaseCacheStore ........................ SKIPPED
[INFO] Infinispan JPA CacheStore ......................... SKIPPED
[INFO] Infinispan LevelDB CacheStore ..................... SKIPPED
[INFO] Infinispan Memcached Server ....................... SKIPPED
[INFO] Infinispan WebSocket Server ....................... SKIPPED
[INFO] Infinispan REST Server ............................ SKIPPED
[INFO] Infinispan RHQ Plugin ............................. SKIPPED
[INFO] Infinispan Rolling Upgrade Tooling ................ SKIPPED
[INFO] Infinispan Spring Integration ..................... SKIPPED
[INFO] Infinispan QL Interpreter ......................... SKIPPED
[INFO] Infinispan CLI Client ............................. SKIPPED
[INFO] Infinispan GUI Demo ............................... SKIPPED
[INFO] Infinispan EC2 Demo ............................... SKIPPED
[INFO] Infinispan Distributed Executors and Map/Reduce Demo SKIPPED
[INFO] Infinispan EC2 Demo UI ............................ SKIPPED
[INFO] Infinispan Directory Demo ......................... SKIPPED
[INFO] Infinispan Lucene Directory Demo .................. SKIPPED
[INFO] Infinispan GridFileSystem WebDAV interface ........ SKIPPED
[INFO] Infinispan Near Cache Demo ........................ SKIPPED
[INFO] Infinispan JCACHE (JSR-107) implementation ........ SKIPPED
[INFO] Infinispan CDI support ............................ SKIPPED
[INFO] Infinispan Near Cache Demo Client ................. SKIPPED
[INFO] Infinispan AS/EAP modules ......................... SKIPPED
[INFO] Integration tests: Lucene Directory with Infinispan Query SKIPPED
[INFO] Integration tests: AS Module Integration Tests .... SKIPPED
[INFO] Integration tests: Infinispan compatibility mode .. SKIPPED
[INFO] Infinispan Distribution ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:52.887s
[INFO] Finished at: Tue Jul 16 10:17:58 BST 2013
[INFO] Final Memory: 94M/1003M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project infinispan-client-hotrod:
Could not resolve dependencies for project
org.infinispan:infinispan-client-hotrod:bundle:6.0.0-SNAPSHOT: Failed
to collect dependencies for
[org.infinispan:infinispan-commons:jar:6.0.0-SNAPSHOT (compile),
org.jboss.marshalling:jboss-marshalling-river:jar:1.3.15.GA (compile),
commons-pool:commons-pool:jar:1.6 (compile),
org.apache.hadoop:avro:jar:1.3.3 (provided),
org.infinispan:infinispan-core:jar:tests:6.0.0-SNAPSHOT (test),
org.infinispan:infinispan-server-hotrod:jar:6.0.0-SNAPSHOT (test),
org.infinispan:infinispan-server-hotrod:jar:tests:6.0.0-SNAPSHOT
(test), org.infinispan:infinispan-server-core:jar:tests:6.0.0-SNAPSHOT
(test), org.scala-lang:scala-library:jar:2.10.2 (test),
org.infinispan.protostream:protostream:jar:1.0.0.Alpha1 (compile),
org.infinispan.protostream:sample-domain-definition:jar:1.0.0.Alpha1
(test), com.thoughtworks.xstream:xstream:jar:1.4.1 (test),
commons-logging:commons-logging:jar:1.1 (test), log4j:log4j:jar:1.2.16
(compile?), net.jcip:jcip-annotations:jar:1.0 (compile?),
org.mockito:mockito-all:jar:1.9.5 (test),
org.jboss.jbossts.jta:narayana-jta:jar:4.17.4.Final (test),
org.jboss.logging:jboss-logging-processor:jar:1.0.3.Final (provided),
org.slf4j:slf4j-log4j12:jar:1.6.4 (test), org.testng:testng:jar:6.8
(test)]: Failed to read artifact descriptor for
org.infinispan.protostream:protostream:jar:1.0.0.Alpha1: Could not
find artifact org.infinispan.protostream:parent:pom:1.0.0.Alpha1 in
redhat-earlyaccess-repository-group
(http://maven.repository.redhat.com/earlyaccess/all/) -> [Help 1]
10 years, 9 months
WriteSkewException: Write skew detected on key
by Matej Lazar
Testing CapeDwarf in a cluster I get WriteSkewException.
Any idea what could cause this ? Config issue ?
Matej.
15:39:25,171 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-1) Exception executing command: org.infinispan.transaction.WriteSkewException: Write skew detected on key UPDATE(2)/__entity_group__(1) for transaction null
at org.infinispan.transaction.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:99)
at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.createNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:478)
at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitPrepareCommand(VersionedEntryWrappingInterceptor.java:73)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:116)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:109)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:130)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:117)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:118)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:58)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:118)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:118)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:147)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:137)
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
10 years, 9 months
Integrating Karsten's FCS
by Galder Zamarreño
Hi,
I think we all agree that Karsten's file cache store is a good base replacement for the current file cache store, particularly for caches with relatively small keys, or not a huge amount of them.
I'm working with Radim to try to figure out what would be the tipping point at which LevelDB based cache store behaves better compared to Karsten's cache store.
In the mean time, I think we should integrate Karsten's FCS into master and have it ready for people to try from Alpha1. There's one or two things that do require a bit of review, but I can pin point them in the pull req.
The question is, how should we name it? And what should we do about the current FCS implementation?
Shall we keep the current FCS implementation, deprecate it, and get rid of it in the next minor/major version? Some users might have data stored in the current FCS and would be quite abrupt to just get rid of it right now.
If so, what do we name karsten's file cache store? AppendOnlyFileCacheStore?
Cheers,
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
10 years, 10 months
L1 Consistency with Sync Caches
by William Burns
First off I apologize for the length.
There have been a few Jiras recently that have identified L1 consistency
issues with both TX and non TX sync caches. Async caches with L1 have
their own issues as well, but I only wanted to talk about sync caches.
https://issues.jboss.org/browse/ISPN-3197
https://issues.jboss.org/browse/ISPN-2965
https://issues.jboss.org/browse/ISPN-2990
I have proposed a solution in
https://github.com/infinispan/infinispan/pull/1922 which should start L1
consistency down the right track. There are quite a few comments on it if
you want to look into it more, but because of that I am moving this to the
dev mailing list.
The key changes in the PR are the following (non-tx):
1. Concurrent reads for a key that can retrieve a remote value are
"corralled" into a single thread of execution for that given key. This
would reduce network traffic with concurrent gets for the same key. Note
the "corralling" only happens on a per key basis.
2. The single thread that is doing the remote get would update the L1 if
able (without locking) and make available the value to all the requests
waiting on the get.
3. Invalidations that are received would first check to see if there is a
current remote get occurring for it's keys. If there is it will attempt to
cancel the L1 write(s) before it occurs. If it cannot cancel the L1 write,
then it must also wait on the current remote get completion and
subsequently run the invalidation. Note the cancellation would fail when
the remote get was done and it is in the middle of updating the L1, so this
would be very small window.
4. Local writes will also do the same thing as the invalidation with
cancelling or waiting. Note that non tx local writes only do L1
invalidations and don't write the value to the data container. Reasons why
I found at https://issues.jboss.org/browse/ISPN-3214
5. Writes that require the previous value and don't have it in the L1 would
also do it's get operations using the same "corralling" method.
4/5 are not currently implemented in PR.
This approach would use no locking for non tx caches for all L1 operations.
The synchronization point would be done through the "corralling" method
and invalidations/writes communicating to it.
Transactional caches would do almost the same thing as non-tx. Note these
changes are not done in any way yet.
1. Gets would now update the L1 immediately after retrieving the value
without locking, but still using the "corralling" technique that non-tx
does. Previously the L1 update from a get was transactional. This
actually would remedy issue [1]
2. Writes currently acquire the remote lock when committing, which is why
tx caches are able to update the L1 with the value. Writes would do the
same cancellation/wait method as non-tx.
3. Writes that require the previous value and don't have it in the L1 would
also do it's get operations using the same method.
4. For tx cache [2] would also have to be done.
[1] -
https://issues.jboss.org/browse/ISPN-2965?focusedCommentId=12779780&page=...
[2] - https://issues.jboss.org/browse/ISPN-1540
Also rehashing is another issue, but we should be able to acquire the state
transfer lock before updating the L1 on a get, just like when an entry is
committed to the data container.
Any comments/concerns would be appreciated.
Thanks,
- Will
10 years, 10 months
WriteSkewTest help
by Pedro Ruivo
Hi guys,
I need your help to find what is the objective of this
WriteSkewTest.testWriteSkewWithOnlyPut.
Is it to test the write skew check in Local mode caches?
Cheers,
Pedro Ruivo
10 years, 10 months
protobuf as a marshalling format for infinispan remote-query
by Adrian Nistor
Hi all,
as some probably know protobuf was chosen as the serialization format
for the remote query [1]. Them main reason for choosing it was that is
/simple/, time tested, has good support for schema evolution, is nearly
ubiquitous (some people say google uses it :)), has multiple language
support and most importantly it mandates the existence of a schema for
our objects - the proto file. We need that schema on the server side to
be able to extract indexable fields from those binary cache values and
index them without the need to unmarshall them into plain java domain
objects.
And I need to stress that the format was chosen, rather than the actual
API/library provided by Google [2]. While the protobuf wire format is
superb, Google's approach to create a library for marshaling objects
to/from a protobuf stream is heavily based on code generation via the
protoc tool. Both the marshalling code /and the entities/ to be
marshaled (your beloved domain model) are generated. This does not work
well if you want to bring your own domain classes to the party.
So what we did is create a small set of support classes on top of
google's low level wire format classes to assist users in marshaling
their own domain model to the protobuf wire format without using
google's code generator. These attempts are hosted on a small github
project experiment [3] that will be moved to infinispan once we have a
conclusion. This project contains several modules demonstrating the
attempts. [4] explains the purpose of each module. So far, the approach
in module /stream-like/ offers the best user experience. I would name
this plan A, which is going to be implemented for infinispan 6.0.
Plan A can also use the classes generated by google's protoc code
generator tool. So if somebody prefers to go old school it can still
work well. Also, the amount of new support code we added on top of
google's library is small, so I don't foresee any nightmare in porting
this to another language.
I would like to get some feedback from anyone who has some time to have
a look at [3], specifically at the /stream-like/ module.
Stream-like is about 89.9% implemented. There are still some
unimplemented methods but do not mind them for the review.
And finally, if anyone could suggest a better name for the stream-like
package? I can't think of any other option except /streamlike/ :) (which
might be trade-marked) So any other options? If no options then we'll
just call it /marshaling/. I'll move this to the ispn branch once I know
the package name :)
Have a nice weekend guys!
--------------
[1] https://community.jboss.org/wiki/RemoteQueryDesignInInfinispan
[2] https://developers.google.com/protocol-buffers/
[3] https://github.com/anistor/protobuf-playground
[4] https://github.com/anistor/protobuf-playground/blob/master/README.md
10 years, 10 months
