sqlite support -------------- Key: ISPN-2980 URL: https://issues.jboss.org/browse/ISPN-2980 Project: Infinispan Issue Type: Feature Request Components: Loaders and Stores Affects Versions: 5.2.5.Final Reporter: Aleksandar Kostadinov Assignee: Mircea Markus Labels: cache-loader, cache-store, jdbc, sqlite Attachments: sqlite-jdbc-3.7.15-SNAPSHOT-f18.jar It would be very nice is we have SQLite support for infinispan. SQLite is a powerful database supporting terabyte sized databases in a file with competitive performance. I tried to use it as a JDBC store but the best driver I find in the internet ([xerial sqlite jdbc driver|https://bitbucket.org/xerial/sqlite-jdbc]) does not implement full jdbc specification and trying to use it results in exceptions. I think that perhaps using the [non-jdbc wrapper sqlite4java|http://code.google.com/p/sqlite4java/] may make sense for infinispan because: 1. it promises better performance 2. it allows using the sqlite library from OS (xerial driver uses a customized build of sqlite) FYI here is how I setup sqlite for infinispan (unsuccessfully): {code}jboss as cli commands: /subsystem=datasources/jdbc-driver=sqlite:add(driver-name="sqlite",driver-module-name="org.xerial",driver-class-name=org.sqlite.JDBC) data-source add --name=SQLiteDS --connection-url="jdbc:sqlite:${sqlite.database.string}" --jndi-name=java:jboss/datasources/SQLiteDS --driver-name="sqlite" /subsystem=datasources/data-source=SQLiteDS:enable {code} {code}JBoss AS module definition (modules/org/xerial/main/module.xml): <?xml version="1.0" encoding="UTF-8"?> <module xmlns="urn:jboss:module:1.0" name="org.xerial"> <resources> <resource-root path="sqlite-jdbc.jar" /> </resources> <dependencies> <module name="javax.api" /> <module name="javax.transaction.api"/> </dependencies> </module> {code} {code}cache store/loader configuration snippet: <stringKeyedJdbcStore xmlns="urn:infinispan:config:jdbc:5.2" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false" key2StringMapper="com.jboss.datagrid.chunchun.util.TwoWayKey2StringChunchunMapper"> <dataSource jndiUrl="java:jboss/datasources/SQLiteDS" /> <stringKeyedTable dropOnExit="false" createOnStart="true" prefix="ISPN_STRING_TABLE"> <idColumn name="ID_COLUMN" type="VARCHAR(255)" /> <dataColumn name="DATA_COLUMN" type="BINARY" /> <timestampColumn name="TIMESTAMP_COLUMN" type="BIGINT" /> </stringKeyedTable> </stringKeyedJdbcStore> </loaders> {code} sql driver needs to be copied in the same directory as module.xml The Exception I'm getting is:{code} 12:53:10,683 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-3) ISPN000136: Execution error: org.infinispan.loaders.CacheLoaderException: Error while storing string key to database; key: 'user41', buffer size of value: 4918 bytes at org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore.storeLockSafe(JdbcStringBasedCacheStore.java:253) [infinispan-cachestore-jdbc-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore.storeLockSafe(JdbcStringBasedCacheStore.java:87) [infinispan-cachestore-jdbc-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.loaders.AbstractCacheStore.applyModifications(AbstractCacheStore.java:126) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.loaders.AbstractCacheStore.commit(AbstractCacheStore.java:163) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.CacheStoreInterceptor.commitCommand(CacheStoreInterceptor.java:164) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.CacheStoreInterceptor.visitCommitCommand(CacheStoreInterceptor.java:146) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:116) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:101) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommitCommand(TransactionSynchronizerInterceptor.java:73) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.statetransfer.StateTransferInterceptor.visitCommitCommand(StateTransferInterceptor.java:121) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.5.Final.jar:5.2.5.Final] at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:167) at com.jboss.datagrid.chunchun.jsf.InitializeCache.startup(InitializeCache.java:125) [classes:] at com.jboss.datagrid.chunchun.jsf.InitializeCache.processEvent(InitializeCache.java:76) [classes:] at javax.faces.event.SystemEvent.processListener(SystemEvent.java:106) [jboss-jsf-api_2.1_spec-2.0.7.Final-redhat-1.jar:2.0.7.Final-redhat-1] at com.sun.faces.application.ApplicationImpl.processListeners(ApplicationImpl.java:2169) [jsf-impl-2.1.13-redhat-1.jar:2.1.13-redhat-1] at com.sun.faces.application.ApplicationImpl.invokeListenersFor(ApplicationImpl.java:2145) [jsf-impl-2.1.13-redhat-1.jar:2.1.13-redhat-1] at com.sun.faces.application.ApplicationImpl.publishEvent(ApplicationImpl.java:303) [jsf-impl-2.1.13-redhat-1.jar:2.1.13-redhat-1] at org.jboss.as.weld.webtier.jsf.ForwardingApplication.publishEvent(ForwardingApplication.java:288) [jboss-as-weld-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at com.sun.faces.config.ConfigManager.publishPostConfigEvent(ConfigManager.java:602) [jsf-impl-2.1.13-redhat-1.jar:2.1.13-redhat-1] at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:371) [jsf-impl-2.1.13-redhat-1.jar:2.1.13-redhat-1] at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:223) [jsf-impl-2.1.13-redhat-1.jar:2.1.13-redhat-1] at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:3392) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.apache.catalina.core.StandardContext.start(StandardContext.java:3850) [jbossweb-7.0.17.Final-redhat-1.jar:] at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:89) [jboss-as-web-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] Caused by: java.sql.SQLException: not implemented by SQLite JDBC driver at org.sqlite.Unused.unused(Unused.java:29) [sqlite-jdbc-3.7.2.jar:] at org.sqlite.Unused.setBinaryStream(Unused.java:60) [sqlite-jdbc-3.7.2.jar:] at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.setBinaryStream(WrappedPreparedStatement.java:871) at org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore.storeLockSafe(JdbcStringBasedCacheStore.java:247) [infinispan-cachestore-jdbc-5.2.5.Final.jar:5.2.5.Final] ... 73 more{code} The driver [does not support|http://code.google.com/p/xerial/issues/detail?id=99] setBinaryStream(), only setBytes(). Not sure if there are any other methods required by infinispan but not implemented. As a simple comparison between JDBC and direct storage, I tried an app that caches 3000 records of around 5k and 60000 records of around 0.5k (total of less than 60MiB). Bdbje store operation completes in less than a minute. With a local mysql server it takes 10 minutes. And this is on a machine with plenty of CPU and memory over an SSD. Unfortunately bdbje does not work clustered for me (ISPN-2968). So my point is that a local disk based, fast, reliable, transactional engine is highly needed.