[overlord-issues] [JBoss JIRA] (ARTIF-683) Switch to RDBMS, JPA (Hibernate), Hibernate Search, and Hibernate 2nd Level Cache as the default persistence solution

[ https://issues.jboss.org/browse/ARTIF-683?page=com.atlassian.jira.plugin.... ] Brett Meyer edited comment on ARTIF-683 at 5/12/15 11:01 AM: ------------------------------------------------------------- [~hxp], I should clarify: this isn't currently *replacing* the JCR plugin. For the time being, we'll continue to support that and you'll be able to continue using it. But this new setup will be the new *default*. Further, in the future, I might still consider removing JCR altogether. That boils down to community needs, maintenance, and performance (ie, if there are situations JCR has an edge on JPA, great, but otherwise, there may not be reasons to keep it). I'm obviously biased (I'm a committer and one of the core devs of Hibernate ORM), but this new setup has been a long time coming and has a large amount of benefits: - Horizontal scaling: Several potential users have brought up their typical enterprise-level use cases, and they all involved primary artifacts on the order of *one million*. Once derivation kicks in, that could look more like a hundred million, maybe more. I had a lot of concerns about JCR's ability to handle queries with that much data. RDBMS + proper indexing + in-mem caching has a proven track record... - CQRS pattern: Being able to query-by-column using pure SQL, rather than pulling out entire nodes (ala JCR), has a large amount of benefits on its own (http://martinfowler.com/bliki/CQRS.html). - Full-text search: Fully-indexed, full-text search is improving in ModeShape, but it's not quite there. Using Hibernate Search and Lucene opens the doors to several new features. - BLOB storage: I've actually added a new SPI that allows both SQL BLOB or the filesystem to be used for binary storage. For most use cases, BLOBs tend to have several pros over filesystem, so at least having it as an option is beneficial. Note that this SPI would also allow for new plugins: Ceph, Gluster, etc. - Flexibility: Right out of the gate, this tremendously increases what we're able to support. Any RDBMS (that's supported by Hibernate), any Hibernate caching solution (not limited to Infinispan), etc. - Less external dependencies: nearly the entire stack is already included in Wildfly/EAP out of the box - Less moving parts - Potential increase in community developers: Bluntly, JPA is more widely understood and used, as opposed to JCR. I'm hoping this opens the doors to additional community members. {quote} Use JPA {quote} FWIW, I am using pure JPA, rather than Hibernate ORM. Although, the driving force behind that is mainly the JPA Criteria Query API... {quote} Then Hiberante OGM comes essentially for free. {quote} Definitely a fair point. OGM/NoSQL, Cassandra, GraphDBs, etc. could certainly be added as additional plugins, if the community wants them (and is willing to help develop them ;) {quote} so your whole argument really only boils down to a matter of runtime efficiencies (storage space & execution speed). {quote} Simply not true -- there's *a lot* more to it {quote} I strongly suggest you see Artificer as a relatively high level functional application platform for business organizations and technical organizations, rather than yet-another-lower-level-engine (of which there are hundreds and hundreds). While space and time efficiency are always important, they don't make the top 10 list of what CIOs (should, and do) consider critical – such as reliability, manageability, scalability, security, longevity, stability, developer ease, etc. So architect with the key longterm open standards {quote} This is an absolutely vital point, and one that I focus on. Artificer needs to continue to become a powerful, out of the box application. It certainly needs to support developers and be usable as a powerful platform, but in the end, focusing on it as an *application* is where it will shine. But I'd argue that you're contradicting yourself, at least a bit. Through this tried-and-true stack, the "reliability, manageability, scalability, security, longevity, stability" increase across the board. And 80-90% of users will see the storage solution as simply an internal implementation detail. The new stack *does not change* current Artificer functionality. {quote} Instead of stripdown hotrodding, go Architectural {quote} Again, you're focusing only on the performance considerations, which are only a small piece of the decision. Realistically, we have a *long* backlog of tasks and new features to add, many of which would at least be difficult to support with JCR. A flexible architecture is definitely one area being gained... {quote} Most importantly, stick with open standards, so you let your users (like me) make our own best decision of what to put behind the standard interface. {quote} With JPA, that's what you'd be getting... {quote} and then you can have nice integrity between the 2 paradigms {quote} I have to say, I fully agree with Randall and Horia's points on https://developer.jboss.org/message/928792. Mixing JPA with JCR doesn't make much sense. {quote} IMHO, you ought to have a chat with Randall and figure out an optimal architecture. {quote} Myself, Randall, and several other committers on the Hibernate/Infinispan teams have been involved in these discussions and architecture. That should probably have been more in the open... I definitely value alternative opinions -- don't want this to sound like I'm just outright shooting you down. But, just wanted to clarify how this has shaped up. Feel free to continue the discussion! was (Author: brmeyer): [~hxp], I should clarify: this isn't *replacing* the JCR plugin. For the time being, we'll continue to support that and you'll be able to continue using it. However, this new setup will be the new *default*. I'm obviously biased (I'm a committer and one of the core devs of Hibernate ORM), but this new setup has been a long time coming and has a large amount of benefits: - Horizontal scaling: Several potential users have brought up their typical enterprise-level use cases, and they all involved primary artifacts on the order of *one million*. Once derivation kicks in, that could look more like a hundred million, maybe more. I had a lot of concerns about JCR's ability to handle queries with that much data. RDBMS + proper indexing + in-mem caching has a proven track record... - CQRS pattern: Being able to query-by-column using pure SQL, rather than pulling out entire nodes (ala JCR), has a large amount of benefits on its own (http://martinfowler.com/bliki/CQRS.html). - Full-text search: Fully-indexed, full-text search is improving in ModeShape, but it's not quite there. Using Hibernate Search and Lucene opens the doors to several new features. - BLOB storage: I've actually added a new SPI that allows both SQL BLOB or the filesystem to be used for binary storage. For most use cases, BLOBs tend to have several pros over filesystem, so at least having it as an option is beneficial. Note that this SPI would also allow for new plugins: Ceph, Gluster, etc. - Flexibility: Right out of the gate, this tremendously increases what we're able to support. Any RDBMS (that's supported by Hibernate), any Hibernate caching solution (not limited to Infinispan), etc. - Less external dependencies: nearly the entire stack is already included in Wildfly/EAP out of the box {quote} Use JPA {quote} FWIW, I am using pure JPA, rather than Hibernate ORM. Although, the driving force behind that is mainly the JPA Criteria Query API... {quote} Then Hiberante OGM comes essentially for free. {quote} Definitely a fair point. OGM/NoSQL, Cassandra, GraphDBs, etc. could certainly be added as additional plugins, if the community wants them (and is willing to help develop them ;) {quote} so your whole argument really only boils down to a matter of runtime efficiencies (storage space & execution speed). {quote} Simply not true -- there's *a lot* more to it {quote} I strongly suggest you see Artificer as a relatively high level functional application platform for business organizations and technical organizations, rather than yet-another-lower-level-engine (of which there are hundreds and hundreds). While space and time efficiency are always important, they don't make the top 10 list of what CIOs (should, and do) consider critical – such as reliability, manageability, scalability, security, longevity, stability, developer ease, etc. So architect with the key longterm open standards {quote} This is an absolutely vital point, and one that I focus on. Artificer needs to continue to become a powerful, out of the box application. It certainly needs to support developers and be usable as a powerful platform, but in the end, focusing on it as an *application* is where it will shine. But I'd argue that you're contradicting yourself, at least a bit. Through this tried-and-true stack, the "reliability, manageability, scalability, security, longevity, stability" increase across the board. And 80-90% of users will see the storage solution as simply an internal implementation detail. {quote} Instead of stripdown hotrodding, go Architectural {quote} Again, you're focusing only on the performance considerations, which are only a small piece of the decision. Realistically, we have a *long* backlog of tasks and new features to add, many of which would at least be difficult to support with JCR. A flexible architecture is definitely one area being gained... {quote} Most importantly, stick with open standards, so you let your users (like me) make our own best decision of what to put behind the standard interface. {quote} With JPA, that's what you'd be getting... {quote} and then you can have nice integrity between the 2 paradigms {quote} I have to say, I fully agree with Randall and Horia's points on https://developer.jboss.org/message/928792. Mixing JPA with JCR doesn't make much sense. {quote} IMHO, you ought to have a chat with Randall and figure out an optimal architecture. {quote} Myself, Randall, and several other committers on the Hibernate/Infinispan teams have been involved in these discussions and architecture. That should probably have been more in the open... I definitely value alternative opinions -- don't want this to sound like I'm just outright shooting you down. But, just wanted to clarify how this has shaped up. Feel free to continue the discussion! -- This message was sent by Atlassian JIRA (v6.3.15#6346)