[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch commented on ISPN-1103: ------------------------------------- I completely agree that storing metadata as a JSON structure is a brilliant approach. I've added several features the list in the description. Storing the jsonObject as a string might work, especially if it is simply loaded and stored as an atomic unit. However, there are several advantages to storing it in a BSON (or BSON-like) representation, and all boil down to the fact that any schema-aware service will need to access the document contents for indexing, validation, computing differences (for DeltaAware functionality), and even applying changes (e.g., something similar to MongoDB's [atomic operations|http://www.mongodb.org/display/DOCS/Atomic+Operations]). Using a BSON representation will also make it more natural to handle binary values within the document. Now, if we did store the document internally as BSON, we actually don't need to store the metadata separately. For example, we can store a single document with this structure: {code} { "document" : /* user's document */ "metadata" : { "schema-ref" : blah-blah ... } {code} If represented as BSON, then the user's document is actually a nested BSON object stored under the "document" property name. This approach means that all the differencing, atomic operations, serialization, etc. functionality will work without having to distinguish between a user's document and a system metadata document. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch edited comment on ISPN-1103 at 5/13/11 5:57 PM: -------------------------------------------------------------- I completely agree that storing metadata as a JSON structure is a brilliant approach. I've added several features the list in the description. Storing the jsonObject as a string might work, especially if it is simply loaded and stored as an atomic unit. However, there are several advantages to storing it in a BSON (or BSON-like) representation, and all boil down to the fact that any schema-aware service will need to access the document contents for indexing, validation, computing differences (for DeltaAware functionality), and even applying changes (e.g., something similar to MongoDB's [atomic operations|http://www.mongodb.org/display/DOCS/Atomic+Operations]). Using a BSON representation will also make it more natural to handle binary values within the document. Now, if we did store the document internally as BSON, we actually don't need to store the metadata separately. For example, we can store a single document with this structure: {code} { "document" : /* user's document */ "metadata" : { "schema-ref" : blah-blah ... } {code} If represented as BSON, then the user's document is actually a nested BSON object stored under the "document" property name. This approach means that all the differencing, atomic operations, serialization, etc. functionality will work without having to distinguish between a user's document and a system metadata document. The resulting value class would be: {code} class SchematicValue { BSON json; } {code} was (Author: rhauch): I completely agree that storing metadata as a JSON structure is a brilliant approach. I've added several features the list in the description. Storing the jsonObject as a string might work, especially if it is simply loaded and stored as an atomic unit. However, there are several advantages to storing it in a BSON (or BSON-like) representation, and all boil down to the fact that any schema-aware service will need to access the document contents for indexing, validation, computing differences (for DeltaAware functionality), and even applying changes (e.g., something similar to MongoDB's [atomic operations|http://www.mongodb.org/display/DOCS/Atomic+Operations]). Using a BSON representation will also make it more natural to handle binary values within the document. Now, if we did store the document internally as BSON, we actually don't need to store the metadata separately. For example, we can store a single document with this structure: {code} { "document" : /* user's document */ "metadata" : { "schema-ref" : blah-blah ... } {code} If represented as BSON, then the user's document is actually a nested BSON object stored under the "document" property name. This approach means that all the differencing, atomic operations, serialization, etc. functionality will work without having to distinguish between a user's document and a system metadata document. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch edited comment on ISPN-1103 at 5/13/11 6:07 PM: -------------------------------------------------------------- I completely agree that storing metadata as a JSON structure is a brilliant approach. I've added several features the list in the description. Storing the jsonObject as a string might work, especially if it is simply loaded and stored as an atomic unit. However, there are several advantages to storing it in a BSON (or BSON-like) representation, and all boil down to the fact that any schema-aware service will need to access the document contents for indexing, validation, computing differences (for DeltaAware functionality), and even applying changes (e.g., something similar to MongoDB's [atomic operations|http://www.mongodb.org/display/DOCS/Atomic+Operations]). Using a BSON representation will also make it more natural to handle binary values within the document. Now, if we did store the document internally as [BSON|http://www.bson.org], we actually don't need to store the metadata separately. For example, we can store a single document with this structure: {code} { "document" : /* user's document */ "metadata" : { "schema-ref" : blah-blah ... } {code} If represented as a [BSONObject|https://github.com/mongodb/mongo-java-driver/blob/master/src/m...], then the user's document is actually a nested BSON object stored under the "document" property name. This approach means that all the differencing, atomic operations, serialization, etc. functionality will work without having to distinguish between a user's document and a system metadata document. The resulting value class would be: {code} class SchematicValue { org.bson.BSONObject json; } {code} The downside of doing this is the increase in size of the {{SchematicValue}} class, from 2 references to a 1 reference plus an extra BSONObject implementation. The {{org.bson.BasicBSONObject}} extends {{LinkedHashMap<String,Object>}} (which is fairly substantial), but we could probably provide an optimized implementation for the top-level that still implemented the {{org.bson.BSONObject}} interface but more directly. Is it worth the apparent simplicity? was (Author: rhauch): I completely agree that storing metadata as a JSON structure is a brilliant approach. I've added several features the list in the description. Storing the jsonObject as a string might work, especially if it is simply loaded and stored as an atomic unit. However, there are several advantages to storing it in a BSON (or BSON-like) representation, and all boil down to the fact that any schema-aware service will need to access the document contents for indexing, validation, computing differences (for DeltaAware functionality), and even applying changes (e.g., something similar to MongoDB's [atomic operations|http://www.mongodb.org/display/DOCS/Atomic+Operations]). Using a BSON representation will also make it more natural to handle binary values within the document. Now, if we did store the document internally as BSON, we actually don't need to store the metadata separately. For example, we can store a single document with this structure: {code} { "document" : /* user's document */ "metadata" : { "schema-ref" : blah-blah ... } {code} If represented as BSON, then the user's document is actually a nested BSON object stored under the "document" property name. This approach means that all the differencing, atomic operations, serialization, etc. functionality will work without having to distinguish between a user's document and a system metadata document. The resulting value class would be: {code} class SchematicValue { org.bson.BSONObject json; } {code} The downside of doing this is the increase in size of the {{SchematicValue}} class, from 2 references to a 1 reference plus an extra BSONObject implementation. The {{org.bson.BasicBSONObject}} extends {{LinkedHashMap<String,Object>}}, but we could probably provide an optimized implementation for the top-level that still implemented the {{org.bson.BSONObject}} interface but more directly. Is it worth the apparent simplicity? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Manik Surtani updated ISPN-1103: -------------------------------- Fix Version/s: 5.2.0.Final (was: 5.1.0.BETA1) (was: 5.1.0.Final) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Manik Surtani updated ISPN-1103: -------------------------------- Fix Version/s: 5.1.0.BETA1 5.1.0.Final (was: 5.2.0.Final) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch commented on ISPN-1103: ------------------------------------- Here's a still-incomplete prototype of a new 'schematic' module, and I'm looking for feedback as to whether this is going in the right direction: https://github.com/rhauch/infinispan/tree/ISPN-1103/schematic The new 'schematic' module contains a small document-oriented API layer on top of Infinispan and supports storing JSON and BSON documents and using JSON Schema documents to validate the structure of the documents. Rather than store the documents as strings within the Infinispan values, the value consists of a single Document with functionality that's a superset of BSON and JSON, with the goal being very efficient access and modification of the documents. Values are serialized using the BSON binary format representation. Documents are also immutable, with a simple "editable document" mechanism to allow the Infinispan values to "listen" for changes and to support DeltaAware functionality. I had originally started to use other JSON and BSON implementations, but was not happy with the APIs. JSON libraries do not natively support binary values and other types, while BSON libraries were less mature. No JSON or BSON libraries had support for JSON Schema, though there is a relatively simplistic and mostly-complete add-on for the Jackson JSON library. Finally, implementing our own also means that we could provide an immutable API for general document access while still making it easy to create and edit documents such that DeltaAware and transaction functionality will work cleanly. Perhaps it's useful/desirable to extract the JSON/BSON/JSON Schema functionality into a separate module or even a separate project (as it is likely useful in projects other than Infinispan). If so, please say so. But at this time I'm not worrying about that. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch commented on ISPN-1103: ------------------------------------- The design has been evolving, and I've been pushing (overwriting) new versions of the branch. Here's a summary of the basic design: The primary goal is to enable storing dynamically-structured values with metadata, and to also enable describing the structure of each value (and metadata) using a schema-based approach. [JSON|http://json.org] documents provide an excellent way to offer structure that is extremely flexible, while [JSON Schema|http://json-schema.org/] offers a way to define the structure of JSON documents in a way that can be easily validated. (Note that a JSON Schema is just a JSON document that conforms to the JSON meta-schema, which is rich enough to be self-describing. It's actually a very nice specification.) Manik originally suggested storing the metadata and value (henceforth referred to as 'content') as strings, but doing so would mean that in order to access any information within the metadata or content, the JSON strings would first need to be parsed into an in-memory representation. Plus, if the content is to be modified, the JSON document would need to be modified and written as a string before being stored. This parsing and writing would become prohibitive. Since Infinispan is essentially an large heap of memory, it makes far more sense to represent the content and metadata as _in-memory documents_, as long as the in-memory representation were compatible with JSON, were easy to use, and could be validated using JSON Schemas. Additionally, if the representation also supported [BSON|http://bsonspec.org] data types (e.g., binary values, UUIDs, dates, regular expressions, etc.), more types of user-content could be supported (including just raw binary data). These in-memory documents could at any time be read from or written to JSON or BSON formats. Having the schematic values be _delta-aware_ with _fine-grained locking_ (see ISPN-1115) would provide significant advantages w/r/t performance and concurrency. (Note that efficient support for delta-aware means that the schematic value can capture the changes made to the documents by client application and use those changes as the delta, rather than having to compare the changed document to a prior version to compute the changes.) Using an in-memory representation also means that the content and metadata need not be stored as separate objects, but could instead be represented by a single document that is conceptually: {code} { "metadata" : { /* metadata as a nested document */ } "content" : /* user's content, as a nested document or binary value */ } {code} This is the approach taken by the current design. The primary packages are: * org.infinispan.schematic * org.infinispan.schematic.document * org.infinispan.schematic.internal.* The first two packages contain the public API, whereas all implementation-specific classes are contained within the "internal" packages. The primary API interfaces are: * SchematicDb - similar to Cache but tailored to make it easy for users to store a content document (or binary value) with a metadata document. Each SchematicDb has a JSON Schema library, and providing a map-reduce-based validation mechanism. Internally this uses a Cache<String,SchematicEntry>. * SchematicEntry - the value actually stored within Infinispan, and which contains a content object (that is a Document or a Binary value) and a metadata Document. There are methods for getting a mutable interface to the content document and metadata documents. Since tracking the MIME type of the content is likely very common, the SchematicEntry interface provides methods for getting and setting the MIME type (which is actually stored in the metadata. * Document - an immutable interface to an in-memory document * EditableDocument - a mutable interface to an in-memory document * Json - utility class for parsing JSON formatted streams/files into Document instances, and for writing Document instances as JSON * Bson - utility class for parsing BSON formatted streams/files into Document instances, and for writing Document instances as BSON * JsonSchema - utility class for working with JSON Schemas * Various interfaces for reprenting JSON/BSON values: Array, Binary, Symbol, Timestamp, Code, CodeWithScope The current status is that this works for LOCAL mode, but additional work is required before DISTRIBUTED and REPLICATED modes will work correctly with delta-aware and [fine-grained locking|MODE-1115]. As always, feedback is appreciated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch commented on ISPN-1103: ------------------------------------- It's just using Infinispan's Map-Reduce functionality to validate each document using the JSON Schema referenced by the document. It should distribute just fine, because a) validation is dependent only upon the document (the value in Infinispan) itself and the schema registry, which can be serialized. The {{SchematicDb.validateAll()}} is the method that uses Infinispan Map-Reduce and is [here|https://github.com/rhauch/infinispan/blob/ISPN-1103/schematic/src/ma...], with the Mapper implementation [here|https://github.com/rhauch/infinispan/blob/ISPN-1103/schematic/src/ma...] and the Reducer implementation [here|https://github.com/rhauch/infinispan/blob/ISPN-1103/schematic/src/ma...]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Galder Zamarreño updated ISPN-1103: ----------------------------------- Fix Version/s: 5.1.0.BETA3 (was: 5.1.0.BETA2) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Galder Zamarreño updated ISPN-1103: ----------------------------------- Fix Version/s: 5.1.0.BETA5 (was: 5.1.0.BETA4) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1103?page=com.atlassian.jira.plugin.... ] Randall Hauch updated ISPN-1103: -------------------------------- Fix Version/s: 5.2.0.FINAL (was: 5.1.0.FINAL) (was: 5.1.0.CR1) After discussing this with Manik, we think it's best to push this to the 5.2 release. It simply won't be ready to include in the 5.1 release, and in the mean time it is living and getting thorough usage and testing in the [ModeShape repository|https://github.com/ModeShape/modeshape/tree/3.x/modeshape-sche...], where it is serving as the primary store for JCR content for the [ModeShape 3.0|http://community.jboss.org/message/634352#634352]. Therefore, retargeting to the 5.2 release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira