6:23 a.m.
> What is DynamicBoost?
I use it for computing boost at runtime - not just a static boost number.
Since my boost can depend on current state of different resources.
For example - I need my articles that are new, pushed higher than the
old one's.
public @interface DynamicBoost {
float coef() default 1.0f;
String transformer() default "number";
String limit() default "identity";
String formula() default "quotient";
String query() default "";
boolean isHQL() default true;
}
public Float getBoost(Object value, DynamicBoost db) {
Transformer t = transformers.get(db.transformer());
double fd = t.toDouble(value);
Limitizer limitizer = limitizers.get(db.limit());
double limit = limitizer.limit(value, db, t);
BoostFormula formula = formulas.get(db.formula());
return formula.calculate(db, fd, limit);
}
> I fixed the concurrency issues in the Lucene event a while back
using a reentrant lock per DirectoryProvider.
:-)
Doing the same thing.
public class SynchLuceneEventListener implements
PostDeleteEventListener, PostInsertEventListener,
PostUpdateEventListener, Initializable {
private Map<String, ExtendedDocumentBuilder> documentBuilders = new
HashMap<String, ExtendedDocumentBuilder>();
private Map<File, IndexWriter> tmpWriterMap = new TreeMap<File,
IndexWriter>();
private boolean initialized;
protected final Log log = LogFactory.getLog(getClass());
/**
* Using JNDI to access DynamicBooster instance.
* Can have problems with serializability.
* @see JBossSynchLuceneEventListener for MBean implementation
*/
protected DynamicBooster lookupDynamicBooster(Properties properties)
throws Exception {
String lookupName = DynamicBooster.JNDI_NAME;
String boosterJndiName =
properties.getProperty(DynamicBooster.PROPERTY_JNDI_NAME);
if (boosterJndiName != null) {
lookupName = boosterJndiName;
}
Context jndiContext = NamingHelper.getInitialContext(properties);
return (DynamicBooster)jndiContext.lookup(lookupName);
}
/**
* No need to synchronize, since this is the only intialized once.
* When used in multiple .par files, they are initialized sequentially.
*/
public void initialize(Configuration cfg) {
if (initialized) return;
try {
final Properties properties = cfg.getProperties();
DirectoryLocker directoryLocker = DirectoryLocker.getInstance();
directoryLocker.initialize(properties);
DynamicBooster dynamicBooster =
lookupDynamicBooster(properties);
Iterator iter = cfg.getClassMappings();
while (iter.hasNext()) {
PersistentClass pc = (PersistentClass)iter.next();
Class mappedClass = pc.getMappedClass();
if (mappedClass != null) {
if (mappedClass.getAnnotation(Indexed.class) != null) {
String entityName = pc.getEntityName();
final ExtendedDocumentBuilder documentBuilder =
new ExtendedDocumentBuilder(entityName, mappedClass);
documentBuilder.setDynamicBooster(dynamicBooster);
documentBuilders.put(entityName, documentBuilder);
File file = documentBuilder.getFile();
try {
IndexWriter iw = tmpWriterMap.get(file);
if (iw == null) {
boolean create = !file.exists();
SynchDirectory sdir =
DirectoryLocker.getInstance().getDirectory(file, create);
try {
iw = new IndexWriter(sdir,
documentBuilder.getAnalyzer(), create);
} finally {
sdir.unlock();
}
tmpWriterMap.put(file, iw);
}
} catch (IOException ioe) {
throw new HibernateException(ioe);
}
log.info("index: " +
documentBuilder.getFile().getAbsolutePath() + " - " + entityName);
}
}
}
initialized = true;
} catch (Exception e) {
throw new HibernateException(e);
} finally {
for (IndexWriter iw : tmpWriterMap.values()) {
LuceneUtils.close(iw);
}
tmpWriterMap.clear();
tmpWriterMap = null;
}
}
public void onPostInsert(PostInsertEvent event) {
final Object entity = event.getEntity();
ExtendedDocumentBuilder builder =
documentBuilders.get(event.getPersister().getEntityName());
if (builder != null) {
add(entity, builder, event.getId());
}
}
public void onPostUpdate(PostUpdateEvent event) {
final Object entity = event.getEntity();
ExtendedDocumentBuilder builder =
documentBuilders.get(event.getPersister().getEntityName());
if (builder != null) {
final Serializable id = event.getId();
remove(builder, id);
add(entity, builder, id);
}
}
public void onPostDelete(PostDeleteEvent event) {
ExtendedDocumentBuilder builder =
documentBuilders.get(event.getPersister().getEntityName());
if (builder != null) {
remove(builder, event.getId());
}
}
private void remove(ExtendedDocumentBuilder builder, Serializable id) {
Term idTerm = builder.getTerm(id);
File file = builder.getFile();
log.info("removing: " + idTerm + ", " + file);
try {
IndexReader reader = null;
SynchDirectory sdir =
DirectoryLocker.getInstance().getDirectory(file);
try {
reader = IndexReader.open(sdir);
reader.delete(idTerm);
} finally {
LuceneUtils.close(reader);
sdir.unlock();
}
} catch (IOException ioe) {
throw new HibernateException(ioe);
}
}
private void add(final Object entity, final ExtendedDocumentBuilder
builder, final Serializable id) {
Document doc = builder.getDocument(entity, id);
File file = builder.getFile();
Analyzer analyzer = builder.getAnalyzer();
log.info("adding: " + doc + ", " + file + ", " +
analyzer.getClass().getName());
try {
IndexWriter writer = null;
SynchDirectory sdir =
DirectoryLocker.getInstance().getDirectory(file);
try {
writer = new IndexWriter(sdir, analyzer, false);
writer.addDocument(doc);
} finally {
LuceneUtils.close(writer);
sdir.unlock();
}
} catch (IOException ioe) {
throw new HibernateException(ioe);
}
}
}
public class SynchDirectory extends Directory {
private static final Log log = LogFactory.getLog(SynchDirectory.class);
private Directory directory;
private String name;
private java.util.concurrent.locks.Lock lock;
public SynchDirectory(Directory directory) {
this.directory = directory;
lock = new ReentrantLock();
}
void lock(String name) {
this.name = name;
log.debug("Locking synch directory: " + name);
lock.lock();
}
public void unlock() {
lock.unlock();
log.debug("Unlocking synch directory: " + name);
this.name = null;
}
public String[] list() throws IOException {
return directory.list();
}
public boolean fileExists(String name) throws IOException {
return directory.fileExists(name);
}
public long fileModified(String name) throws IOException {
return directory.fileModified(name);
}
public void touchFile(String name) throws IOException {
directory.touchFile(name);
}
public void deleteFile(String name) throws IOException {
directory.deleteFile(name);
}
public void renameFile(String from, String to) throws IOException {
directory.renameFile(from, to);
}
public long fileLength(String name) throws IOException {
return directory.fileLength(name);
}
public OutputStream createFile(String name) throws IOException {
return directory.createFile(name);
}
public InputStream openFile(String name) throws IOException {
return directory.openFile(name);
}
public Lock makeLock(String name) {
return directory.makeLock(name);
}
public void close() throws IOException {
directory.close();
}
}
> What is additional POJO handling?
I'm using predetermined HQL to select only Lucene indexable properties,
associations - minimizing the stuff that is pulled out when running a
full new indexation.
public @interface LuceneReadQuery {
String value();
}
@LuceneReadQuery("select new Article(a.id, a.title, a.body, a.intro,
a.subtitle, a.source.name, a.category.id, a.publishDate, a.locale) from
Article a")
I also need different Analyzer instances for different POJOs - some are
localizable, some are plain english text, some are just a bunch of
numbers, ...
public @interface Analyzer {
Class<? extends org.apache.lucene.analysis.Analyzer> analyzerClass();
}
I had some issues when different pojos where indexed in different files
- I needed them to be in the same one. Ok, you can do this by setting
index attribute.
But I needed to handle identity stuff - so that based on given result I
could pull out the right pojo from Session.
doc.add(Field.Keyword(ENTITY_FIELD_NAME, entityName));
doc.add(Field.Keyword(IDENTITY_FIELD_NAME,
createIdentityString(id)));
private String createIdentityString(Serializable id) {
return (entityName + "#" + id);
}
//possible identifying fields
String entityName =
doc.get(ExtendedDocumentBuilder.ENTITY_FIELD_NAME);
if (entityName != null) {
// todo - currently expecting only integer id's
Object entity = session.get(
entityName,
Integer.parseInt(doc.get(ExtendedDocumentBuilder.ID_FIELD_NAME))
);
> PS I remember an old JIRA issue of you talking about a filter
capability. I thing it's a nice idea, would probably make sense after
the core work is stable.
Yep.
Having @Permission annotation - used the following way:
Permission p = currClass.getAnnotation(Permission.class);
if (p != null) {
permissions.add(p.permission());
}
Permissions ps = currClass.getAnnotation(Permissions.class);
if (ps != null) {
permissions.addAll(Arrays.asList(ps.permission()));
}
if (!permissions.isEmpty()) {
doc.add(Field.UnStored(
PERMISSION_FIELD_NAME,
StringHelper.join(",", permissions.iterator()))
);
}
And this filter takes care of applying permissions to Lucene search.
public class SinglePermissionFilter extends Filter {
private String permission;
public SinglePermissionFilter() {
}
public SinglePermissionFilter(String permission) {
this.permission = permission;
}
public BitSet bits(IndexReader reader) throws IOException {
BitSet bits = new BitSet(reader.maxDoc());
String fieldName = ExtendedDocumentBuilder.PERMISSION_FIELD_NAME;
TermDocs td = reader.termDocs(new Term(fieldName, permission));
while(td.next()) {
bits.set(td.doc());
}
return bits;
}
public String getPermission() {
return permission;
}
public void setPermission(String permission) {
this.permission = permission;
}
}
Executing search with permission filter:
Hits hits = searcher.search(sqh.getQuery(), permissionFilter);
Ok, if I remember some other issues that I had - will email them. :-)
Rgds, Ales
Emmanuel Bernard wrote:
Hi Ales
Yes let's move that to the mailing list
What is DynamicBoost?
I've added support for @Boost on both attributes and entity
I have also introduced a Bridge notion that do the translation between
the property and the Lucene field. I'm almost done with that one. This
is very much like the Hibernate Type in it's flexibility and I'll add
some specific annotations support for numeric padding and date resolution
I fixed the concurrency issues in the Lucene event a while back using
a reentrant lock per DirectoryProvider. Did you check the code in
http://anonsvn.jboss.org/repos/hibernate/branches/Lucene_Integration/
I'm interested in seeing some additional issue if any.
What is additional POJO handling?
PS I remember an old JIRA issue of you talking about a filter
capability. I thing it's a nice idea, would probably make sense after
the core work is stable.
Ales Justin wrote:
> Hey,
>
> I extended the initial usage quite a lot - needed some additional stuff.
> Some additional annotations - Boost, DynamicBoost, DateField, ...
> Rewritten current LuceneListener - had problems with concurrency,
> additional pojo handling, ...
>
> I can sum up some stuff (non customized stuff) and send it - if
> you're interested.
> If so - to you or to the mailing list?
>
> Rgds, Ales
>
> ps: coming to JBW Berlin?