<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Jul 4, 2008, at 21:06, Sanne Grinovero wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Yes what you formalized is exactly what I meant; there's an additional point in the<br>"piggyback" strategy, which is to actually use some volatile read (or any other memorybarrier)<br>which you couldn't avoid anyway, to avoid the cost of reading a volatile.<br> No need to make it excessively complex of course, but if we could reuse some other<br>barrier it would just come free.<br>This is also the drawback if it is not properly commented: the code would completely hide<br>the fact it is providing some memory barrier idiom, it could look-like you are aquiring some<br> lock you need or read some needed field and forget to synch some more data.<br><br>What I mean with the "readeProviders and DPProviders" issue is that I am sure I need to do something like<br>this. You just told me I can't trust the SearchFactory initialization: "(especially Containers can do stupid things)",<br> I didn't know that and I wasn't sure we had a problem with initialization too; I'll rewrite my opinion as<br>"don't know if it's needed with Search initialization as I don't know how that happens, but<br> I'm quite sure we need some more locking in ReadeProviders and DPProviders".<br><br>After reading your post I think we concluded we need it everywhere;<br><br>IMHO a good candidate to become volatile is<br>"protected SearchFactoryImplementor searchFactoryImplementor"<br> in FullTextIndexEventListener, but it will hurt performance, even if minimal;<br>if we could replace that initialize with a similar constructor it<br>would be "free" as in Zurich's beer.</blockquote><div><br></div><div>Can't do that. Listeners can be instantiated by a user and passed to the configuration instance. At that time the configuration is not in a good state to proceed with initialization. Plus generally speaking, it breaks the symmetry of initialize / cleanup</div><div><br></div><blockquote type="cite"><br>This would involve an "extension" in the Hibernate listeners initialization,<br> but as you are releasing a new version it shouldn't be much of a problem?</blockquote><div><br></div><div>see above</div><br><blockquote type="cite"><br><br>I could commit my proposed fix for the ReadeProviders and DPProviders,<br>so you can look at it and get an idea of the little change: there's just<br> and happen-once reflection usage to see if an appropriate constructor<br>is available to be used as replacement for initialize();</blockquote><div><br></div><div>Can you send a patch I can look at? I'm curious to see what your actual gains are as you can't move the start() method in the constructor. Regardless of this patch, we need to build the piggyback strategy.</div><br><blockquote type="cite"><br><br>Also if you look at the work I had done on SearchFactoryImpl you'll<br>notice that just 2 fields are not final yet, I converted most already.<br> <br>Sanne<br><br><div class="gmail_quote">2008/7/5 Emmanuel Bernard <<a href="mailto:emmanuel@hibernate.org">emmanuel@hibernate.org</a>>:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div style="">Hibernate Search cannot guarantee that threads using the SearchFActory have been started after the thread initializing the SearchFactory (especially Containers can do stupid things).<div>So you need to do someting like that:</div> <div><br></div><div>SF.init() {</div><div> ...</div><div> myvolatile++; //write</div><div>}</div><div><br></div><div>and upon access to the SearchFactory</div><div> //note the thread local variable is per search factory instance</div> <div> if ( searchFactory.getThreadLocalCheck().get() == null) {</div><div> //note the thread local variable is per search factory instance</div><div> boolean checked = myVolatile != 0;</div><div> searchFactory.getThreadLocalCheck().put(checked);</div> <div>}</div><div><br></div><div>If I understand correctly, this ensures that any threat Tn will see what Ti (the initialization thread) has written by acquiring the "lock" only once. (this is the formalization of what you proposed in Zurich). Someone corrects me if I'm wrong.</div> <div><br></div><div>But you scare me with "In the readeProviders and DPProviders I can't get this guarantee". What do you mean? initialize and start are always called before the SearchFactory initialization ends.</div> <div><br></div><div>I don't see the code being that scary, the sync issue is dealt by the framework without exposing the user to the "piggy" details.</div><div><div class="Ih2E3d"><br><div> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style=""> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style=""> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style=""> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style=""> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style=""> <div>--</div><div>Emmanuel Bernard</div><div><a href="http://in.relation.to/Bloggers/Emmanuel" target="_blank">http://in.relation.to/Bloggers/Emmanuel</a> | <a href="http://blog.emmanuelbernard.com" target="_blank">http://blog.emmanuelbernard.com</a> | <a href="http://twitter.com/emmanuelbernard" target="_blank">http://twitter.com/emmanuelbernard</a></div> <div>Hibernate Search in Action (<a href="http://is.gd/Dl1" target="_blank">http://is.gd/Dl1</a>)</div></div></span></div></span></div></span></div></span></div></span> </div><br></div><div><div></div><div class="Wj3C7c"> <div><div>On Jul 4, 2008, at 18:36, Sanne Grinovero wrote:</div><br><blockquote type="cite">Thanks for your blog Pavitar;<br>I would like to add some clarification about the "piggyback" just to confirm:<br>there is no such concept as "THE shared memory" in the JMM, shared memory is about memory shared between some threads, not necessarily all.<br> the "piggyback" trick works basing on this: after ThreadB reads a volatile variable, it is guaranteed to see at least ALL state TrheadA had written before (and during)<br>a write to THAT SAME volatile variable. Note the "at least" wording: more changes could happen to the other variables after the write to the volatile, and some (in no order,<br> especially not the code order) could be also seen, but no guarantee about.<br>so you could:<br>write field1, field2, field3 and then the volatile field4 in T1<br>read volatile field4, field1, field2, field3 in T2 (after T1 did)<br> and you will get a guarantee that T2 will "see" at least the state written by T1.<br><br>So this is a "trick" to avoid longer locks or having to convert them all to volatile, but IMHO<br>the code is made difficult to maintain, and tricky to get it right.<br> <br>In our practical case:<br>you could write to some volatile field in the SearchFactoryImpl after the initialization is done,<br>but then you still have to ensure all subsequent uses will read the same field before anything else;<br> this has a minimal impact on performance, the good think about the "piggyback" is you<br>could use a read to a volatile you would have anyway.<br>I don't think this is at all needed for the SearchFactoryImpl as long as you guarantee that<br> the threads going to use it are started ("start()") after and by the initialization thread;<br>if this is correct no further discussion is needed there.<br>In the readeProviders and DPProviders I can't get this guarantee, that's why they need a fix.<br> <br>Shall I use this trick then? It isn't so bad if you think it's accepatable to use it, it's good for performance<br>but I dislike it for code readability; I'll add a big fat scary warning.<br>IMHO this should be avoided when possible, especially since "final" works fine and is<br> very explicit to another code reader.<br><br>If you think I should go for the pig, I would appreciate if Pavitar could read the code after I commit it<br>to check my code, even if this case is trivial.<br><br>Sanne<br><br> <div class="gmail_quote">2008/7/4 Emmanuel Bernard <<a href="mailto:emmanuel@hibernate.org" target="_blank">emmanuel@hibernate.org</a>>:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div>Hey,<div>Can you tell me more about the piggyback synchronization. I could not find any decent knowledge online.</div><div>how far reading a volatile guarantee that all "local" values of the thread we are reading from will be pushed to the shared memory?</div> <div>For example, could reading a volatile value after HSearch is done with initialization (all init is done in a single thread) guarantee that all states held by this thead will be pushed back to the shared memory?</div> <div>The use case is quite specific, I init everything in a single thread, want to push all the state to the shared memory. I know post init() use of HSearch will never change the state so I don't "need" locking.</div> <div><br></div><div><font color="#888888"><div> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div> <span style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div> <div>--</div><div>Emmanuel Bernard</div><div><a href="http://in.relation.to/Bloggers/Emmanuel" target="_blank">http://in.relation.to/Bloggers/Emmanuel</a> | <a href="http://blog.emmanuelbernard.com" target="_blank">http://blog.emmanuelbernard.com</a> | <a href="http://twitter.com/emmanuelbernard" target="_blank">http://twitter.com/emmanuelbernard</a></div> <div>Hibernate Search in Action (<a href="http://is.gd/Dl1" target="_blank">http://is.gd/Dl1</a>)</div></div></span></div></span></div></span></div></span></div></span> </div></font><div><div></div><div><br> <div><div>On Jul 4, 2008, at 04:33, Sanne Grinovero wrote:</div> <br><blockquote type="cite">Hi Pavitar Singh,<br><br>I thank you very much about your explanations but actually I opened the issue myself<br>because I have read the same specs and am aware of that.<br> in H.Search (and several other hibernate code) there's this quite common pattern for starting<br> "replaceable" objects (something like user-written plugins, you can provide your own implementation<br>to do some stuff) but this same pattern is also used to start the built-in default strategies.<br> <br>It looks like this:<br> - an empty constructor, to use class.newInstance();<br>- an initialize() to set configuration options<br>- a start() method (eventually) used to start background tasks<br>- some doStuff() and/or getXX() which need to be fast & threadsafe<br> <br>As you can see in Concurrecy in Practice at page 50, this is BAD, as for<br>example in the FSSlaveDirectoryProvider nobody takes care of locking<br>or visibility, and nobody is doing anywhere where I see this pattern used<br> (several times in the project).<br>I'm not saying it is all broken, because usually the threads consuming<br>these unsafely-initialized objects are started after the initialization, so<br>that's ok. In this specific case the state will be used to communicate<br> between threads, so some visibility fix is needed.<br><br>I know I could use it only for final fields, but this is exactly what I want:<br>there are currently 10 instance variables, of these<br>4 have no concurrent use<br> 4 are configuration constants and could use the "final" (they're not safely published)<br>2 would need some lock/volatile anyway, bot only one is used frequently, so IMHO 1 volatile is ok.<br><br>I was thinking in using the same Piggyback technique you mention to<br> safely publish the initialization constants,<br>but I'm afraid the code will become more difficult to maintain and more<br>"unreadable", possibly breaking at the first next patch:<br>IMHO using some unchanging fields "final" is the most clean and<br> readable solution (and best performing), but I need a different<br>constructor for that.<br><br>your opinion is really welcome as I couldn't find other feedback,<br>if you would like to take a look at the sources download the Search trunk<br> and look at:<br>org.hibernate.search.store.FSSlaveDirectoryProvider<br>or the FIXME in <br>org.hibernate.search.reader.SharingBufferReaderProvider<br><br>Sanne<br><br><div class="gmail_quote">2008/7/4 Pavitar Singh <<a href="mailto:pavitar.singh@gmail.com" target="_blank">pavitar.singh@gmail.com</a>>:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Sanne,<br><br>I don't think moving everything in constructor can guarantee safe publication.<br> <br>From the JMM Specification Section 3.5<br><br>"An object is considered to be completely initialized when its constructor finishes. A thread that can only see a reference to an object after that object has been completely initialized is guaranteed to see the correctly initialized values for that object's final fields."<br> <br>That means there are no guarantees for other fields which are non-final.<br><br>But once things are moved in constructor then by using Safe Publication one can enforce visibility guarantees.<br><br>Other thing i was wondering was why do one need to make every field as volatile. As just by using a single volatile variable one can enforce memory barriers. A technique documented in JCIP as Piggyback Synchronizations and used by Doug Lea in implementing ConcurrentHashMap.(You will find get method in ConcurrentHashMap is without any locking and visibility is enforced by use of a single volatile variable.)<br> <br>Also if you can elaborate more on how you are facing the visibility issue then may be i can also spend time on it on figuring performant solution.<br><br>Regards<br>Pavitar Singh<br><a href="http://pitfalls.wordpress.com" target="_blank">http://pitfalls.wordpress.com</a><br> <br><div class="gmail_quote"><div><div></div><div>On Fri, Jul 4, 2008 at 5:13 AM, Sanne Grinovero <<a href="mailto:sanne.grinovero@gmail.com" target="_blank">sanne.grinovero@gmail.com</a>> wrote:<br> </div></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div><div></div><div> Hello all,<br>I'm sorry I've been very busy but as promised I would like to fix HSEARCH-189<br>(and others) very soon;<br><br>I would like to propose a API extension (backwards compatible) that would simplify the patch a lot:<br> keeping it as is it is very tricky to fix the visibility issues in FSSlaveDirectoryProvider<br>and FSMasterDirectoryProvider without introducing a performance penalty.<br><br>I have these options to close the issue:<br>1) add a "volatile" to more than six fields per class (ugly and not good for performance)<br> 2) use some Locks/synch (more readable, still performance hits)<br>3) move the "initialize" arguments to a constructor.<br><br>As Emmanuel knows I would really love the third option, but he's worried about<br> the fact we can't force a constructor in an interface*1, so my proposal is:<br><br>if we find there exists a constructor having the same arguments as the initialize method,<br>we use that, otherwise we use a no-arg constructor and then call the initialize.<br> <br>Reflection is used anyway to instantiate these components,<br>the code in DirectoryProviderFactory doesn't get much more complicated<br>and much more cleanup is made possible in all DPs because of this<br>(as the equals/hashcode comments already ask for).<br> <br>I actually think this same pattern is needed for other components,<br>such as all ReaderProvider, so I hope you'll want to give it a try<br>and let me apply it on other components too if you like the resulting code.<br> <font color="#888888"> <br>Sanne<br> </font><br></div></div>_______________________________________________<br> hibernate-dev mailing list<br> <a href="mailto:hibernate-dev@lists.jboss.org" target="_blank">hibernate-dev@lists.jboss.org</a><br> <a href="https://lists.jboss.org/mailman/listinfo/hibernate-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/hibernate-dev</a><br> <br></blockquote></div><font color="#888888"><br><br clear="all"><br>-- <br> Pavitar Singh<br>Blog: <a href="http://pitfalls.wordpress.com" target="_blank">http://pitfalls.wordpress.com</a> </font></blockquote></div><br> _______________________________________________<br>hibernate-dev mailing list<br> <a href="mailto:hibernate-dev@lists.jboss.org" target="_blank">hibernate-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/hibernate-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/hibernate-dev</a><br> </blockquote></div><br></div></div></div></div></blockquote></div><br></blockquote></div><br></div></div></div></div></blockquote></div><br></blockquote></div><br></body></html>