BTW I don't really think this should have been a WEBSITE issue. That comment was to highlight that what we discussed on the meeting doesn't seem all right (according to the decision, this should have been a WEBSITE issue).

To do add a pull-requests checker build: just clone one of the projects already doing it, for example job http://ci.hibernate.org/view/Pull%20Requests/job/hibernate-search-PR/, and make the small adaptations you need (mainly the repositories being monitored). If you could name it consistently to other projects it will be shelved in the right tab (as we use regex for that).