I think one thing not mentioned on this thread is the fact that we are not controlling
the root of the webserver (afaik). The docs are hosted on docs.jboss.org
and I think we
only have write access to the hibernate sub-folder. And even if we had access to the root
and could modify robots.txt and add a sitemap.xml we would need to take all the other
hosted on this server into account.