Richfaces is bad with a cold browser cache as it's delivering its JS and CSS content in dozens of requests instead of bundling the data in a few requests. This is obviously going to be on the table. However, with a warm browser cache it does conditional GET and cache control headers just fine and data is not downloaded again. Simplified: Yes, you can optimize the first visit to a page but any subsequent visit is not executing any requests and using the cached JS and CSS. The only thing that needs to be done is bundling up the resources better. As we are not using Richfaces components on sfwk.org on regular pages, this doesn't affect the website much.
Initial request performance is important too and is an area the RichFaces team is going to investigate for 4.0. How to bundle not only the static resources, but also the dynamic resources into as few requests as possible.
.....
(Yes, Page Speed is great because it analyses some very obscure possible improvements. However, none of them would have any real impact compared to the two issues above.)
I think that page speed is good at visualizing offenders and pointing the way, but it is like a bird dog. It shows you where the meal is, but does not get it on the plate ;-)