As far as I could tell using the debugger, the assocation it was trying to load was not associated to the object that loaded in the query. The assocation belonged to another object entirely than the one that was part of the query. It looked to me like it was trying to load an association that belonged to a previous query that had not been cleaned up.