The factories I created are just for unit testing, mocking the behaviour of the real one.
Anyway, I see now that DefaultClassPoolFactory is always used. It seems that the default classpool has reference to the classloader loading up /server/default/conf/bindingservice.beans which is part of the default domain which explains why the default domain is able to load up the classes.
I will try using RepositoryClassPoolFactory instead and modify AS startup as required.