As you don't know what you run I suppose you start JBoss like this:
'bin/run.sh'
then you are using the 'default' profile, if you use the -c option you have to use that directory.
So assume you are not using -c, change server/default/conf/jboss-log4j.xml.
Also what kind of debug messages you see? Your application or JBoss?