Looks like that some environment is missing.
What version do you use? I check with 5.1.0 (shutdown.sh -S) and it works.
Maybe there is a problem in the shellscript to locate the correct directory.
What if you set JBOSS_HOME to the root directory of your installation or start from root directory with bin/shutdown.sh