We have basic support, to get response from them would take time, by that time I would like to implement some workaround, if that do not work, definately will raise support request with EMC.
As far as I know the Windows service is started using dctmservice.exe and the configuration given in %Documentum%\jboss4.2.0\server\serviceConfig\MethodServer\conf\wrapper.conf. The wrapper.conf refers to DmMethodServer.xml (in the same folder) which specifies using startMthodServer.cmd to start the Method Server.
So I believe "startMthodServer.cmd", right place to set the Java OPTS ?