For domain, you can not remove until you have unassigned from all server groups. The unassign would result in an undeploy.
For standalone, I think that remove implies that the deployment will try to undeploy first. It might be an open question as to whether or not we require an explicit "disable" before we allow a remove operation.
In any case, it doesn't make sense to remove without undeploy. For instance, what if the deployment needs to load static conent like an html page? If the deployment is removed then the server wouldn't be able to find it.