I believe the problem here is that you need to programmatically login with the new credentials so that the underlying security context can be updated. When you login (step 2) a security context is established in the web layer. This context contains the authenticated principal/password and is propagated to the EJB container when calling EJBs. It is established once the call is served by a thread and is cleaned up after the thread returns.
In your example, you are invalidating the session and calling the EJB as part of the same method, so this is what happens:
1- context is established using the principal/password associated with the session, lets say userA/passA.
2- session.invalidate() is called, causing the security cache to purge the entry that corresponds to userA.
3- you call the LoginHistoryService EJB. The WEB layer will use the established security context to propagate the identity/password to the EJB container. As there is no cache entry for userA, a new authentication will take place and the DBLoginModule will be called.
3a - if you changed the DB password to the same password (passA), then the authentication will succeed as the exisiting context will propagate the userA/passA pair and this will match the values found in the DB.
3b - if you changed the password to a different password, then the authentication will fail because the existing context will propagate the userA/passA pair and the DB will have a different value for the password.
This is why you see no error in the first case and a login failure in the second.
When you call loginSuccess, you successfully invoke the EJB because the security cache is still active and the security context that was established in the web layer still reflect the correct username and password pair that exists in the DB. So when the invocation reaches the EJB container, the security cache is hit and the user is authenticated. Note that there's no way to escape authentication in the EJB container as all EJB calls are required to be authenticated by the EJB specification. Finally, you invalidate the session and return, so the existing security context is removed and when the client (browser) attempts to call a protected resource he should be redirected to the login page as the session has been terminated.
I believe you can get around the error in the first scenario by calling request.login(username, newPass) after invoking logout (and before invoking the EJB) so that the web container updates the existing security context. This would cause the new username/password pair to be propagated to the EJB container in subsequent calls and you should get no errors from the EJB security layer.