[keycloak-user] How to disable user roles updates with subsequent idp logins?

Hello, I have a following scenario: user logs in for the first time from AD FS. There is a mapper in place that assigns him a role. He is then assigned some more roles manually. When he logs in second time, all the roles added by hand are being removed. I've tried looking for something to disable this on keycloak side, but i don't see anything relevant in documentation. Unfortunately, i don't have access to that particular AD FS. Is there a way to stop this overriding on Keycloak side, or is assigning all roles by mappers the only way? Best regards, Dmitry