Perhaps they won't get into your account without your password and phone. However, you've reduced your 2-Factor to 1-Factor since the password is now known. You're still relatively safe - at least safer than not having 2-Factor auth - but changing passwords is cheaper than the risk of relying on 1-Factor auth.
