Seeking financial independence is more about having the freedom to choose whether to keep working or not. Many pursue the FI over the RE and find great joy and meaning in their work. If you'd be in deep trouble if you suddenly lost your job because you're not financially independent, then yes I think you're a slave to your job. People in FIRE communities constantly say to build the life you want to live and then save for it. Not enjoying the journey is a FIRE failure.
