If this is the situation, I would not be so hard on yourself. It seems like there's a lot of history and emotional trauma that would need to be resolved before any progress would be feasible. If this is indeed the situation, getting your wife involved in the process could make it worse, not better. Imagine all the fights they'd have with so much on the line, when they'd have so many fights before when there wasn't much on the line. You just walked into an impossible situation and did your best. You probably would have done better if you were a sympathetic complete stranger with whom your brother-in-law had zero emotional history (and even if he had zero emotional history with you, he certainly had a lot with your wife and probably subconsciously viewed you as her proxy).
edit: Thinking about it, I think the only thing I'd be able to say in that situation is something like this, just be supporting, not judgmental, not trying to convince him about anything: "Hey man. I'm really sorry this is happening. I know that things are hard right now. If there's anything I can do to help you or the situation, let me know, and I'll do it. How are you yourself? Do you need to talk with anyone about how you're feeling? Let me know if I can help, I'll be right here, OK?"
> The family (not my wife) asked me to talk to my brother-in-law about this because nobody else could get through to him.
I am a stranger on the net but your first priority is your wife and your relationship with her. Be very cautious with demands from the family even if well meaning. Protect yourself and your relationship with your wife.
I second this comment. If the wife has not asked for the intervention, the first obvious question is, why not? That's something that needs to be cleared up with the wife before anything else is done.
When going through a similar event, I was able to find resources at the hospital that helped a lot. In life and death situations they often have patient advocates on staff. I was able to reach out and had a patient advocate, either a nurse practitioner or doctor, that helped sort out the communications. A lot of the stress came from doctors' reluctance to put things in simple terms. The patient advocate was able to basically come in and say, "Look, they can put a pacemaker in, but your loved one only has 12% heart function and is still dying." It was received differently than a family member saying it. After all, their job is to advocate for the best interest of the patient. Ours actually got pretty angry with the doctors for how things had been allowed to progress. Hospitals usually have grief counselors and social workers as well. Just getting the brother extra support might be enough to open him up.
edit: Thinking about it, I think the only thing I'd be able to say in that situation is something like this, just be supporting, not judgmental, not trying to convince him about anything: "Hey man. I'm really sorry this is happening. I know that things are hard right now. If there's anything I can do to help you or the situation, let me know, and I'll do it. How are you yourself? Do you need to talk with anyone about how you're feeling? Let me know if I can help, I'll be right here, OK?"