No, you wouldn't be crazy. You'd simply be following medical advice/what's best for your situation. For some chronic pain sufferers opioids are one of the only options (or is something that is used in addition to other methods/things to alleviate pain). For instance, many people with connective tissue disorders use opioids in addition to other physical methods (surgical, splints, etc)[1], and removing something like an opioid can make life untenable, and amounts to torture [2]
1) https://www.ncbi.nlm.nih.gov/pubmed/26497567
2) https://www.statnews.com/2016/07/15/chronic-pain-opioids-pat...