When I went in hospital for trauma repair surgery one of the docs wouldn't believe my lack of drugs. He told me "you should be taking these drugs" so he put them on my chart. Point is that docs, of whatever stripe, are only human. They only have so much time and have to contend with a variety of factors. One of those factors is that patients sometimes lie. They lie deliberately, or by just not remembering. So, they put people in categories. Most of the time those categories are useful. Sometimes not.
If you are concerned about your drugs, as opposed to Elder Whining, which we all do, see another doc. Tell the doc you want less drugs. Tell the doc what you are willing to put up with. It may be you have a couple less but higher quality years of life as a result. If so that decision is yours and yours alone.
Especially if you haven't seriously faced mortality I recommend the book "Being Mortal". This is written by a doc but uses simple language we can all understand. The key points in the book are that we have to acknowledge mortality(death) and that everyone (family, doc, social services) have to be on the same page when discussing how we will live to the highest quality.