The gist is that Google wanted to grow Android into something other than a niche player, and made a deal with the devils: hardware vendors. In effect, hardware vendors said they wanted to take a dump on their customers just like they always have, with bloatware baked in and planned obsolescence and Google said OK. And they don't control the hardware.
On the manufacturing side, a huge team is assembled with piles of money and resources to design and build a complete product out of hardware, firmware (blobs), and software, test it, document it, and then hand it off to marketing. That entire team is disbanded into a pool from which a whole new team will be created for a whole new product. Maybe less than a dozen people remain on a product team following release, and that's just to cover major bugs found once it's in the field. After about 6 months, it might be a team of two people. If it's a small market product, it might be a team of handful that cover multiple similar products.
Anyway, after 18-24 months there's essentially no one around from the manufacturer to support the product. It's not at all the same business model as say, Apple, where it's a giant team supporting a dozen handsets post-release, with coordinated roll outs for bug fixes.
If you want a newer OS on the Android phone, it's a do it yourself proposition. Lineage OS. Hopefully your phone has a relatively easy to unlock bootloader. And while Lineage OS will have a much newer kernel than the stock OS, the code to support, e.g. the NFC, may not be upstream because it's got some patent encumbered code, or maybe something new in the kernel breaks some driver and no one is around who knows enough about it to get it fixed. The exact sequence to put components into low power state, if not correct can cause battery life to suck. There are all sorts of ways this can not work out well, but at least it's an option.
What Google has done since Android 5 is emphasize library stability and security through Google Play updates. New features and bug fixes happen via Google Play itself being updated; it appears to be just an app update but it's actually what the bulk of 3rd party apps use. The low level stuff including the kernel isn't user space, and while your handset gets stuck for life, in effect, with an old kernel, Google are pushing out a chunk of bug and security fixes this way.
On Google devices starting with the Nexus 5X/6P, there's a separate /vendor partition for blobs, and Google publishes monthly updates for that. I'm not sure if other vendors do this…
On the manufacturing side, a huge team is assembled with piles of money and resources to design and build a complete product out of hardware, firmware (blobs), and software, test it, document it, and then hand it off to marketing. That entire team is disbanded into a pool from which a whole new team will be created for a whole new product. Maybe less than a dozen people remain on a product team following release, and that's just to cover major bugs found once it's in the field. After about 6 months, it might be a team of two people. If it's a small market product, it might be a team of handful that cover multiple similar products.
Anyway, after 18-24 months there's essentially no one around from the manufacturer to support the product. It's not at all the same business model as say, Apple, where it's a giant team supporting a dozen handsets post-release, with coordinated roll outs for bug fixes.
If you want a newer OS on the Android phone, it's a do it yourself proposition. Lineage OS. Hopefully your phone has a relatively easy to unlock bootloader. And while Lineage OS will have a much newer kernel than the stock OS, the code to support, e.g. the NFC, may not be upstream because it's got some patent encumbered code, or maybe something new in the kernel breaks some driver and no one is around who knows enough about it to get it fixed. The exact sequence to put components into low power state, if not correct can cause battery life to suck. There are all sorts of ways this can not work out well, but at least it's an option.
What Google has done since Android 5 is emphasize library stability and security through Google Play updates. New features and bug fixes happen via Google Play itself being updated; it appears to be just an app update but it's actually what the bulk of 3rd party apps use. The low level stuff including the kernel isn't user space, and while your handset gets stuck for life, in effect, with an old kernel, Google are pushing out a chunk of bug and security fixes this way.