Is Bitcoin really trustless if the average person still has to trust hardware wallets?

Unless you really understand what’s fundamentally happening at the hardware and software level on these devices which I doubt very many people do or are interested in, are we not just moving trust from institutions to developers?

As I’ve started accumulating more BTC the more nervous I get checking on my hardware wallets and seeing if it’s still there, what’s really stopping an update to the ledger/trezor app that finds an exploit on the hardware to expose the private keys? I know it hasn’t been done yet or there is no current knowledge in how it could be done but what makes everyone so sure that it will always be that way?