What are the risks of sharing an xpub address?

I've heard commentary from Matthew Kratter that you should never share this. But a lot of crypto tax software needs that xpub address from hardware wallets to make its calculations. What are the issues in sharing that with crypto tax software? It's not private keys or anything. Is it just the chain analysis risk exposure? If that's the case we're all screwed anyway unless we have nonKYC Bitcoin and/or have done CJ. Please help me understand this a little better. Thanks!