Risks of Connecting Metamask to a site

I keep seeing posts about people losing crypto and the dangers of using Metamask. However, I am assuming that all of these people have either downloaded a scam Metamask Extension, leaked their secret key, their computer has been hacked.

Is there any real risk in connecting Metamask to a site when the only thing it says it is doing is giving permission to view your address?

I'm asking this because I would like to create a site that makes use of connecting to your Metamask wallet instead of the traditional way of creating a login. However, I don't want users to be afraid of connecting to the site. What bad stuff can a site do when all it says is that it can view your addresses?