What really happens when you restore your wallet using a seed phrase, and the wallet that gets restored shows a different address that is not really your wallet?

So this has happened to me many times before. And I have also read about it happening to other people, so I am guessing it is not something strange. Here's what I am talking about:

I have the seed phrase to I wallet that I have used for a long time, and I wanted to restore this seed phrase in another wallet. So I download a new wallet, and restore using seed phrase. I believe that my old wallet should get restored in this new wallet. That is, my "new" wallet should be able to show my old balances. Unfortunately, the balances showing in my restored wallet are not the balances in my "old" wallet. In fact, the wallet address showing in my new wallet is not even the wallet I recognize.

I believe that each seed phase points to a unique wallet address, right? So why do I get a different address using the same seed phrase? This in not the first time something like this has happened, so what is the technical explanation for such a situation? And... does it mean someone could be trying to restore their wallet address somewhere, and end up restoring a wallet address that belongs to another user? (Yes, I know chances almost impossible, but it could happen, right? at least in theory)