If there are "only" 2^160 different addresses, is there any point in using a seed with more than 160 bits of entropy?

Title says it.

From a practical point of view, 160 bits, or even 128 bits, is obvously enough right now.

But taking obscure attacks like quantum computers etc. into account, is there any benefit from a seed (=mnemonic+passphrase) combination with more than 160 bits of entropy? After all, if an attacker were able to brute force 160 bits, he could simply look for any seed leading to your address, no matter how many bits of entropy your seed generation method has.