Can one secret phrase (eventually) access any wallet?

This may sound dumb but hear me out and then tell me why I'm dumb.

My understanding is private keys are derived from secret phrases and private to public keys are 1:1.

Looking into BIP39 specs, and using the iancoleman website as a tool, addresses are derived by incrementing the path. I understand there is no limit to this ... so eventually a private key may be generated that clashes with the private key of another secret phrase but different index?

I think the limitation is incrementing the index and calculating a new key is just as random as getting the 0 index address of a secret phrase so while plausible it's just brute force and not practical?