Managing Bitcoin Core's records of public keys and unencrypted balances

In the documentation for Sparrow Wallet, I read:

Although you have eliminated some privacy concerns by running your own node (or connecting to someone’s that you trust), others remain. If your wallet software is connecting directly to Bitcoin Core, you are using Bitcoin Core’s wallet internally. This is true not only for Sparrow in this configuration, but always true for Specter, FullyNoded, and of course the Bitcoin Qt wallet itself. Unfortunately, Bitcoin Core stores your public keys and balance unencrypted on the computer it is running on. If this computer is regularly connected to the internet, it is at risk to hackers - which will make you a target once your balance is discovered.

Does this mean that even if I switch to using different passphrase accounts, for example, it will maintain a history of all public keys it comes across over time?

The recommended solution appears to be to run an Electrum Server, but I'm wondering if there isn't another simpler solution for Windows users. Is there no way to control what is stored by Bitcoin Core? Is there perhaps a proper hygiene practice to clear the public key and unencrypted balance records similar to how one clears their cache and cookies after a browser session?