Why wouldn't Proof of Stake drastically reduce block times vs. Proof of Work?

I heard that Proof of Stake will only reduce block time by ~1 second to 12s. Why only 1 second?

Intuitively, it would seem to me that Proof of Stake (PoS) should be able to drastically reduce block times vs. Proof of Work since it replaces the computationally expensive PoW piece and the arms race nature of everyone mining at the same time with random validator assignment. Thus the bottleneck under PoS would only be the network latency it takes to propagate the newly created block to the number of validators required for consensus (51%?) + time it takes for those validators to validate/attest that newly created block and propagate their attestation back to everyone else. I don't know what the block propagation latency on ethereum is to reach 51% of nodes, but I can't imagine that being more than a few seconds.

I understand that reducing block times too low under Proof of Work would be offset by increased computational waste and forking (due to everyone mining concurrently and network latency). But wouldn't this problem be eliminated under Proof of Stake, thus enabling faster block times (and subsequently higher transactions/second)? (EDIT: I elaborated on my reasoning in this comment)

Is there a detailed explanation/analysis somewhere comparing Proof of Stake vs. Proof of Work from a performance standpoint? Why is Proof of Stake only 1 second faster than Proof of Work?

PS: I don't pretend to deeply understand this stuff, so I'm looking forward to my misconceptions being torn apart.