Question about Segwit backwards compatability

My current understanding of how Segwit works is that sw transactions omit the traditional signature field and give an automatic TRUE evaluation with no signature check in the eyes of old software, yet use the additional witness signature field in the new software.

In this case, wouldn't a segwit transaction with an invalid signature still be accepted by old software causing a fork? How do both softwares stay in agreement on the validity of the witness signature when only one is aware of its presence.