KVM for two Mac Minis - garbled keyboard

I have two Minis - a 2012 and a 2018. The 2012 runs some server-ish functionality as well as a VM or two, and I’m discovering that the keyboard / mouse don’t work properly in the VMs via the screen sharing / remote desktop protocols. So I actually need to switch back and forth.

Both running Catalina.

Right now I have a multi-input HDMI TV that I’m using for a monitor, and a two-port KVM switch.

When I switch from my 2018 to my 2012 (and never the reverse, from what I can tell), the keyboard sometimes gets garbled. I’ll run my hand along the home row of keys, and I’ll get characters like:

åß∂ƒ ßå ƒßå ƒßå ƒß∂ ƒåß∂ ƒ©∂߃©∂ƒß©ß∂∂

Switching the KVM back and forth again usually fixes it. Sometimes it takes a couple of times. It looks like an alternate character set.

I’m wondering if there’s something in my Mac’s software settings I can tweak that might be contributing to the problem, or something else that y’all can think of.

Again, this only seems to be the 2012 that’s having the issue. The 2018 is fine all the time.