Here’s how to do it:
firewall-cmd --zone=internal --add-service=ssh --permanent firewall-cmd --zone=internal --add-source=1.2.3.4/32 --permanent firewall-cmd --zone=internal --add-source=5.6.7.8/32 --permanent firewall-cmd --zone=public --remove-service=ssh --permanent
This declares an internal zone with two IPs (add as many or as few as you like) and subsequently removes the SSH service from the public zone altogether. As a result, any other IP gets a message such as “Connection refused” when trying to connect via SSH.
The “–permanent” switch saves the changes. Remove it for testing or if you don’t want this change to be permanent.