I’ve never done this nor do I endorse any of these links, but an internet search on “rebuild macos contacts” comes up with some items, some of which might work for you.
Best to a backup of what’s on iCloud and on local first, of course.
This has happened to me several years ago. I never had it happen before and it never happened afterwards.
If I recall correctly I was able to fix it with number 9 mentioned in this article:
Using these steps, you can prevent the Contacts app from accessing the current address book folder and the files in it. Once the Contacts app cannot access this folder, it will automatically create a new one and properly sync all iCloud contacts to your Mac.
Make sure to backup your contacts first. There are several options doing so. One of the options is to export all contacts to a vcf using iCloud.
Is there any chance some of your contacts are in a separate list or “On my Mac”? If so, selecting All Contacts on your Mac would give you the total count.