Maybe the file/image names/conventions of what is being currently downloaded from Apple Photos to your drive is different than what you currently use on your external drive? Meaning, you may be downloading and storing duplicates of what you already have on your drive, instead of overwriting photos you already have.
Like you (it seems), I am obsessive about making sure I have a ‘local’/master copy of all my actual Photo files and that they are backed up (for me, on a Synology NAS). As rock solid as iCloud is, I want my ‘master’ set of Photos on my own discs. I don’t want them ‘stuck’ on Apple or Google servers. The Photos are too valuable to me and my family.
Also, here is a post I shared in this forum from about ~4 years ago where I outlined Photo mgmt processes I use. Not sure it is a match to what you are trying to do but always good to see how others handle Photo management. Since I posted this, I have stopped uploading photos to Google Photos since they too now charge for storage. Happy to continue discussion and chat ‘offline’ if you want.