As @JoePreiser said, the card itself will probably be the bottleneck. I did some benchmarks on what was supposed to be a fast SD card, if it’s helpful. (Internal slot, 2015 MacBook Pro.)
Another consideration is the iMac SD card slot is in the back, so any speed gains would (for me) have to outweigh fishing around on the back of the iMac to plug in/remove the drive.