Edit: After rereading in more detail, I see that you’re actually talking about a bug with the macOS emoji picker. While my suggestions won’t directly help with that, I still think they’re better options for using emoji on a Mac than the built-in one!
I have had the same issue - pretty annoying, but at least for my case I seem to have a possible solution:
As I mostly use Trackpads - either on my Macbook Pro or Magic Trackpad on my iMac - I always use tap-to-click instead of a real press. But this seems to be the problem for me because when I just tap-to-Click on the Emoji I want to enter in the emoji picker nothing gets added and I have to call the emoji picker again (I think this is what you described?); on the other hand when I deliberately press-to-click on the trackpad when selecting an emoji, the emoji gets added correctly 10/10 times.
I haven’t changed any settings for this and it was more by accident I found this solution for me, but on both machines this issue seems gone for me.
Are you using a trackpad or a mouse? I haven’t used a mouse for a long time so can’t say anything about that scenario. Maybe worth a try in case you are using a Trackpad as well?!
This seems to be the issue. I use ‘tap to click’ on a magic trackpad (the one with two AA batteries) but just tried clicking ‘properly’ and it inserted the emoji straight away. (And, as @timstringer mentioned below, keyboard navigation is reliable too…I didn’t know the pop-up was keyboard navigable!)