Tap to Wake causing trouble

On a number of occasions I had my iPhone in my pocket listening to music or a podcast walking down the street. The audio would stop and I’d pull out the phone and notice that it said the phone was locked for 1 minute. I had to wait for a minute and put in my passcode. I figured that the raise to wake was getting triggered and somehow invalid passcodes were being entered triggering this lock. It kept happening and so I also disabled tap to wake. This solves the problem, but the iPhone is a lot harder to use without tap to wake. Is there any other way I can have tap to wake enabled and not have these accidental passcode entries locking my phone?