There’s a purpose-built solution for this.
I no longer desire to listen until sleep so I don’t use mine any more, but I can attest that it is surprisingly comfortable and sounds pretty great if you can wiggle the ear pieces into the right spot. Once they’re in place, they stay there while worn, but taking it off and on and the charging process (mine require opening the sleeve to charge) do tend to move them around. Still, it’s not a huge task to move them into place each time once you get the hang of it.