Restricting control of HomePod Mini

I’m new to home automation and HomeKit. I just got a HomePod Mini, and have it set up in Home.

I’m the only person set up in Home, and I play my music from Spotify over AirPlay. When my sister came over, we found that she could control the music playing on my HomePod Mini, which surprised me – and which I did not want, because I was having her guess the songs on my playlist!

Basically, I would like to restrict control of the HomePod Mini only to users that I specifically add, and not users in close proximity or on the same network.

I did a quick search and found that “Speakers & TV” in Home’s settings was set to “Anyone on the same network”. I changed it to “Only people sharing this home”, but it seemed to make no difference.

What am I missing?