Fair enough. I admit it's not ideal, but I think the feature makes more sense when it's hidden completely, and in service of that, I'm inclined to live with the occasional odd situation like the one you've described, particularly since even the other way there'd be a lot of extra clicking.
Still, a good point that I hadn't thought of.
Still, a good point that I hadn't thought of.