Hey @itsN1che thanks for reporting this issue. We are already aware of this and already have a fix internally that will be out soon (next monday for RC - and sometime after that for the next beta, most likely next week).
There's no setting that caused it to show every time - it's a bug. It started happening after macOS 26.1 update (as far as I'm aware this does not happen in macOS 26.0.1). macOS 26.1 changed some inbuilt behaviors of the sidebar panels and our code didn't handle this change - thus causing this bug.
I'll merge this issue with the one that's already there.