This seems to happen only when quitting while a window is still open (M1, Macos 12.6.1, Orion beta 122). If all windows are closed and Orion is quit, the warning setting remains on. In this case, the warning popup does not show up (as expected), so the issue might be related to the popup itself.