I can't reproduce this (either using back button or the keyboard shortcut) in 0.99.109-beta.
Recipe for me was to go to https://github.com/utmapp/UTM and use the link under the "install" header ("UTM is also available for macOS: ....")
Clicking back/forward from there seems to update the favicon OK for me at least... Intriguing