When on the start page, allow the key "tab" to toggle focuses between the bookmarks sidebar, favorites and frequently visited.
So for example, the bookmarks sidebar will be focused by default if it is shown. So I do not have to "click" the empty part of the sidebar. And then when I press "tab", it focuses on the "Frequently visited". And again, and it focuses on the "Favorites".
And when I press the arrow keys, it allows me to navigate the favorites, and also enter folders or enter the website when I press "enter".
Here is a simulation. Do not mind the first part if it is inconsistent, the first few seconds of random keystrokes was me just testing.