Making a UI in VR adds a layer of additional concern that pancake UIs don't have to deal with. Someone in Discord asked for feedback of their VR UI. My response turned lengthy, and potentially useful to others.
Make menus accessible. In VR, this has an added step of considering how difficult it is for a user to reach out with their controller to reach the menu. Is it within their play space? Can they alter the location of the menu? Some games use the wrist watch method. Intuitive to open and put away. Intuitive to move around. Another common type is to open with a button press near the user based on their facing angle. This lets the user close the menu and reopen it in a more convenient spot should the menu appear out of the play space.
How are buttons pressed? Does the user need to have their digital hand come into contact with the menu, or can they point from a distance and click? If the latter, a line from the hand should help indicate that this is going to happen. Do you allow both options at the same time? If so, is there a distance that the laser click option disappears? I recall one game that when the digital hand neared a button, open hand would turn into a pointing finger. This is falling out of favor with controllers that can detect what the user's hand is doing, and mimicing that instead.
Are the buttons large enough that they can be uniquely pressed? If it is easy to fat finger (fat fist?) the wrong button, something needs to change.
Don't attach a menu to the user's head facing. It should be either static in the world, or attached to the user's controller. If it is attached to the controller, it should be intuitive to "put the menu away." It's annoying to try and drop something that is stuck to your hand.
Most regular UI design principles apply:
Use understandable labels. Don't use made up words, internal jargon, or memes. Stick with established terminology.
Consistency. Menus throughout the game should appear thematically similar. How a given menu works should be the same throughout the menu tree.
Readability. Menu text should be legible. The menu's colors should have contrast between text, buttons, and the background. Don't use fancy fonts. Don't use fancy effects on those fonts. Careful with over designing for the wow factor of theme. If it isn't easy to use and understand, it's failing it's main job. Remember Me's main menu is an example of going way over the top into the point of being borderline unusable.
The top level menu should limit how many options appear in it to no more than eight. If you need more than that, consider categorizing them into a smaller number.
Bonus: The user should understand where they are within the menu system.
On images.google.com, look up "Video game menu" to see what pancake games have done. It's harder to convey this in VR, but there are many good examples out there. SteamVR overlay. RecRoom's watch. (edited)
