For me on Windows 10 using a NVIDIA GeForce RTX 2080 SUPER, Sea of Thieves resolution settings are ignored. When I set to full screen, it works as expected. When I set to windowed mode, it uses the screen's max height, and displays as a borderless window locked size mode. Changing the resolution doesn't change anything. After messing with it for a while, I got the resolution to listen to me.
First, in Steam, Right Click on the game in Library -> Properties... -> Set Luanch Options -> Enter: -WINDOWED
Within the game, go to windowed mode in the game's Video Settings