Ok, there is a possibilty that your GFX control panel is over riding the changes but I'm not familiar with nVidia panels. I find it odd that the menu's have changed but the ingame rez has not. Check out the Direct X setting inside your nVidia control Panel.
Also, have you still edited the main.cfg file? Maybe it's causing the weirdness.
For me, the reverse happens. The menu's remain @ 1024x768 but the ingame rez changes. It's certainly confusing me.