Hi,
did you download and read
http://www.vanjast.com/NavMod/RealNav_ReadMe2.zip
this ?
It explains every basic knowledge you need to use the mod.
Sextant:
It is 'turned on' by dragging it from the right side of the screen to the middle. If you installed the mod properly, there should be a little something at the right side of the screen (no matter on which station you currently are), it's also on some screenshot in the above link.
If it isn't there it might be a conflict of some mods. The sextant consists of some new images and is also edited into the ...\data\Menu\menu_1024_768.ini
If you installed the RealNavMod and later on another mod which wants to edit this file, the second one would have overridden the menu_1024_768.ini and the sextant won't be useable in the game (just in case you installed more mods than RealNavMod).
Protractor:
Relatively straight forward to use. If activated and drawing a line you can simply read the protractor (over the line you are currently drawing) and get the course from that.
Just be sure that you use a protractor for that which has the 180° mark on top and the 360° mark at the bottom.
Further links which should help:
http://www.subsim.com/radioroom/showthread.php?t=113975
http://www.subsim.com/radioroom/showthread.php?t=114812
http://www.subsim.com/radioroom/showthread.php?t=116837 (where we finally figured how to get correct longitude values :> )
But you should read the to guides in my first link at first. They explained everything really well, with screenshots.
Good hunting, i.