Install orders can be checked for validity with TDW's SH5 Validator:
http://www.subsim.com/radioroom/showthread.php?t=186975
If you're using any combination of IRAI, FX Update & TDW's New UIs, they should be installed in that order. The rest of it is just common sense for the most part - remember that whatever is installed later will overwrite any earlier versions - so if you install two different wave mods, for example, you'll only see the effect of the last one, as they both use the same file and the last will overwrite the first.
In terms of 'must have' mods, opinions differ of course. I have a list in my sig containing the mods I'm running at the moment, in the right order, and Sober and Mia both have good lists too.
For quick reference, all of these I consider essential:
Magnum Opus
IRAI
FX_Update 019
New UIs with TDC 6.7.0
Open Horizons II campaign mod
Dynamic Environment pack
I also really favour KZS' Hull Wetness mod, Vorkapitan's Enhanced Caustics and Mia's Real Sink. All of these can be found in the Mods forum with a little searching.
Finally, you'll also need TDW's SH5 Exe patcher regardless of what mods you use (this fixes the broken Hydrophone), and the D3D Antilag file that improves fps. Neither of these last use JSGME, just follow the instructions included:
http://www.subsim.com/radioroom/showthread.php?t=181433
http://www.subsim.com/radioroom/down...o=file&id=2871
