Given the current price it probably wouldn't hurt to just try all three. Of course if you end up liking one over the others then you may think you wasted the extra money, no matter how little.
SH3 is the most complete, in that you can run any u-boat in any period of the war. If you want to run a Type II in 1939 or a Type IX in 1944, or even a Type XXI, then that's the way to go.
SH4 is much better looking and has some improvements, and you can play with u-boats or with US fleet boats, so it does have that advantage. Its biggest disadvantage is that it sort of got ignored by the modders. Still, it has its fun points.
SH5's is even better looking and contains some new features, such as interacting directly with the crew and walking around the whole boat. It only has one type of u-boat, the Type VII, but it has all four variants of that type and can be forgiven because the devs were faced with the option of doing all the boats or doing one really well, and they wanted the whole atmospheric bit of feeling like you're there. Improvements are being made daily and it's looking more and more like someday all those other features will be available.
So it comes down to what kind of game you want to play. I hope I've helped and not made it worse.
