If you watch the history channel show Dogfights and the episode about the Taffy 3 escort group that was attacked by the Japanese navy you will see how a single torpedo launched from (I forget the details) a destroyer escort broke the bow completely off a Japanese Heavy Cruiser. One torp and its done.
Thats not to say one torp can kill a ship. Its possible but it all depends on how you hit it. Sometimes a ship can absorb multiple torps because they hit and explode without doing any real serious damage (hull breaches but thats about it).
Ive saved many torps by just shooting 2 or 3 fish then wait a while to see if the deck is slowly sinking. If so itll go under eventually. I think the ship damage model is quite accurate.
Oh and on a side note, if you use the deck gun and aim for just below the base of the smoke stack you can blow the stack out and have it collapse into the ship. Very funny
And I think it also really reduces the ships speed.
Cheers