Maybe the torpedos are set to a wrong depth,check your torpede panel,and better set the depth to 1~1.5m.Sometimes the torpedo which detonated under the ship's keel will damage them instead of destroying them,especially destroyers.
I usally sink destroyer with one torpedo,if you hit the
ammunition depot,it will blow up immediately.
Hope it will help you,good hunting!