Don't listen to Jim!
Well normally you should, but he's got a rare affliction; he got so many radio messages from BdU saying 'Be more aggressive!' that he has the uncontrollable compulsion to scream that sentence now and again on the forum
Seriously though, if you're already being depth-charged, attacking a destroyer is the last thing you should do. Go to silent running-mode, dive to 180 to 200 meters (first check the depth under keel of course) and try to zig-zag your way out of there. Keep your speed under 2 knots, otherwise the destroyer will hear you.
If you really want to take out a destroyer, here's what I sometimes do when I attack a convoy, but only if there's only two destroyers around, an only one is close by:
After I attacked on or two merchants in the convoy, I make sure the destroyer knows where I am. I purposely keep my periscope up, so it'll see me, and I turn up the engines a bit. I change course so I'll either face the destroyer or (even better) I'm traveling away from it. Set a torpedo to magnetic, depth one meter under the destroyer's keel and fire. Early in the war, they're not that smart and the destroyer will come straight at you, so if you got your aiming right, you'll usually hit.
Later on in the war, they'll zig zag towards you, so then it's just attacking merchants and getting the hell out of there.
Happy hunting!