If the direction to the destroyer is not moving you can be reasonable assured it is far away (or still homing in to you
). If it drifts noticeably in degrees while you are listening, then it is probably quite close. And if you don't pop your periscope too high (just crossing the waterlevel, still in the waves) then you should be able to see it's smoke and know you should be on your guard.
For crash dives you also better have your negative tank filled beforehand. I never said you should stay at full throttle. Once you are under you are clear and can do whatever you think is needed. Just be sure to leave halve of a tick in the negative tank, as that puts you better neutrally buoyant. (That is, until next week as they are bugfixing buoyancy stuff they said. So next weekend all could be new again.)