Size matters, very much.
In a smaller boat you can get closer on the surface without being detected and avoiding asdic is much easier too. I also think larger boats can be heard more easily, if speed/depth/range are the same. At least that's how it feels to me. All this, combined with the better maneuverability of smaller boats, makes it much easier to escape destroyers in a type VII than a type IX. A type II (I don't play type II's much, so someone else might verify this) is even better at sneaking away.
Edit: Forgot to mention, that early in the war destroyer skill isn't very high, and a type IXB is very survivable. However, this starts to change somewhere around 1941, and by 1943 type IX's do indeed become death traps.