I have had little trouble with armed merchants. Their fire is generally innacurate over 1500 meters and they have a hard time spotting surfaced slow moving u-boats.
Really no reason you shouldn't be able to get within 1500 meters at night without being spotted. Day is another story, they can spot you up to 16km away.
Better luck next time and remember... Virtually every ship out there in stock SH3 is British.
But with GWX it could be any nationality... Check those flags!