I know this is off topic, but the USS Baya held a 'swimming call' for 20 minutes whilst on patrol. A PBY gave it ASW cover whilst they did so.
I don't think the U-Boats would have had swimming in the Atlantic, as it's a bit cold

. I don't know about the warmer climates though

.