I find map contacts off to be more of a challenge but very tedious in addition to that fact that your crew does not distinguish between radar contacts when they detact them so you wont know if you have an SD contact or a SJ contact.There is a mod that removes all contacts on map except for aircraft.
I find a great compromise for realism is leaving cams off but contacts on.Stock contacts are bit much because it lets you know too much info just looking at them TMO's are all black suares dots and lines and only display a name after you ID them.
The sonar lines counter balance the terrible ability of even the "best" sonar operator in the game to keep track when you are submerged.
Having said that, once in a while I like to patrol with cams and contacts off for fun.Yesterday loaded up the mod the only displays planes on the map and went to sea in Jan 1944 out of Fremantle with Gato Class sub(USS Gato actually) running TMO 2.1 with RSRD. I was patroling the Celebes Sea chokepoint below Davao, Phillippines when radar detected a convoy of six ships with 2 escorts.This lead to a 2 1/2 day (game time) battle(2 days in RL, about 8 hours of game time total) battle with multiple attacks submerged and on surface and several tough depth chargings.The convoy was six tankers(3 large, two medium one small) and two very tough Type C escorts.I had two stern Mark 14's left and about a week later in same area encountered 3 large nippon maru tankers(10,000 tons each) and using my last two fish attacked.The tanker was loaded with fuel obviously because it burst into a fireball and sank in about 15 minutes.
The escort's counter attack was rough, pushing me to 400 feet at one point, draining my battery down to 30 percent (due to constant stop and go runs to avoid huge patterns of charges) the final evasion took about 3 hours real time.Doing so without the sonar lines you get with contacts on is tough but rewarding and exciting.
I made it back to fremantle with little of 54,000 tons down in one patrol.Seven tankers plus a lone merchant I sent to the bottom earlier in the patrol.
Anyway, once you are sure you understand about tracking, attacking, evading without contacts, I suggest give it a try if you have time, its a blast but definitely tough.