The truth is actually much simpler.
Goering convinced most of the defendants to follow his line of defence of denying responsibility for everything.
Speer took a different tack, accepting collective responsiblity, while fudging the details of his personal involvement. Not all the evidence against Speer was found and presented at Court. Speer was a sympathetic character and was able to portray himself as a naive young man who had been fooled by Hitler.
On the larger issue of "Allied culpability", I don't see any of it. The Germans started the war and tried to conquer all of Europe, the Germans murdered 6,000,000 jews, 4,000,000 Soviet POWs, etc., etc.
After the war, the Allies set up the tribunals to punish the guilty. There were 13 trials in all, against 200 defendants stretching all the way into 1949. How many Germans should they have locked up to not feel guilty? 10,000? 100,000? 1,000,000?
By 1948, after taking down the worst dictator in modern history, the US was facing the second worst dictator in modern history. Stalin had staged coups in every country in eastern europe and was "purging" their government. Stalin had shipped off every German scientist he could get his hands on to the USSR and was working on an atomic bomb.
What should the US have done? should they have said:
Oh no, we can't deal with any German scientist who worked for the Nazis! Better to remain pure even if it means the Communists will take over all of Europe! 