Mine is also a 650 watt, and I have problems. For these cards I believe you need a power supply with minimum 3 separate 12volt rails.
Most have only 1 rail, if your PS is "SLI Capable" then it should have at least 2 (as mine does!), but the GTX260 has 2x6pin power connectors (or 1x8pin), which means it draws a sh@tload of power and needs still an extra 12volt rail
.
That's why your older card works, because it draws so much less power.
If you have a really expensive high quality 650watt, then it should be ok, but power supplies are in no way equal and the manufacturers use terms and names to deceive people into buying sub-standard stuff. "650 watt" can mean anything, depending on how they decide to calculate that 650 watts.
I'm going to have to spend over to $200us to get a power supply capable of running my setup!
I still think it's worth investigating.
As a test you could try removing all "unnecessary" equipment from your PC, ie minimum RAM, 1 hdd, underclock your CPU if possible, disconnect every accessory that draws power. See if it runs any better.