Is your current Mac OS a 32-bit version? I'm not familiar with them at all anymore. If it is, I'm wondering if it can even address the require memory. I know the Mac addressing scheme is different from the Windows XP version, of which you can only use 3.25gig maximum anyway, no matter the addressing scheme, since the OS uses all of the rest.
All you need to do though is set the bits on the SH4.exe file... do you have a PC available that you could copy the SH4.exe to, even by sneaker-net, and set the bits on the PC with NTCore, copy it back to the Mac and see what happens?...