Where do you have SH4 installed? You can't be in a Program Files folder with it, else you won't get a proper mod activation. Also, are you using an LAA app on the SH4.exe file? I'm not certain how big of a mod you are dealing with, but the 'modern' Windows sure uses a lot more resources than it used to, so an LAA app might help some. CapnScurvy has a write-up on its use in Windows 10 from a fresh install in "
Windows10; Large Address Aware; SH4 Install", which has links in it for two versions of the app. You could also use the
NTCore 4GB Patch version, or Jeff-Groves' (privateer)
Win XP 4 Gig RAM patch for SH4, which you probably have to "Run as administrator".