You are fine starting the game directly from the executable file. Do you have the LAA (Large Address Aware enabled on the SH4.exe file? That definitely helps, especially on larger mods. The best thread for details on LAA is CapnScurvy's
Windows10; Large Address Aware; SH4 Install. He comes at it from a Windows 10 fresh install of SH4 and its use of LAA, but it applies for the most part, all the way from Vista 64-bit through Win10 64-bit.
Be sure and try Bleiente on his facebook page linked above by Eichhörnchen.