I believe the tradition got started in the Sengoku period when the Samurai of Takeda faced gunfire from the
teppos of Tokugawa forces in what is considered the first "modern" battle in Japanese warfare where western style rifle formations were used.
Upon getting shot and hurled from his horse when struck with a ball from one of these Japanese arquebuses, during the last calvary charge of Takeda's soliders, Samurai Fuoku Yorumama's last words exclaimed, "Oh shi...!"
Hence the number four was forever linked with bad fortune for generations to come.