But if you "traveled" back in time, your "mistake" would make just as much sense as it did the first/second/third/... time you made it.
Why would we assume that if we "traveled" back in time that we would maintain our "current" state of understanding?
So if I know today's lottery numbers, when I did not know them yesterday, if I traveled back in time to yesterday, I still would not know "tomorrow's" numbers.