When you went to manual, did you set your Gyro angle to 0 degrees ( or 180 if stern shot)and set the speed knob to 0? What ever the angle and speed was before will stay there until you move it. I have done that myself before, scratching my heading and saying WTF, then I look at the Gyro angle knob or speed knob and see it pointing at 7 degrees and 5 knots.
Easy mistake to make if that's the case.