Boris Johnson must "lead or step aside", senior Tory and former minister Tobias Ellwood has told the BBC.
Mr Ellwood, who chairs the defence select committee, said "we need leadership" following reports of parties being held in Downing Street while Covid restrictions were in place.
It comes as hundreds of angry constituents contact their MPs.
The government has urged people to reserve judgement until senior civil servant Sue Gray's inquiry is finished.
But Andrew Bridgen, Conservative MP for North West Leicestershire, said: "I don't need to see what Sue Gray says to know that for me Boris Johnson has lost the moral authority to lead the country.
"If there's another emergency where he has to call on the public to make sacrifices, he doesn't have that authority. That makes his position in my book, as prime minister, completely untenable."
Mr Bridgen is the fifth Conservative MP to publicly declare they have written to the chairman of the 1922 Committee - which organises Tory leadership contests - to say they have no confidence in the prime minister. Fifty-four Conservative MPs have to write a letter to trigger a vote.
Ahead of Boris Johnson's apology on Wednesday, Mr Ellwood, MP for Bournemouth East and one-time defence minister, said Mr Johnson needed to "show some contrition" and get a grip of the situation - or he would be "out of office".
Another former minister told the BBC: "Johnson is toast... if you were the chief whip looking at him you'd say he's not fit to do any other jobs in government, you wouldn't make him a junior minister, he doesn't work hard enough."
And a senior Tory MP said "there is a lot of scepticism around that there is anyone ready to take the reins. That buys Boris time. But he shouldn't confuse that with another chance."
https://www.bbc.co.uk/news/uk-politics-60005134