A senior source on the committee said the departures of Rishi Sunak and Sajid Javid would “concentrate minds” among rebels.
The rules could be changed to allow a second confidence vote in Mr Johnson’s leadership, enabling MPs to hold a contest to replace him. Under the current rules, a second confidence vote cannot be held within a year of a Conservative Party leader winning the first.
But Tory MPs disagree over whether it is possible to change the rules before the parliamentary summer recess begins on July 21. One MP on the committee said it was unlikely that any change could take effect before September, giving Mr Johnson a stay of execution.
“It’s a matter of guessing how people will play it,” the MP told The Telegraph. “I suspect that the resignations will probably concentrate minds on what we are going to do.
“It’s extremely unlikely that anything would happen before September. It would be a nightmare during recess.”
But Andrew Bridgen, a rebel Tory MP standing for election to the committee, said: “There will be plenty of time for the new committee to be constituted, meet and if necessary change the rules regarding confidence votes within one year.
“I guess they [Mr Sunak and Mr Javid] finally got where much of the party got weeks if not months ago – that we just can’t carry on like this. What a shambles. It has been a shambles.”
Anthony Browne, a Tory backbencher, released his letter of no confidence in Mr Johnson on Twitter on Tuesday night, declaring that if the Prime Minister did not resign “he must be removed from office”.
MPs have also begun to speculate on whether it would be possible to replace Mr Johnson in a leadership election before the summer because the first rounds of voting take place among MPs when Parliament is sitting.
“Obviously one of the things you always look at is if you are going to have a leadership contest within the wider party, you would have that over the summer break,” said one senior MP.
“But even if Boris resigned tomorrow, we would be very hard-pressed to conduct the parliamentary side of the election this side of the recess. The House rises in 16 days time.”
Prior to the resignations of Mr Sunak and Mr Javid, Mr Johnson was criticised by several senior Conservative backbenchers over his handling of the allegations against Chris Pincher.
Several MPs who will question Mr Johnson at a hearing of the Commons liaison committee have criticised him or called for the Cabinet to move against him. William Wragg, Caroline Nokes, Sir Bob Neill and Tobias Ellwood, who all sit on the committee, publicly condemned the Prime Minister on Tuesday.
After the resignation of two of his top ministers, Mr Johnson gathered 80 MPs in his office in Parliament to discuss his future.
A source in the room said he had apologised for his decision to appoint Mr Pincher to the Whips’ Office in February and said it was now time for him to get on with governing the country and cutting taxes.
He is said to have joked that tax cuts would be easier following the departure of Mr Sunak, who hinted at division between the two men on economic policy in his resignation letter.
“Our people know that if something is too good to be true then it’s not true,” he wrote. “They need to know that whilst there is a path to a better future, it is not an easy one.
“In preparation for our proposed joint speech on the economy next week, it has become clear to me that our approaches are fundamentally too different.”