Rep. Jim Jordan, the top contender in the race to be the next Speaker of the House of Representatives, arrives inside the House Chamber prior to the first vote for a new Speaker of the House at the Capitol in Washington, DC, on Tuesday. (Jonathan Ernst/Reuters)

(CNN) — Conservative Republican Rep. Jim Jordan failed to secure enough GOP support to win the House speakership on the first ballot, falling well short of what the Ohio Republican’s allies had hoped, with 20 Republicans voting against Jordan.

Now Jordan will try to sway the holdouts – a group of GOP moderates and allies of former speaker Kevin McCarthy and Steve Scalise – to change their votes, as the high-stakes chaos over the speaker’s gavel continues two weeks after McCarthy was removed.

The slim margin is what led to McCarthy’s removal at the hands of a band of eight GOP rebels – and a small group of House Republicans unhappy with Jordan could block his ascension, too.

The 20 Republicans who voted against Jordan included House Appropriations Chair Kay Granger, Rep. Mario Diaz-Balart of Florida, Rep. Ken Buck of Colorado and a quartet of New York Republicans in purple districts.

The anti-Jordan contingent cast six votes for McCarthy, seven votes for Scalise and three for former New York GOP Rep. Lee Zeldin, among other alternatives.

Jordan and his allies felt they had made significant headway over the past several days, with the Ohio Republican pitching skeptical lawmakers one on one – and his allies outside Congress attacking the holdouts and threatening political consequences if they stand against a favorite of the Trump-aligned GOP base.

“I feel really good,” Jordan told CNN just before the vote. “Whatever it takes to get a speaker today.”

“We need to get a speaker today and we feel really good about where we’re at,” he added.

It’s been two weeks since McCarthy’s historic ouster. Until the House selects a speaker, the chamber is in a legislative paralysis, unable to consider legislation, such as passing additional military aid to Israel or government funding – with the threat of a shutdown just a month away thanks to McCarthy’s six-week stopgap spending deal that prompted the move against him.

Jordan’s allies believe the number of GOP opponents has shrunk from the 55 who voted Friday against supporting him on the floor to roughly eight-to-10 holdouts.

On Monday, several key holdouts said they would support Jordan, including Rep. Ann Wagner of Missouri, who had previously called Jordan a “nonstarter.”

“I feel like he can bring together everybody, from the moderates to the ultra conservatives, and Republicans across the spectrum,” said Rep. Nicole Malliotakis of New York. “The bottom line is we’ve got to get back to work. We don’t have any time here to waste.”

Even if Jordan doesn’t have the votes on the initial ballot, he could force additional votes, just as McCarthy did in the 15 rounds it took him to be elected speaker in January.

But there are still a group of lawmakers publicly opposing Jordan, including GOP lawmakers still angry that a small group of Republicans forced out McCarthy and then opposed the speaker nomination of House Majority Leader Steve Scalise, who initially defeated Jordan inside the GOP conference, 113 to 99.

“I can’t get past the fact that a small group in our conference violated the rules to get rid of Kevin, and then blocked Steve,” said Rep. Don Bacon of Nebraska. “You don’t have a process where I play by the rules and these other people can’t and then they get what they want. That’s not American. Americans want fair play and rule of law.”

Rep. Mario Diaz-Balart of Florida said he planned to continue to support Scalise on the floor. “You can remember we had an election; the guy who won was the guy who I was with,” he said Monday, while warning that any attempts to pressure him would backfire.

Rep. Ken Buck, a Colorado Republican, said he’s opposed to Jordan, noting Monday evening that he needed to hear Jordan publicly say that Donald Trump lost the 2020 election. Jordan on Tuesday doubled down on objecting to the 2020 election certification.

Several Republicans – including from districts won by President Joe Biden – declined to say Monday evening whether they would vote for Jordan on the floor.

House Democrats have blasted Republicans for putting forward Jordan as the next potential speaker. “I was the last person on the floor January 6, and the idea that this guy is the Republican nominee to be speaker, a guy who aggressively agitated the activities that happened on January 6, I think is disgusting,” said Rep. Jim McGovern, a Massachusetts Democrat.

Jordan’s backers have urged the conference to unify around him – even those who went after McCarthy and opposed Scalise.

Rep. Scott Perry of Pennsylvania argued that those angry shouldn’t take it out on Jordan because he supported both McCarthy and Scalise.

“Feelings are hurt,” Perry said. “But Jim didn’t have anything to do with that. So they need to assign their ire, if you will, to those who they think deserve it – but certainly not Jim Jordan.”