Dominic Cummings has revealed he discussed ousting Boris Johnson within days of the Conservatives winning the December 2019 election by a landslide.
The PM's former chief adviser told the BBC it seemed that, by mid-January 2020, Mr Johnson did not "have a plan".
Mr Cummings also alleged the PM's wife Carrie Johnson had tried to influence government appointments.
But, despite quitting Downing Street last autumn following a power struggle, he denied being motivated by revenge.
Asked about Mr Cummings's comments, a government spokesperson said ministers were fully focused on recovery from the pandemic and restoring the economy.
In his first major TV interview, Mr Cummings - who ran the Vote Leave campaign in the 2016 EU referendum campaign before working as the prime minister's adviser - told BBC political editor Laura Kuenssberg that:
* While he thinks Brexit was a good idea and the right thing for the UK, he says anyone who expresses certainty over this "has a screw loose"
* The controversial claim made by Vote Leave during the referendum campaign that the UK was giving the EU £350m a week was a trap set to antagonise the Remain camp
* He has not spoken to the prime minister since he resigned last autumn
The Conservatives won the 2019 general election with a far better than expected 80-seat Commons majority, after a campaign involving Mr Cummings and other former Vote Leave staff in which they promised to "get Brexit done".
Mr Cummings claimed that Carrie Johnson - then Carrie Symonds - had been pleased to have Vote Leave veterans working in Downing Street until then.
But he added: "As soon as the election was won, her view was 'Why should it be Dominic and the Vote Leave team? Why shouldn't it be me that's pulling the strings?'"
Mr Cummings said he and his allies began to fear for their positions by January 2020 and started discussing Mr Johnson's future.
"[People] were already saying, 'By the summer, either we'll all have gone from here or we'll be in the process of trying to get rid of [Mr Johnson] and get someone else in as prime minister'," he said.
Defending the discussions about removing the democratically elected Mr Johnson, Mr Cummings said: "He [the prime minister] doesn't have a plan, he doesn't know how to be prime minister and we only got him in there because we had to solve a certain problem not because he was the right person to be running the country."
He also said: "The situation we found ourselves in is that, within days... the prime minister's girlfriend is trying to get rid of us and appoint complete clowns to certain key jobs."
Bad feelings remained within Downing Street, Mr Cummings said, with his relationship with the prime minister effectively broken by July last year.
Four months later, on 14 November, he quit his government job.
His departure also came at a point when the prime minister was "fed up with the media portrayal of him being a kind of puppet for the Vote Leave team - it was driving him round the bend", Mr Cummings said.
A Downing Street spokesperson declined to comment in detail on Mr Cummings's allegations against Mrs Johnson, but said: "Political appointments are entirely made by the prime minister."
Brexit uncertainty?
Despite running the successful Vote Leave campaign in the referendum, Mr Cummings said that "no-one on Earth" could be certain it had been the right decision to quit the EU.
"I think anyone who says they're sure about questions like that has a screw loose, whether you're on the Remain side or on our side," he said.
"One of the reasons why we won is... we didn't think we were all right and all Remainers were idiots or traitors or anything else."
But the Leave campaign has come in for criticism over its use during the campaign of a controversial claim that the UK was giving £350m a week to the EU.
Questioned over this, Mr Cummings said it had been a trap "to try and drive the Remain campaign and the people running it crazy, so they would start arguing about it".
UK voters decided by 51.9% to 48.1% to leave the EU in the 2016 referendum.
Asked whether the UK had become more divided and politics more brutal in the years since, and whether he had damaged to the country through his campaigning style, Mr Cummings replied: "Obviously I think Brexit was a good thing… I think that the way in which the world has worked out since 2016 vindicates the arguments that Vote Leave made in all sorts of ways.
"I think it's good that Brexit happened."
Motivated by revenge?
In recent months, Mr Cummings has written several blogs highly critical of Mr Johnson and attacked his competence and handling of the the pandemic when appearing before a parliamentary select committee.
But he denied that doing all this, and agreeing to be interviewed by the BBC, was part of a quest for revenge.
Mr Cummings admitted that people thought of him "generally as a nightmare", but said it "doesn't matter if people are upset" by attempts to reform government.
"A lot of people have a pop at me, but you don't see me crying about it," he said.
He also revealed that he had not spoken to the prime minister since quitting, having not answered a call from Mr Johnson shortly after he left Downing Street.
It did not "bother me one way or the other" whether they would speak again, he added.
Mr Cummings was repeatedly challenged throughout his BBC interview to back up his version of events.
He said many of his claims about the workings of government would be corroborated if there was a public inquiry into its handling of the Covid pandemic.