Boris Johnson's plan to save money aims to see staffing return to 2016 levels within three years.
The aim is to return to 2016 staffing levels within three years, Cabinet Secretary Simon Case said in a letter to civil servants seen by the BBC.
Downing Street said it had not ruled out a recruitment freeze or compulsory redundancies as part of planned cuts.
A civil service union said the plan was "ill thought-out" and added strike action was "very much on the table".
Mr Case, in the letter to civil servants on Thursday night, acknowledged the job cuts would be "challenging" and that civil service staffing had grown "substantially" since 2016, partly because of the pandemic.
"We must consider how we can streamline our workforce and equip ourselves with the skills we need to be an even more effective, lean and innovative service that continues to deliver for the people we serve," he wrote.
The cabinet secretary added the prime minister believed "this matters all the more at a time when the government is focused on controlling expenditure and delivering the best possible value for taxpayers in challenging circumstances".
On Friday, the prime minister's spokesman said it was up to government departments to "come forward with their plans" to cut staff numbers - and refused to pre-empt any specific measures.
"It's important for the civil service to be as efficient as possible and make sure there's no duplication," they added, saying changes in technology could allow departments to work differently.
"It's right that the public would expect the government to make sure we are spending taxpayers' money as efficiently as possible."
Rising numbers
There were 384,000 civil servants employed in 2016 - the lowest number since World War Two - but as the UK prepared to leave the EU, numbers steadily rose until they reached 475,000 at the end of last year.
The civil service delivers public services and supports the government to develop and implement policies, but is politically impartial. The work civil servants do touch all aspects of life, from staff paying benefits in the Department for Work and Pensions to people running prisons working for the Ministry of Justice.
On Friday - following press coverage - the permanent secretary at HMRC apologised that staff had learned of potential job cuts "from the media rather than from me or civil service leaders".
He said no decisions had been taken on how the cuts would be made, or how they would impact HMRC, but pledged to keep staff informed and thanked them for their hard work.
Cabinet Office minister Jacob Rees-Mogg said a suggested £3.75bn annual saving was "realistic".
Mr Rees-Mogg - the minister in charge of efficiency - told the BBC the government needed to have "control of budgets" and ensure efficiency during the cost of living crisis.
He said: "It's not a question of doing less of, it's doing things more efficiently.
"We've taken on quite a number of additional staff - indeed 91,000 - to deal with Covid and some of the consequences of Brexit.
"Those two issues are now fading therefore we can get back to the numbers we previously had."
Mr Rees-Mogg said the government did not need "duplication" and it was hoped "natural departures" would be the basis of most of the cuts.
He would not be drawn on what the freed-up cash would be spent on, saying it was up to Chancellor Rishi Sunak to decide.
At an away day in Stoke-on-Trent on Thursday, the prime minister gave ministers a month to produce plans to drastically shrink the civil service, as the government comes under increasing pressure over the rising cost of living.
The Public and Commercial Services union (PCS) responded angrily on Friday, with a threat of strike action.
General secretary Mark Serwotka said "hardworking PCS members who kept the country running throughout the pandemic... will not be the scapegoats for a failing government".
"The government complains about longer delays for passports and driving licences at the same time as sacking the people who are working so hard to clear the backlog," said Mr Serwotka.
"This is not about efficiency. This is about the prime minister trying to create a smokescreen to detract from his utter shambles of a government."
The union will hold its annual conference in 10 days, with national strike action "very much on the table", he added.
Recruitment freeze
Mr Rees-Mogg would not be drawn on which departments could face cuts. He said that secretaries of states and quangos - organisations funded by taxpayers but not controlled directly by government - were responsible for the resourcing of their own departments.
He said his own department, the Cabinet Office, had imposed an employment freeze for the last six months.
While large-scale job cuts are likely to take time, a government source said "instant savings" could be achieved by imposing a recruitment freeze in Whitehall and abolishing any vacancies that had not been signed off by a minister.
Examples include the passport office and 70 staff said to be still working on climate change conference COP26.
A source said the cuts were "not ideological" but about "good housekeeping".
The prime minister suggested to the Daily Mail that the savings could be used for tax cuts, saying: "Every pound the government pre-empts from the taxpayer is money they can spend on their own priorities, on their own lives."
But civil service union the FDA said the expansion of Whitehall since 2016 had been necessary to tackle "two unprecedented events", Brexit and the Covid-19 pandemic.
The number of civil servants grew from a post-war low in 2016 as the UK faced Brexit and the pandemic
FDA general secretary Dave Penman said the government could decide to cut the civil service back to 2016 levels, but he said it would also have to choose "what the reduced civil service will no longer have the capacity to do" which may affect services such as passports, border control or health.
"Without an accompanying strategy, these cuts appear more like a continuation of the government's civil service culture wars - or even worse, ill-thought out, rushed job slashes that won't lead to a more cost-effective government," he said.
Major job reductions through natural wastage - waiting for people to resign or retire rather than enforcing redundancies - would take many years, Mr Penman warned.
The civil service has grown a lot since 2016 - with major changes like Brexit and the response to the pandemic.
No 10 insists its approach is not "ideological" and is instead about "good housekeeping".
Sources in Downing Street argue these are jobs that aren't needed - and that more efficiency and automation will mean no cut in services.
But the specifics of where the numbers will be reduced are not clear. What would cuts to big departments like works and pensions or defence mean in practice?
Although the civil service is often associated with Whitehall and the machinery of government, tens of thousands of people work on things like delivering benefits and keeping the courts running.
Unions are arguing if you cut tens of thousands of staff, there will be an inevitable impact on what the government can do.
Labour accused the government of "pointless rhetoric and lack of action" instead of implementing an emergency budget to offer more support to people struggling with bills.
A government spokesperson said that the prime minister and ministers were "clear that the civil service does an outstanding job delivering for the public and driving progress on the government's priorities".
But they added when people are facing rising costs, they "rightly expect their government to lead by example and run as efficiently as possible".