Cabinet minister Jacob Rees-Mogg has been criticised for leaving a note for civil servants, saying "sorry you were out when I visited".
The note, printed on government paper with Mr Rees-Mogg's title, was left at empty desks and read "I look forward to seeing you in the office very soon."
Mr Rees-Mogg has said all civil servants must stop working from home.
FDA union boss Dave Penman said the note was crass and insulting, and undermined civil service leadership.
A source close to Mr Rees-Mogg confirmed the note, which has been widely shared on social media, was genuine.
The note, which was signed off by the Cabinet Office minister "with every good wish", was left in a "completely empty" office, the source said.
The source added that it was "very important for the taxpayer and British public that government is working properly and the estate is well used".
FDA General Secretary Dave Penman said civil servants were planning to leave because of a renewed effort to have them working from Whitehall.
"Day after day we have good people telling us they will go because they know this is not about effective working practices," he said.
"They know it's a culture war that some ministers think plays well with their base.
"These notes from JRM are not only condescending, crass and insulting, they completely undermine the leadership of the service."
Liberal Democrat MP Helen Morgan also condemned Mr Rees-Mogg's actions in a tweet, saying: "What a pointless nasty gesture."
Former Conservative MP David Gauke said the Cabinet Secretary "really needs to have a word" with Mr Rees-Mogg over the note.
"Not convinced this will get the best out of civil servants," Mr Gauke said in a tweet.
The BBC has asked the Cabinet Secretary for a comment.
Civil servants have been told to return to the office
Earlier this week, civil servants were told they must stop working from home and return to the office to ensure government buildings are at full capacity.
Mr Rees-Mogg wrote to cabinet colleagues urging them to send a "clear message" to the civil service about returning.
He also sent a league table of how many officials were at their desks on a daily basis this month, which shows in some departments daily average of staff in the workplace was lower than a third.
Average daily attendance in the week of 4 April was 44%, figures show.
The government's recommendation for people to work from home ended in January, after being reintroduced in the face of the emergence of the Omicron variant last year.