The first minister has appointed Angela Constance as a minister to take charge of the response to the drugs crisis.
She will be a full-time drugs minister, reporting directly to Nicola Sturgeon.
Labour and the Liberal Democrats had been demanding Mr FitzPatrick's resignation and were preparing a vote of no confidence in him.
Figures released this week showed that Scotland again had the worst drug death rate in Europe.
The figures for last year showed that the number of deaths increased to a record high of 1,264 - double the number in 2014 and giving the country a death rate three and a half times higher than England and Wales.
In a statement, Mr FitzPatrick said he had spoken to the first minister and agreed to leave government.
He said the "most heart-breaking and difficult problems" he faced as public health minister were the harms and deaths caused by drug use.
"As the minister responsible for this area I, ultimately, take my responsibility," he added.
"It is clear that my presence as a minister will become a distraction, when we should be focused on achieving the change we need to save lives."
Ms Sturgeon thanked the Dundee City West MSP and added: "While the time has now come to make a change in the public health brief, no-one should doubt Joe's hard work, dedication and sincerity.
"He will continue to champion the interests of his constituents at Holyrood, and I wish him well in the future."
She added that Angela Constance, a former social worker who worked with drug users and their families, would work alongside her as drugs minister to help people whose lives were affected by drugs.
Ms Sturgeon told the Scottish Parliament on Thursday that the drugs death figures were "indefensible" and that her government had to do more to stop people dying.
Who is Joe FitzPatrick?
Joe FitzPatrick was born in Dundee in 1967 and was educated at Whitfield High School. He originally studied forestry at Inverness College and worked for a spell for the Forestry Commission.
His political career started when he was elected to Dundee City Council in 1999.
He won the Scottish Parliament seat for Dundee West in May 2007, and went on to serve on the Holyrood finance committee and as convener of the local government and regeneration committee.
He became minister for parliamentary business in September 2012 and in June 2018 he was appointed minister for public health, sport and wellbeing.
Mr FitzPatrick has faced calls to resign since record drugs death numbers were published earlier this week.
She also apologised to families who had lost loved ones, and admitted that they had been let down.
Ms Sturgeon pledged to work with Mr FitzPatrick to "to make sure we collectively accept this responsibility and take the actions required to fix the problem".
But she appeared to stop short of openly defending her minister from the criticism that has been directed at him over his performance in role.
Was there any other option?
In his resignation statement, Joe Fitzpatrick said he "agreed" to leave the Scottish government. I am not sure there were other options available to him in his talks with the first minister.
Nicola Sturgeon stood by him in the Scottish Parliament on Thursday, when she said she would work with him to lead change.
Today, she decided that change must include replacing the minister who has become the focus for the row over Scotland's rising drug deaths.
Both Labour and the Liberal Democrats were calling for him to be sacked and it's not clear he would have survived a possible vote of no confidence.
The new minister, Angela Constance - who was reshuffled out of government two years ago - has personal experience of working with drug-addicted prisoners as a former criminal justice social worker.
Ms Sturgeon hopes she can help to reduce drug deaths in Scotland, which have doubled during her time as first minister to more than 1,200 last year.
Scottish Conservative leader Douglas Ross said Mr FitzPatrick's resignation "changes nothing" and described the government's drug strategy as "disastrous".
"If this was about the shocking number of drug-related deaths this year, he would have gone on the day the numbers were released," Mr Ross said.
He added: "We urged the first minister to agree to our proposed £20m funding for rehabilitation but got no commitment.
"All the focus must be on the urgent public health crisis of Scotland's drugs deaths epidemic so we can finally start to reverse the tragic number of lives being lost from drugs."
Meanwhile, Monica Lennon, Scottish Labour's health spokeswoman, said it was right that Mr FitzPatrick had resigned and called on ministers to give Scotland's drugs deaths their full attention.
"Urgent funding is needed to boost access to treatment and residential rehab," she said. "The Scottish government must get behind safe consumption facilities, like the voluntary service being run in Glasgow."
She added: "This must be a turning point. The SNP has failed badly and Joe Fitzpatrick doesn't bear that responsibility alone. Our parliament can lead the world on progressive public health policies when we put our minds to it."
Willie Rennie, the leader of the Scottish Liberal Democrats, said Mr FitzPatrick's departure was "inevitable".
He said: "It doesn't ease the pressure on the Scottish government to make up for the terrible failures over the last 13 years on drugs policy.
"Change is needed and fast."