The confusion arose after Environment Secretary George Eustice said people could go to those countries if they quarantined when they returned.
It comes after news that some holiday companies are refusing refunds to such destinations.
That is because it is now legal to travel to countries on the amber list.
Mr Johnson said: "I think it's very important for people to grasp what an amber list country is: it is not somewhere where you should be going on holiday, let me be very clear about that.
"And if people do go to an amber list country, they absolutely have to for some pressing family or urgent business reason, then please bear in mind that you will have to self-isolate, you'll have to take tests and do your passenger locator form and all the rest of it."
The warning came amid reports that thousands of people had headed for destinations such as France, Greece, Spain and the US, with more than 150 flights reported to have departed on Monday.
After the legal ban on foreign holidays ended on Monday, Environment Secretary George Eustice said that people could go to those countries if they quarantined when they returned.
"We don't want to stop travel altogether and the reason, as [Health Secretary] Matt Hancock set out, that we have the amber list is there will be reasons why people feel they need to travel, either to visit family or indeed to visit friends," he told the BBC Radio 4 Today programme.
"They can travel to those countries but they then have to observe quarantine when they return and have two tests after returning.
"So people can travel to those areas, yes, but they will then have to subject themselves to the quarantine requirements on return."
But the prime minister's spokesman insisted travel to amber list destinations was only allowed for a strictly limited number of reasons.
He said people should not be going to those countries for holidays or leisure.
"The position remains that people should not travel to amber list countries and that is to protect public health," the spokesman said.
"There are some limited reasons why it might be acceptable to travel - for work purposes, protecting essential services or compassionate reasons such as a funeral or care of a family member - but otherwise people should not be travelling to these countries."
He said holidays and leisure travel should be restricted to the short "green list" of destinations such as Portugal.
"We are working with the travel industry and others on this. All of us have a personal responsibility to protect one another as we cautiously reopen international travel," the spokesman said.
"We are moving to a situation where the public can take responsibility for their actions. I think it is important to stress that, by and large, that is what we are seeing.
"Obviously we will keep this under review, we are keeping a careful eye on this and we will take further action if necessary."
A health minister, Lord Bethell, advised people not to take a foreign holiday this year.
Asked about reports that passengers at Heathrow had been forced to stand next to people arriving from red list countries, the minister told the House of Lords that "segregation is unbelievably difficult during travel".
And he said that was why travelling was dangerous.
He told peers: "The ultimate sanction here is that, particularly as we go into the summer, we tell people: travelling is not for this year. Please stay in this country."
Chief executive of Airlines UK Tim Alderslade accused the government of causing confusion.
"People should not travel to red countries, we know that, but to generalise against perfectly legal travel even to green countries is deeply unhelpful," he said.
The government's traffic light system is guidance rather than a legal requirement, and some travel companies have opted to still offer holidays to amber destinations so long as the Foreign Office has not advised against all but essential travel there.
That means some travel companies are refusing refunds for holidays booked last year for summer 2021, or that were moved from 2020 by tour operators - or even to defer the bookings.
The UK's largest tour operator, Tui, is currently offering holidays to amber countries such as Cuba, Barbados and Antigua and to some Greek and Spanish islands.
The company is offering customers free changes to holiday bookings if they don't want to travel, but they will be required to pay any price difference for the new dates, which can be more expensive because demand is high.
Flying to an amber list country for a holiday isn't illegal - but it remains deeply frowned upon.
Until Monday, travelling without good reason was illegal - and you could be fined a great deal of money for doing it.
Now, although you're technically allowed to travel, the government doesn't want you to - unless you go to a green list country.
If you find all this confusing, you're not alone, and reports suggest there are plenty of people for whom "not illegal" is a big green light for a holiday in France, Greece or the US.
With some travel companies refusing to refund would-be holidaymakers who pre-booked trips to countries which are now on the amber list, it's clear that in many cases there's a powerful incentive to disregard official advice.
But passengers should be clear about what they're letting themselves in for: mandatory self-isolation on their return, with possible fines if they don't; and the risk that if anything goes wrong, their insurance might not pay out.