England looked on course for Sunday's final until an assault from Jimmy Neesham swung the game in New Zealand's favour in a thrilling finale.
Neesham smashed three sixes, each a body blow to England's hopes, as the Black Caps' pursuit of 167 was given life.
The all-rounder was caught for 27 from 11 balls in the 18th over but, with 20 needed from 12 balls, opener Daryl Mitchell took on the attack.
He hit Chris Woakes for two sixes, finishing 72 not out from 47 balls as New Zealand won by five wickets with an over to spare.
England cantered through the group stage with four comprehensive wins but their quest to become the first team to hold both the men's 50-overs and T20 World Cups is now over.
New Zealand, beaten in incredible fashion when the sides met in the 2019 World Cup final, will face Australia or Pakistan - who meet on Thursday - in Dubai on Sunday.
New Zealand turn the tables on England
The win may not avenge England's victory on that famous day at Lord's in 2019 - it is unlikely anything could - but, as many of the same New Zealand players celebrated Mitchell hitting the winning runs, that defeat may not hurt quite as much now.
As with that game, there were dramatic swings in this one and England looked strong favourites through much of the New Zealand chase.
Woakes looked to have dealt key blows in the opening three overs when he had the dangerous Martin Guptill caught off a leading edge for four and captain Kane Williamson taken at short fine leg for five. New Zealand were 13-2.
Even as Mitchell and Devon Conway combined for a stand of 82, it felt like England were in control, with the required run-rate climbing.
But Neesham, who batted in that 2019 super-over loss, played the crucial, match-winning hand this time.
With 57 needed from 24 balls he swiped Chris Jordan for two leg-side sixes. As 23 runs came from that over, England's bowling began to buckle.
Left-handed Neesham followed that by thrashing Adil Rashid high and long over mid-wicket in the following over and, from nowhere, New Zealand were cantering towards victory.
Neesham was caught at extra cover off the final ball of the 18th to give England a glimmer of hope, but when Woakes faltered Mitchell pounced. Two more leg-side sixes were hit and New Zealand had their dramatic success.
England's turn to feel pain of chase
In the incredible 2019 meeting it was New Zealand's bowlers hit into the stands - on that occasion by Ben Stokes - but here it was England's turn to feel the heat.
Jordan's over, in which he bowled two wides, proved to be the turning point. At the end, he and the other bowlers also put the ball into Neesham and Mitchell's hitting arcs.
For a moment England looked to have stopped Neesham's hitting early when, in smashing the second of those sixes, Jonny Bairstow and Liam Livingstone looked to have combined for a remarkable relay catch.
Replays, though, showed Bairstow landed on the boundary sponge just before off-loading to Livingstone. It was another incident reminiscent of 2019, when Trent Boult caught Stokes late on, but did so while treading on the rope.
Dew may have been a factor, but England undoubtedly missed the death-over skills of Jofra Archer and Tymal Mills, both absent through injury.
While unable to be at their explosive best with the bat, their 166-4 - which included 51 not out from Moeen Ali and 41 from Dawid Malan - looked a good total given the pressure of a knockout match.
They do at least only have another year to wait until the next T20 World Cup, which will be played in Australia next October and November.
Captain Eoin Morgan also suggested after the game that he would stay on to lead England's quest for another white-ball trophy.
'We were right in the game until the 17th over' - reaction
England captain Eoin Morgan: "I can't fault anything we've done tonight. We have fought unbelievably hard throughout this tournament. We've represented ourselves unbelievably well, but tonight we have come up short. I am extremely proud of the guys.
"It's hard to identify key moments. We were right in the game from our innings right through to the 17th over."
Player of the match Daryl Mitchell: "It was a bit of a whirlwind at the end. I can't remember what was going on.
"We tried to hang in there as long as we could and knew that one or two big overs would bring us back into the game. We were lucky to get there in the end."
New Zealand captain Kane Williamson: "They had a pretty competitive total at the halfway stage.
"It was an outstanding knock at the top from Daryl Mitchell but it was also about staying in there and cashing in on small match-ups. That was the difference."