Germany now faces weeks, if not months, of paralysis with a lame-duck government that is unlikely to take bold policy action at home or on the European stage.
With no other viable coalition in sight, Germany may be forced to hold new elections that risk being as inconclusive as September's poll.
Merkel, whose liberal refugee policy has proved deeply divisive, had been forced to seek an alliance with an unlikely group of parties after the ballot left her without a majority.
But following more than a month of gruelling negotiations, the leader of the pro-business FDP, Christian Lindner, walked out of talks overnight, saying there was no "basis of trust" to forge a government with Merkel's conservative alliance CDU-CSU and the ecologist Greens.
"It is better not to govern than to govern badly," he said, adding that the parties did not share "a common vision on modernizing" Germany.
Voicing regret at the FDP's decision, Merkel vowed to steer Germany through the crisis.
"As chancellor... I will do everything to ensure that this country comes out well through this difficult time," she said.
News magazine Der Spiegel called the breakdown in negotiations a "catastrophe" for Merkel and said Germany, long seen as an island of stability, was having its "Brexit moment, its Trump moment".
French President Emmanuel Macron, who has sought Merkel's backing for an ambitious EU reform plan, expressed concern about Germany's political deadlock. He said he hoped Berlin would remain a "stable and strong" partner to allow the two partners to "move forward together," his office said in a statement.
The euro and Germany's blue-chip DAX index fell slightly on the news of the stalemate, but later back edged up, erasing early losses.
READ ALSO: Germany's political uncertainty rattles markets
'Populist agitation'
The acrimonious negotiations stumbled on a series of issues including immigration.
Merkel let in more than one million asylum seekers since 2015, sparking a backlash that allowed the far-right AfD party to win its first seats in parliament.
Its parliamentary co-leader Alexander Gauland welcomed the collapse of the talks, saying that Merkel had "failed" and that his party "looks forward to potential new elections" in which it stands to make strong gains.
The negotiating parties also differed on environmental issues, with the Greens wanting to phase out dirty coal and combustion-engine cars, while the conservatives and FDP emphasized the need to protect industry and jobs.
The Greens angrily deplored the failure to form a government, saying they had believed a deal could be done despite the differences and accusing the FDP of negotiating in bad faith.
Lindner, who had taken a harder line on refugees as the talks progressed, "opted for his kind of populist agitation instead of political responsibility", Greens Europe MP Reinhard Buetikofer tweeted.
Chancellor in danger
Merkel could now try to convince the Social Democratic Party (SPD), which has been the junior coalition partner in her government since 2013, to return to the fold.
But after suffering a humiliating loss at the ballot box, the party's top brass has repeatedly said the SPD's place was now in the opposition.
Merkel, who has been in power for 12 years, could also lead a minority government although she had signalled that she was not in favour of such instability.
Germany could therefore be forced to hold new elections, which would have to be called by President Frank-Walter Steinmeier.
But that is not without peril for Merkel, who would face questions from within her party on whether she is still the best candidate to carry its banner into a new campaign.
As talks dragged on without a breakthrough, Steinmeier had on Sunday issued a warning to parties not to recklessly force new elections.
"All sides are aware of their responsibilities. And this responsibility means not returning their mandate to voters," he told Welt am Sonntag.
Merkel was meeting with Steinmeier Monday to discuss the crisis but planned talks with Dutch Prime Minister Mark Rutte were hastily cancelled.
Voters in central Berlin expressed shock about the sudden injection of high drama into the often staid world of German politics.
But law student Alexander Streb, 20, insisted that the upheaval did not spell "the end of the Merkel era".
"She stands for stability, that's also why young people like her," he told AFP.
Comments