Merkel, who after 12 years in power held a double-digit lead for most of the campaign, scored around 33 per cent of the vote with her conservative Christian Union (CDU/CSU) bloc, according to preliminary results. It was their worst score since 1949.
Its nearest rivals, the Social Democrats and their candidate Martin Schulz, came in a distant second, with a post-war record low of 21 per cent.
But in a bombshell for the German establishment, the anti-Islam, anti-immigration Alternative for Germany (AfD) captured around 13 per cent, catapulting it to become the country’s third biggest political force.
Commentators called the AfD’s strong performance a "watershed moment" in the history of the German republic. The top-selling Bild daily spoke of a "political earthquake".
AfD supporters gathered at a Berlin club, cheering as public television reported the outcome, many joining in a chorus of the German national anthem.
Hundreds of protesters rallied outside, shouting "Nazis out!" while smaller AfD demonstrations were held in other cities across the country.
The four-year-old nationalist party with links to the far-right French National Front and Britain’s UKIP has been shunned by Germany’s mainstream but was able to build on particularly strong support in ex-communist eastern Germany.
It is now headed for the opposition benches of the Bundestag lower house, dramatically boosting its visibility and state financing.
Alarmed by the prospect of what Foreign Minister Sigmar Gabriel branded "real Nazis" entering parliament, the candidates had used their final days of campaigning to implore voters to reject the populists.
Turnout was markedly higher than four years ago, up to around 76 per cent from 71.5 per cent.
French President Emmanuel Macron was among the first to congratulate Merkel, promising that the two key European partners would keep up their "essential cooperation".
Big new challenge
Merkel admitted that she had fallen far short of the 40 per cent goal her party set.
"There’s a big new challenge for us, and that is the entry of the AfD in the Bundestag," said Merkel, adding: "We want to win back AfD voters."
Germans elected a splintered parliament, reflecting a nation torn between a relatively high degree of satisfaction with Merkel and a desire for change after more than a decade of her leadership.
Another three parties cleared the 5 per cent hurdle to be represented in parliament: the liberal Free Democrats at around 10 per cent and the anti-capitalist Left and ecologist Greens, both at about 9 per cent.
As Merkel failed to secure a ruling majority on her own and with the dejected SPD ruling out another right-left "grand coalition" with her, the process of forming a viable government was shaping up to be a thorny, months-long process.
Merkel, 63, often called the most powerful woman on the global stage, ran on her record as a steady pair of hands in a turbulent world, warning voters not to indulge in "experiments".
Pundits said Merkel’s reassuring message of stability and prosperity resonated in greying Germany, where more than half of the 61 million voters are aged 52 or older.
Her popularity had largely recovered from the influx since 2015 of more than one million mostly Muslim migrants and refugees, half of them from war-torn Syria, Iraq and Afghanistan.
But the AfD was able to capitalise on anger over the asylum issue during what was criticised as a largely lacklustre campaign bereft of real clashes among the main contenders.
The party has made breaking taboos its trademark.
Top AfD candidate Alexander Gauland has called for Germans to shed their guilt over two world wars and the Holocaust and to take pride in their veterans.
He has also suggested that Germany’s integration commissioner Aydan Ozoguz, who has Turkish roots, should be "disposed of in Anatolia".
Merkel bound for ’Jamaica’?
The SPD said its catastrophic result would lead it to seek a stint in opposition to rekindle its fighting spirit.
"This is a difficult and bitter day for German social democracy," a grim-faced Schulz, a former European Parliament chief, told reporters, adding that he hoped to remain party leader.
This would leave Merkel in need of new coalition partners.
If the SPD sticks to its refusal to play ball, mathematically the most likely scenario would be a link-up with the pro-business Free Democrats, who staged a comeback after crashing out of parliament four years ago, and the left-leaning Greens.
That so-called "Jamaica" coalition, based on the party colours and the Caribbean nation’s flag, would be a risky proposition, given the differences between the parties on issues ranging from climate policy to migration issues.
Schulz, 61, struggled to gain traction with his calls for a more socially just Germany at a time when the economy is humming and employment is at a record low.
Meanwhile Merkel faced accusations from within her conservative camp that she had left its "right flank exposed" to the AfD’s challenge with her centrist stance on issues such as border policy.
"This is competition for the Union and the conservative spectrum in general," said political scientist Lothar Probst of the University of Bremen of the AfD.
"A very difficult period is beginning for the chancellor.", Source from Vietnamnews.