It could amount to a stinging defeat for Mrs Merkel's Christian Democrats, who are projected to gain 19% or 20% of the vote, putting them in third place.
Although the centre-left Social Democrats are expected to retain first place, they are projected to win 30.5% of the votes - a fall of five percentage points compared to 2011.
One of Mrs Merkel's top deputies in Berlin, Michael Grosse-Groehmer, told ZDF: "This isn't pretty for us. Those who voted for the AfD were sending a message of protest."
The deputy leader of the AfD, Beatrix von Storch, told Sky News that the exit poll result would be a "huge success" for her party, which was only founded three years ago.
"This is what we call probably the beginning of the end of Angela Merkel's chancellorship," Ms von Storch added.
Mecklenburg-Vorpommern, a small coastal state in northeastern Germany, only has 1.3 million eligible voters.
The German leader's open-door refugee policy was a key issue in Sunday's state election.
Mrs von Storch told Sky News that Mrs Merkel's approach to refugees is "going to destroy Germany and Europe".
She added: "We have to protect our borders. People do not support this policy anymore."
Back in March, voters in three other states rejected the Christian Democrats and voted in increasing numbers for the AfD - a trend which could prove problematic for the Chancellor, who may run for a fourth term.
In a last-minute campaign appearance on Saturday before she flew to China for the G20 summit, Mrs Merkel had urged voters to shun the politics of "angst" offered by the AfD.
Despite the strong result, there is no prospect of the AfD going into government as the Social Democrats and the Christian Democrats have said they would not form a coalition with the party.
SKY News.
