Mr Rutte's centre-right VVD party is thought to have taken 32 of the 150 parliamentary seats, 12 more than Mr Wilders' anti-Islam, anti-immigrant PVV party who is trailing in third.
He said his victory had stopped the "wrong kind of populism" in its tracks, after last year's Brexit vote and the election of President Trump.
Mr Rutte said: "We want to stick to the course we have - safe and stable and prosperous."
Mr Wilders, who wants to close mosques, ban the Koran and leave the EU, received much of the media coverage during the campaign.
In a tweet, the politician, sometimes called the Dutch Donald Trump, thanked his backers and warned: "Rutte is not rid of me by a long shot".
Meanwhile, the Green party saw a big rise in support, winning 16 seats, with their share of the vote up from 2% five years ago to 11% this time.
There was a high turnout where voters had 28 parties to choose from.
Coalition talks are now expected to last weeks or possibly months, with most of the main parties having already stated they would not work with the PVV.
Sky's Europe Correspondent Mark Stone said: "Geert Wilders has failed to do what many suspected he might be able to achieve which was to win the most seats in the Dutch parliament. That hasn't happened."
He added: "The populist uprising that many had expected - that Geert Wilders might be able to achieve - has not come to pass."
Mr Rutte, who had 40 seats in the previous parliament, vowed never to work with Mr Wilders again, turned off by his incendiary remarks and after the PVV caused an earlier coalition to collapse in 2010.
The election in the Netherlands came ahead of polls later this year in France and Germany, when right-wing nationalists will also be key players.
France's President Francois Hollande hailed a "clear victory against extremism" and Emmanuel Macron, the presidential candidate tipped to win the election, said it showed "progressives are gaining momentum".
But the secretary general of France's far-right National Front, Nicolas Bay, said he was encouraged by Mr Wilders' gains, calling the result a "success".
The head of German Chancellor Angela Merkel's office congratulated the Dutch on a "terrific" result.
Peter Altmaier tweeted: "Netherlands oh Netherlands you are a champion."
And EU Commission chief Jean-Claude Juncker hailed a "vote for Europe, and a vote against extremists".