Some 64% of voters said “No” to the agreement, with 36% voting in favour of it, pollster Ipsos says.
However, it is not clear if turnout has reached the 30% threshold of eligible voters needed to be valid.
Prime Minister Mark Rutte said his government may have to reconsider the treaty if the vote is valid.
The Dutch parliament has already ratified the EU agreement and the result of the vote is not binding.
“It is clear that ‘No’ have won by an overwhelming margin, the question is only if turnout is sufficient,” Mr Rutte said in a televised reaction, according to Reuters news agency.
“If the turnout is above 30% with such a large margin of victory for the ‘No’ camp then my sense is that ratification cannot simply go ahead.”
The referendum was triggered by an internet petition begun by Eurosceptic activists that attracted some 450,000 signatures.