The sign appears to show the silhouette of an orthodox Jewish man wearing a traditional Fedora hat and was found 200 yards from a synagogue in Stamford Hill, north London, on Tuesday.
It was reported to police by a member of Shomrim NE London - a local Jewish neighbourhood group.
The group's operational supervisor in the area, Barry Bard, said the incident had caused concern in the community because it seemed to be well planned.
He said: "The people of Stamford Hill are very sadly used to instances of anti-Semitic hate crime, but most of those times it will be verbal abuse or even assault.
"A lot of the time it will be more of a person-to-person kind of thing, or graffiti, which is more unprofessional.
"The person who planned (this sign) has obviously gone to an effort to cause alarm and distress to local people."
Diane Abbott - the area's MP and shadow home secretary - said the sign was "disgusting" and "unacceptable".
Labour's Tottenham MP David Lammy said the sign was an example of "despicable, nasty behaviour that has absolutely no place in our community".
Stamford Hill is home to a community of around 30,000 Haredi Jews and there are said to be around 50 synagogues in the surrounding area.
A survey last year by Shomrim recorded 32 anti-Semitic incidents in a month in the area.
Among the victims were an eight-year-old boy who was beaten up on his way home and a woman confronted by a man giving a Nazi salute.