Joseph Sincavage put the flag up on Sunday to go along with his American, Marine Corps and Vatican flags.
He said he feels there's no other way to protest against President Barack Obama's policies.
"I had uncles who fought to destroy that flag in the Second World War," said Sincavage, who feels he's merely exercising his First Amendment rights. "I'm not encouraging anyone to burn cars or go out and loot. ... To me, this seems to be a very simple way to say 'I disagree with what's going on.'"
Neighbors disagreed so much with it that Stratford Police showed up at Sincavage's home to ask him to remove it.
"He's just a racist. That's the bottom line," said Jeff Jones, who's lived on Connors Lane for 10 years. "What would you have an American flag next to a Nazi flag? That makes no sense?
John O'Grady, of Stratford, said there are other ways to protest.
"It's kind of a weird way to protest, flying a Nazi flag," he said.
"We're all taxpayers around here and no one should have to come out their doorstep to look at that nonsense, that's because that's straight evil," Jones said.
To the neighbors who say take it down Sincavage responded, "My answer is no. It's staying up. It's my property and that's what I intend to do with it," he said.
Sincavage, who flew a Confederate flag in the same place six months prior, said he took an oath in the Marine Corps to defend the Constitution.
"Whatever kind of attention he's trying to attract from it ... he's going to get it,' said Jones.
Stratford City Council members said, for the most part, that while the flag is offensive, there's nothing that can be done.
Sincavage said, weather permitting, the flags will be back up Thursday morning.
See Video Here