Many of those opposed to the proposed Islamic cultural center (and mosque within) say they're all for religious freedom -- and then comes the "but." But not Islam. But not there. But not if it offends 9/11 families.
"But" is un-American.
The entire point of the First Amendment's protections of freedom of speech, assembly and religion is that these rights can't be restricted based on their substance. Of course there are "time, place and manner" restrictions on speech and prayer -- so people are not entitled to talk at 120 decibels in a residential neighborhood no matter their religion or views. Zoning laws can prohibit incompatible or dangerous activities, like a liquor store near a school. And obviously government can punish the bad conduct of anyone claiming to act in the name of a religion or group: For example, would-be Times Square bomber Faisal Shahzad or anyone conspiring with him.
What we cannot do, however, is condition speech or religion based on its content. That's the bedrock of the First Amendment.