Yeah, I'd draw the line at public art which is actually physically hazardous to the public.
On a similar bent, I remember reading last year about some university students (both sexes, if I remember correctly) making a giant snow phallus which was promptly destroyed by some female students who found it offensive. "If they have the right to build it in a public space, we have the right to destroy it!" they angrily argued. Insofar as both activities are probably legal, I guess they do have the right to destroy it -- but I do feel that destroying art requires a greater moral justification than the act of creating it; I don't think they did have the moral "right" to destroy the giant snow penis just because it was on public property and offended them.
Perhaps would bothers me the most is that the way they chose to destroy it -- knocking it over -- is just so uncreative. If they wanted to protest against the giant phallus, they at should at least have thought of a clever way of co-opting the image or destroying it. Knocking it over doesn't cut it.

There was a beautiful giant snow phallus on the main green on Brown a few years ago, but I only remember it being there one night.
I think those condo dwellers should be given a few thousand bucks to tint their windows, and that's about it. It saddens me to think that the symphony hall is going to be changed so that a few rich people can relax. The value of having good art and architecture in public far outweighs the value of keeping every citizen's condo comfortable.

It sounds horrendous. When I am appointed supreme executor of the world, I will make sure that only attractive pieces of art are put in place. Naturally, it is in everyone's best interest that this happen, so I shall expect my letter of confirmation soon...

Its really a matter of values here. Do we value the aesthetics of something over the right to have a peaceable living space for a handful of people? Generally speaking, I think the aesthetics of something has generally low value, mostly because of the practical implications of placing too high of a value on the aesthetic. Suddenly the practicality of the thing is less important than the appearance or artistic statement of the thing.
Look... I want to make an aesthetic statement by playing Nine Inch Nails loudly outside my home. But my neighbors want to live in a quiet neighborhood. Who's value trumps? Despite the fact that I know a lot of people would enjoy the fact that my home would be that house the blasts NIN all the time, the people who live with it everyday would not be very appreciative. Their rights are being trampled by the piece of art.
What is more disturbing is the increasingly abstract nature that artists give their artworks. The GFP bunny by Eduado Kac is supposed to include not only the glow-in-the-dark bunny itself, but the controversy and dialogue that forms around it. This comment is part of the GFP bunny "installation." Had Gehry said that the heat generated by the building is a PART of the installation, would make it even more problematic. Its one thing for a piece of art to annoy people, its another to say that the annoyance is part of the work. But increasingly artists are saying just that, its the reaction to the work that is the art itself.
But this gives artists, artistic license to offend, harm, bother, annoy people, for ART'S sake.