Only a couple of times during my photography career have I been hassled. I guess if I worked harder at it, I might have had it happen more often.
It turns out, however, that law enforcement officers and security guards that are not familiar with the rights of photographers have been confiscating film and detaining photographers in defense of "National Security." Avoiding the situation if possible, is probably the best advice, but knowing your rights is a good backup. Bert P. Krages is an attorney who has written a book on the subject and published a one page flyer that is available for download from the web.
qv: The Photographer's Right - A Downloadable Flyer