Yeah...I would certainly think hard about it before making a decision....not everyone views it the same way, either, so if you did decide to go, I wouldn't advertise it to everyone....some people tend to judge, even though they aren't supposed to.
And if you do go, be careful....I went to school with the same people for 12 years as well, and none of them ever pressured me to do stuff, up until my junior and senior years....depends on how comfortable they feel around you, and whatnot. People surprise you at times.