I think the decision should have been yours. If 100 people were there, you could have easily avoided the 5.
I was once the victim of a massive match-making scheme. A brother liked me and my friend gave him the impression that I was as good as his. So he has high hopes and is telling tons of people about thise scheme to get us together. I eventually found out and told my friend that she needed to handle this because I was in no way interested. Even though it wasn't my fault, I felt like the bad guy. It stinks becuase I then felt uncomfortable around this guy who was a good friend of a friend. Anyhow, news travels, you can't help it.