Also, even if the cast were all young girls, why would that make you a lolicon? That's like saying playing Halo makes you massively gay for men in uniform.
In pure theory, yes.
However, in my experience, the effect is similar to being the only person to possess a gun at the scene of a murder. Even though there is no proven correlation, people are immediately going to suspect you were the killer. Yet, take the same situation, except everyone at the scene of the crime possesses a firearm. Everyone is still a suspect, but since having a gun is the norm rather than the exception, the chance of each person being the killer appears to be lower.
My friend concluded the only reason someone would play a game filled with little girls would be to fulfill lustful desires toward little girls. Since no one else on my campus has even heard of Touhou, he figured the game's obscurity was due to how the premise can appeal to a specific demographic, which he envisioned to be male lolicons.
Even though I am not a lolicon, I can understand why he made the accusation. I am sure some Touhou fans came for the little girls (or became lolicons as they engaged with more Touhou material). After reading more than a few perverted Touhou doujinshi, I have no choice but to say he is right in some instances. On the other hand, most Touhou doujin I have encountered are appropriate for all ages, which gives me faith in the goodness of humanity.
I'll tell you, in my last 7 years of studying, I've met 0 people IRL who had a high interest in anima/manga, let alone Touhou. You only run into those if you take Japanese classes or partake in those types of communities.
In my anecdotal experience, I have met many people throughout middle and high school who would have called themselves anime and manga fans. I did participate in my high school's anime club, but for the most part, I found them by overhearing (eavesdropping) on their conversations. Although I can imagine the situation would be different outside of the United States, where anime and manga may not be as common an interest.
Since I found out about Touhou right before I graduated high school, I could not confirm the existence of any fans there. So far, I have only met six or seven people (out of 300) who told me they watch anime at my university, and none of them said they knew about Touhou. This is probably the case everywhere with most niche interests. The "envy" part of my last post comes from the assumption I am in the same culture as most other posters (American), and thus I am jealous of other American schools with larger anime/manga fanbases. I realize I may be making an unfounded conclusion here, so anyone outside of the US should correct me.
But yeah, I guess Japanese culture naturally diffuses to me, being Asian and all.