I have two specific concerns/fears about directly asking strangers. Hopefully they can be assuaged in some capacity.
1) If someone is actively trying to pass, I feel like asking them could be taken as an implication that they're not doing a good enough job of it, and I don't want to offend.
Depends on how you ask. The accepted way to ask is "what pronouns do you prefer?". This does not imply that I'm being read trans, just that they're asking me what I prefer. Asking someone "hey are you trans?" while a bit more offensive, is still a better alternative than just being misgendered outright.
I would rather have someone take me aside and ask me before assuming, rather than put me out in the open that they read me. That gives me the chance to assert my identity, rather than my identity being asserted for me and then making me refute that.
2) As a tall, masculine neckbeard, I feel like asking around a lot risks painting me as a dude trolling the con for poon, and I don't want security to escort me from the premises.
I think if you stick to the pronoun question it will never come off as that. There's people who look and present female who may not actually prefer female pronouns at all. To assume that only trans people get to pick their pronouns is also kind of crappy and denies other gender variant people the right to self-identify. I know it may seem silly to you, and in some ways it even seems sort of silly to me. I'm firmly on the gender binary, there's no question I'm a girl and I want to be identified as such, period. But if something like pronouns makes someone feel a lot better, it's a small burden for me to ask them before using them.
The 'they' thing I am willing to try but I know I will cock it up repeatedly because work requires me to actively avoid using it as a singular pronoun.
English is kind of weird because gender is only assigned to people, not objects, so using something like "it" is extremely offensive because of that. "They" is a grammatically correct singular gender neutral pronoun, and is generally accepted as an ok substitute that doesn't imply any sort of gender. Some gender variant people actually prefer "they" and "them". If they would prefer something else, they will typically correct you. Correcting from "they" is a lot less stressful, as having to correct from he or she implies they are trans, when they may not necessarily be (or if they are, may not necessarily want the world to know).
This is why there's a big push in places like Sweden to make everything gender neutral. This isn't destroying the notion of gender. On the contrary, this is asserting it as a point of self-identification. By removing the assumption of gender from words like "mailman" and removing having to publicly identify yourself to do things like drive a car or goto the bathroom, it puts the power back to the person to decide if and how they wish to express their gender. There's no reason you need to know my gender on my license, nor force me to pick a bathroom based on how other people accept my identity. The only time it is relevant what parts I have is between my doctors, and my partner. That's it.
I have no problem with whatever you choose to identify yourself as, but it seems to me that asking everyone you meet what gender they want to be referred to as seems it could potentially offend some people or just make them think you're a weirdo. (like what Kilga said)
People you meet in real life for the first time should at least have a little bit of leeway, like a reminder or two, or forgiveness if it just slips out on the high of a (positive) emotion. I would hate to see someone ostracized for something they did without the intention of offending.
Putting it that way attempts to equate the discomforts as equal, and I have to say it really isn't on the same level. Having someone think you're a weirdo for asking pronouns and going "x, duh" is far less offensive than misgendering someone who may not necessarily have the strength of will to correct you. As it stands right now cisgendered people don't have to actively assert their gender, but I do. Think about that for a second. You're asking me to continue to have to actively assert my gender because you don't want to potentially have to. I'm not saying you are intending this, but that's how it comes off as.
Asking someone their pronouns is not taking away the rights of cisgendered people to identify themselves; merely, it removes the assumption that physical appearance is linked to gender. A cisgendered person doesn't have to actively assert their gender everyday, it's just assumed properly. Trans and other gender-variant people, on the other hand, have to actively identify themselves or be at the mercy of whatever box people want to toss them in. That's where the pressure to "pass" comes in, pressure to act or appear masculine or feminine in order to fit into the gender binary. Before I could potentially pass as female I tried very hard to pass as male to avoid ridicule and harassment. This forced a lot of psychological trauma on me that to this day I'm still dealing with. Now that I'm able to shed that persona a lot of my psychological problems and anxiety have gone away, but it's still somewhat of a struggle to deal with people who misidentify me. There's still a strong pull when I'm with some people to identify as male around them because it's simply "easier" to hurt myself and internalize the pain than potentially deal with the conflict that may arise from asserting myself as female. Some people don't even have that option and are forced to present a way in order to survive, and some people have no interest in even being on the binary. Yet they're still beholden to their physical appearance to determine what gender they get identified as.
Changing your lexicon to use they and them in place of <insert gendered pronoun> until you actively know it's ok to use it isn't all that difficult, yet it saves a ton of grief for gender variant people. When I was with Vic over the weekend he straight up asked me if it was cool if he used "dude" and stuff. I was ok with it. I really appreciated he asked though, because not everyone is comfortable with it, and saying "oh I say dude to everyone" doesn't make it any more right. When you meet someone new use they and them, and if you really want to use gendered pronouns simply ask them what pronouns they'd prefer or wait for them to assert them if you're uncomfortable asking.
I know it's not necessarily going to be easy at first, which is why I tend to be a lot more accepting about it when people flub. But realize it doesn't make it any less painful to me because I'm more accepting of it. It really doesn't. In fact it tends to hurt a lot more at cons and stuff where I feel like I should be more readily accepted, I tend to take my defenses down a bit more versus out in public where being misgendered could potentially mean violence. It's really awkward and sad how in the safe spot I get misgendered a lot more than I do out in public where it's far more riskier to be misgendered. The onus should definitely be on the person in a position of privilege to actively seek out ways to make things more comfortable for those who aren't, rather than assume they're the majority and to expect others to correct their behavior for them.