I'm pretty sure it could be what the characters will be like. Gameplay is fine, you can't copyright gameplay mechanics (but god have companies tried), it's mostly with the characters really. If your Wii U version has character that are still very similar to Touhou characters from your TSSB game then there could be issues. If they look completely different but have a color palate similar then I believe it to be fine. Look at all the fighting games that do this, Filia from Skullguirls has a Fiona color pallet from Adventure Time. Having a maid character the uses knives is mostly pushing your boundaries, why not have green haired pigtailed elven maid who uses a giant hammer that inexplicably explodes into tuna when ever she attacks? That's just an example but it's a way to show you can use them as a base but you can turn them into something else entirely. There are many characters based on other things but as long as the end result is not, "Wait that's just X with a different hairstyle!" then you should be fine. I've made characters based on some Touhou characters and after working on them for long enough they became their own person with their own personality from what they were based on. (Ex in my case Yuugi and Utsuho)
I don't think you realize how much work that would be. The bulk content of fighting game development is around individual characters, rather than primary game mechanics. Animation work in particular is very heavy in comparison to everything else. I agree having completely different characters would be best, but that would be a stupendously difficult thing to do.
What I'm curious about is whether it's possible for a commercial game to later go doujin. [...] If I make game A commercially, can I then make game B from game A and call game B "doujin" because the differences from game A satisfy the doujin guidelines, or I can't because the basis for game B (game A) did not in the first place?
It would be impossible for a commercial game to somehow branch out as a doujin game. If you really intended the project to go doujin, it would be changed to free to play... but that's still isn't really doujin per se, it's just a now-free indie work. Expanding on that, trying to paste someone else's doujin IP onto your work after the fact, even if you make your original work free, would be IP theft at worst and just look like a pathetic attempt to grab attention at best. You'd
better have some sort of solid deal with the IP owner to pull off something like that. On a completely different note, if you have a game you published commercially but are making side games based off of it like pet projects, that seems fine, especially if it's your own IP.
@ N-Forza, A lot of things that supposidely are beyond doujin happen to doujin's though. Even though I've been told that Steam is a nogo for doujin because most people on steam don't know doujins, doujin games do exist on steam (just one example).
Three important considerations:
1) Doujin games that get released on Steam are games that were already released as doujin titles. Getting published on Steam is an expansion into indie territory, although they're still considered "doujin games" because of their origin. Generally this is seen as a good thing because everyone likes to see people's hobbies being able to expand into something sustainable.
2) These games are the IP of their creators. These doujin games are not derivative works based on other people's IP.
3) Steam isn't really a "no-go for doujin" besides the under above consideration that they're more accurately labeled as indie works. It's Touhou derivative works in particular that are disallowed from wide-reaching publication.