Author Topic: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)  (Read 130751 times)

UPDATE: Data modification is now fully functional!

So yesterday I decided to poke around the encrypted data files of GST:Y v1.08 out of boredom to see if I could somehow get the key to the encryption, since it's long been a major obstacle in creating a translation for the game. I noticed a repeating pattern of 40 bytes and figured it had been XORed with 00s, so I ran it over the file and, sure enough, it turned out to be the key. I was able to identify headers and the filename/size data at the end of the file, but was not able to extract the files successfully; this turned out to be a problem with the program I was using, which removed some data causing the files to become corrupt.

To solve the problem, I wrote my own small and dirty program in C++ to do the decryption, the source code of which can be found in a link below. To use it, compile and run the program with the data file you want to process as the first argument. The decoded file is created in output.dat (it always overwrites since I didn't add any safeguard, so be careful.)

From what I gather, these are the contents of each data file:
data1.dat - Images: Logo images (still messy?)
data2-1.dat - Images: Portraits
data2-2.dat - Images: Portraits, battle sprites
data2-3.dat - Images: Portraits, battle sprites
data3.dat - Images: Backgrounds, effects
data4.dat - Dialogue/Scripts
data5.dat - Dialogue/Menus/Scripts
data6-1.dat - Audio: BGM
data6-2.dat - Audio: BGM
data6-3.dat - Audio: BGM
data7.dat - Images: Backgrounds
data8.dat - Images: Units, icons
data9.dat - Images: Backgrounds
data10.dat - Data files
data11.dat - Data files: Animations
data12.dat - Data files
data12.dat - Audio: SE

Now that the game accepts the modified files, hopefully the translation can begin.

Links to the tools and instructions on how to use them:
http://pastebin.com/zKmeN0BV
« Last Edit: April 18, 2015, 09:59:16 PM by Yamsman »

Lukesky180

  • Knight of the Hungry Ghost
  • Yuyuko-sama!
    • HSE-Endeavor's Robotics Team
Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #1 on: March 25, 2015, 01:35:08 AM »
Amazing. This looks very helpful.
>This is the little mistress.
>Oh, hello little one!
Is this my dream, the destiny to behold?
~Sister of Scarlet~

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #2 on: March 25, 2015, 02:39:56 AM »
Wow. It's finally starting to happen. I'm guessing that since the key has been found, the only obstacle left is actually translating the strings. Re-encryption should be a trivial process now that you've found the key and algorithm.

Drake

  • *
Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #3 on: March 25, 2015, 03:17:14 AM »
Since all it seems to be is xoring, encryption is literally decryption.

A Colorful Calculating Creative and Cuddly Crafty Callipygous Clever Commander
- original art by Aiけん | ウサホリ -

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #4 on: March 25, 2015, 07:21:31 AM »
First off, thanks a lot for doing this!

Second, I tested it out and data4.dat is actually battle dialogue (and a lot of it is gibberish for some reason) while data5.dat has everything else. More pertinently, though, when I tried re-encrypting the .dats and testing the .exe, it gave me a memory error and refused to load. Well okay, maybe a straight japanese-to-english replacement won't work. Except then I tried decrypting then, without changing anything, reencrypting and got the same problem. So that's bad!

If this problem is fixed I should be able to translate the entire game with the help of a few friends. Hope you can do it!
« Last Edit: March 25, 2015, 07:45:43 AM by BlitzBlast »

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #5 on: March 25, 2015, 10:48:39 AM »
I just played around encrypting and decrypting data5.dat and running an encryption/decryption loop turns most of it in to garbage data, with strings of correctly preserved characters mixed in. It also makes the file larger on both encryption and decryption.

I tried removing the actual encryption part of the program so it just reads data5.dat and writes its characters without doing anything in to output.dat. Even then the output file has 1500 random linebreaks added to it, although the data is otherwise exactly the same. GSW2 still rejects the new file and crashes though so I guess the whitespace is important.

So yeah, something's happening either at the character storing or writing step that's making files that aren't exactly the same.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #6 on: March 25, 2015, 12:21:12 PM »
Eh-heh...

Yeaaah, to be honest, I actually found a way to extract the files awhile ago and actually translated a teensy bit(http://i.imgur.com/7l8PSX6.png - see modified date). The problems I had was putting it back in, although the method I used seems different from yours (I made a hacked exe that read the files from the dats and dumped it into my pc instead of loading it into the game) but doing the reverse stymied me a bit. Then LoT2 happened, and then real life happened.

If you manage to find a way to convert the scripts into a format that's readable by the game, AND find a way to get the strings in the exe translated while still having the exe entirely workable, that'll be great. I'll also be glad to help on the translation effort now too since real life has let off a bit, whether it be translation checking or translating the whole thing or whatever - I still feel a bit unfulfilled from not following up from the first game. If not, I might try poking around with the methods I used a year ago, though I'm kinda rusty in that now...

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #7 on: March 25, 2015, 03:52:23 PM »
I think Yamsman made his program on an unix computer. Windows handles end of lines differently and, by default, tries to support unix text files as well as windows text files. This leads to some end of lines conversion, which isn't wanted in our case.
To fix it, replace the line 30 by :
Code: [Select]
FILE * ofile = fopen("output.dat", "wb");

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #8 on: March 25, 2015, 07:48:38 PM »
Changing that gets the game to accept the .dat files after a straight decryption/encryption loop, but the fact that spaces matter so much predictably also means that actually changing the original Japanese strings leads to the exact same memory issues. I guess it might work if the english text had exactly the same length as the japanese (in bytes, not word count), but that would lead to a clunky at best translation. I'd assume the problem at hand is in the .exe itself, which brings us all back to the original problem that stymied everyone for so long: it's fairly easy to decompile the .exe (it's just Game Maker), but recompiling is a whole nother story.

Just having access to a text dump is a huge boon though, even if this doesn't ultimately pan out. Once again, thanks!

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #9 on: March 25, 2015, 10:37:59 PM »
I think Yamsman made his program on an unix computer. Windows handles end of lines differently and, by default, tries to support unix text files as well as windows text files. This leads to some end of lines conversion, which isn't wanted in our case.
I did write the program on Linux, and I was unaware that it would be different for Windows. Thank you for letting me know, I updated the code in the pastebin.

changing the original Japanese strings leads to the exact same memory issues. I guess it might work if the english text had exactly the same length as the japanese (in bytes, not word count), but that would lead to a clunky at best translation.
I should have included this information earlier, but the reason why it won't work is because there are 4 bytes before the start of each file's data that tells the length in bytes. When you change the size, those bytes must be modified as well to accomodate the change.


As of now, I'm working on a set of tools to unpack the files for modification and then repack them into a data file. Ideally it will make it possible to freely edit the files without having to worry about changing the size bytes. EDIT: The unpacking tool is complete, I've added the link to the OP.
« Last Edit: March 27, 2015, 11:49:07 PM by Yamsman »

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #10 on: March 29, 2015, 11:54:06 AM »
Wow. This sounds like good news. Hopefully, a translation of this game can be done.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #11 on: March 30, 2015, 11:43:32 PM »
Well, I finished the repacking tool and added the link in the OP. Everything works perfectly if the files have not been modified, but even though the repacker changes the bytes that correspond to file size, the program will still refuse to run. At this point I have absolutely no idea what to do to make it accept the files; hopefully someone who has actual experience in creating translation patches can make something out of all this.
« Last Edit: March 31, 2015, 05:00:14 AM by Yamsman »

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #12 on: March 31, 2015, 03:25:13 PM »
Line 68 in repack: you include filename as a parameter to printf, but don't include %s in the format string.

Also, I highly recommend making functions to swap endians for 4 byte ints. It's a really common operation, so that will cut down on copying/pasting

I don't have the game so I can't run this to be certain, but I think I see the problem. If I'm not mistaken, you're reading in the fileoffsets during extraction and storing them into your info file. Then during repacking, you read in those same fileoffsets and reuse them. The problem is that if the filesizes change, so will the file offsets.


Finally, you should really comment your code better. Right now there's just comments on what the program is doing, but nothing on 'why'. Considering that this is a program to handle an archive file, there should at the very least be a comment describing the format of said archive file.

Here's an example from a game I decrypted:
Code: [Select]
//Header:
//4 bytes "PACK"
//4 bytes file count

//Each file entry is
//x40 bytes Filename
//4 bytes unknown (Looks like a sorting key?)
//4 bytes CRC of compressed file
//4 bytes FileStart
//4 bytes FileSize

//Filestart is rounded up to the next 00 on creation; e.g. if the last file ended at x1027, the next file begins at x1100.
//Files may be (always are?) LZSS compressed.
//This is given by the "LZSS" header (on the file), followed by 4 bytes original size(?)

//Total of x50 bytes per entry.

This alone makes the code much easier to read for someone unfamiliar with it... including yourself, if you ever come back to look at this after 2 years.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #13 on: April 04, 2015, 06:33:09 PM »
Hey guys,

I just wanted to offer my help in the translation effort if we can get repacking the files working.  I took a look at the files myself and played around with them, but I didn't figure out much.  The most I was able to do was change a few in-game attributes like Reimu's size.  I just wanted to tell you guys and anyone else working on the unpacking/repacking issue that we really appreciate your work, and based on the comments and what others have said before in these kinds of threads, I don't believe we'll have much problem with the actual translation if we get this working.  I think this is about the closest this translation has ever gotten to happening, and I hope someone will be able to figure out the problem soon.

Again, thank you for the work Yamsman and anyone else who took a crack at the problem!

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #14 on: April 09, 2015, 12:29:25 PM »
I have talked to sando few years back he is creating all his gensou shoujo taisen series by using YoYo Game (game maker). I not sure this will help or not but if you have decrypted yoyogames before then this would be easier by knowing what kind of tool he is using to create his games.

Alcoraiden

  • Do not go gentle into that good night
Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #15 on: April 14, 2015, 11:22:25 PM »
Good lord, it's like everything I ever wanted from this is suddenly happening.

Does this tactic work with Ei too?

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #16 on: April 18, 2015, 09:57:32 PM »
Alright, I have some really good news. I took a look at the executable to see how it functioned and it turns out that sections of data I previously disregarded are used to point to the position of the packed files in the archive. After adding in the functionality for these in the repacker, the game now accepts the modified files! The updated source code has been added to the pastebin in the OP.

Here's a screenshot using the translated lines that Deranged posted:


Finally, you should really comment your code better. Right now there's just comments on what the program is doing, but nothing on 'why'. Considering that this is a program to handle an archive file, there should at the very least be a comment describing the format of said archive file.
Yeah, sorry about that, I'm really lazy when it comes to comments since I never really publish code I write. I added more to the code, but I really should have put them in in the first place.

Does this tactic work with Ei too?
I haven't really looked at it, but from what I could tell it's not the same.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #17 on: April 19, 2015, 12:14:02 AM »
Great job! Guess it's time I start copying over everything I have so far.

Oh god now I have to worry about space limits.  :ohdear:

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #18 on: April 19, 2015, 04:39:44 AM »
Excellent work. Will there be any work on the executable portion though? I do recall important stuff like the spirit descriptions and values, name entries and some other images should be in there.

Great job! Guess it's time I start copying over everything I have so far.

Oh god now I have to worry about space limits.  :ohdear:

Do you have a complete translation already? Would you need any help on that front?

By the way, if it works anything like the first game, space limits isn't a huge problem. If you do overflow in a single line, you can split it up into two lines and create a new line. The game will accept it just fine. As an example:

Akyu0b:(The ... day of the ... month. It is clear again today. The fierce sunlight is slowly abating as Gensokyo prepares to welcome autumn.)

Can become:

Akyu0b:(The ... day of the ... month. It is clear again today.)

Akyu0b:(The fierce sunlight is slowly abating as Gensokyo prepares to welcome autumn.)

And in the game, what was originally one textbox will show as two different textboxes in sequence.

You can even change the 0b part to a different value to call a different face portrait if it fits the mood better.

Encylopedia entries also won't have the same problem since they have infinite space IIRC.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #19 on: April 19, 2015, 05:25:34 AM »
Yeah, I figured most of that stuff out already. Having more issue with getting the UI to cooperate.

I've been working on a FMW2 translation LP since last year, and recently two other people joined in. Translations aren't complete since we don't grab all the boss conversations or get all the alternate dialogue chains, but we've gotten all the main stuff. We're up to the 28s at the moment.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #20 on: April 19, 2015, 06:27:06 AM »
All right, sounds like you have things covered. Good luck!

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #21 on: April 19, 2015, 06:54:57 AM »
@Deranged and @BlitzBlast: OK, now that a translation for GST02 is doable (and seems to be in the works right now), there's an issue that needs to be brought up. This one right here. Apparently, the English patch for GST01 (the patch that Deranged made), has a bug that gives Flandre the wrong item for her fight - an Oni's Nail instead of an Oni's Fang. Isn't that something that should be solved first, since there's no way to get this item otherwise?

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #22 on: April 19, 2015, 07:03:32 AM »
Just switch to the japanese .dats before you take on chapter 15.  :V

You get a second Oni's Fang pretty early on in FMW2, so it's really not much of an issue unless you really want 100%. You can even hack the number you have to 2, it wasn't until FMW3 that Sanbondo really started being stringent about that.

Also as far as UIs go, I've found that everything can be smoothly translated except the spirits. Presumably it's because they're handled in the .exe like Deranged said. I got the spirit names working once, but then it broke every other time so I dunno. It's kind of disappointing, but I guess it's good enough.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #23 on: April 19, 2015, 01:25:06 PM »
Just switch to the japanese .dats before you take on chapter 15.  :V

I tried to do it. But, she does not equip the item.
Not that this is a problem, would be ... just unpleasant.



UPD:
Hmm, I realized what the problem ... anyway, it would be good to fix the patch, if possible.
« Last Edit: April 19, 2015, 02:48:05 PM by Nikkanoffun »

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #24 on: April 19, 2015, 08:35:11 PM »
Alright, I have some really good news. I took a look at the executable to see how it functioned and it turns out that sections of data I previously disregarded are used to point to the position of the packed files in the archive. After adding in the functionality for these in the repacker, the game now accepts the modified files! The updated source code has been added to the pastebin in the OP.

Here's a screenshot using the translated lines that Deranged posted:

Yeah, sorry about that, I'm really lazy when it comes to comments since I never really publish code I write. I added more to the code, but I really should have put them in in the first place.
I haven't really looked at it, but from what I could tell it's not the same.

Sounds like good news! So a few of the games files have been already decrypted!  :V  :V  :V
I wish you guys good luck in translating the game!
1CC Easy: UFO , MOF, PCB, DDC, GFW, POFV, TD, IN, SA, 1CC Normal: MOF, DDC, IN, POFV, PCB, TD, EOSD, HSIFS
Extra Mode: Ran Yakumo, Okina Matara

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #25 on: April 20, 2015, 09:34:41 AM »
So, uh, for some reason the modified data5.dat breaks the Unfocused/Focused properties on attacks and makes it so you can do them while in either. And while it's admittedly pretty hilarious to run around spamming Fantasy Seal and Master Spark, it's not really a good thing. I'm not entirely sure if it's an issue with the executables or a side effect of me unpacking on a mac and repacking on windows (because unpack.exe fails on Windows), but it's pretty weird.

I guess this is the Infinite Alert of this game?  :V
« Last Edit: April 20, 2015, 09:42:09 AM by BlitzBlast »

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #26 on: April 20, 2015, 12:36:17 PM »
@Deranged and @BlitzBlast: OK, now that a translation for GST02 is doable (and seems to be in the works right now), there's an issue that needs to be brought up. This one right here. Apparently, the English patch for GST01 (the patch that Deranged made), has a bug that gives Flandre the wrong item for her fight - an Oni's Nail instead of an Oni's Fang. Isn't that something that should be solved first, since there's no way to get this item otherwise?

Unfortunately, I no longer have the tools that were used to make the patch (lost in a hdd crash), and I lost contact with the original maker of the tools. The fastest way to do so would be to restore the JP dats as BlitzBlast mentioned.

Though technically, if the tools in this topic work on FMWE as well, it should be easy enough to fix - get Flan's enemy data out from data4 english version, put the correct item in, and repack with all the same files otherwise. I don't even have the original game's dats around to test this, but if someone else could, that'd be great.

So, uh, for some reason the modified data5.dat breaks the Unfocused/Focused properties on attacks and makes it so you can do them while in either. And while it's admittedly pretty hilarious to run around spamming Fantasy Seal and Master Spark, it's not really a good thing. I'm not entirely sure if it's an issue with the executables or a side effect of me unpacking on a mac and repacking on windows (because unpack.exe fails on Windows), but it's pretty weird.

I guess this is the Infinite Alert of this game?  :V

Infinite Alert occured because of the way spirits were done in the exe. Their functions, SP costs, etc were all calculated using their display name instead of a internal variable name (unlike a lot of other variables). Thus, all instances of 閃き had to be replaced with "Alert" within the exe. Which would've been fine, except I put "Flash" for one instance subconsciously, and that instance was the one that calculated the SP cost of the spirit. End result: When using Alert, it couldn't find the string "Alert" in the list of spirits and their costs, so the game gave it to you for free.

I'm assuming something similar happened here where the weapon property may have been changed in the dat files but not in the exe, thus nullifying the weapon property entirely since it can't find a "Focused" string (or whatever it was translated to) in the exe. Though from what I recall, data5 was the story scripts and data4 was the battlequotes/unit data in FMWE... but that's my best guess either way.

Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #27 on: April 25, 2015, 08:50:31 AM »
Okay, the first version of the menu patch is complete. As far as I know, it translates everything pertinent to gameplay that was actually changeable without access to the .exe. In other words, no spirit names. It also didn't get anything in data4.dat (names, mostly), but we're working on that now. To use it, backup your old data5.dat and replace it.

As for the actual script, there are some kind people with lots of free time currently copy and pasting stuff in like their life depends on it. We'll probably try and get the main dialogue up first before getting to alternate stuff like boss convos or "the boss says something different if you timed out the spell instead" or "literally any other option besides Alice in 29M".

Soul Devour

  • A Casual Touhou Fan
  • Drifting along, one day at a time
Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #28 on: April 25, 2015, 04:15:54 PM »
I'd just like to say thank you so much for all the hard work everyone is putting for this. I'll be honest, I never thought real progress would ever get made on this game so it really is awesome to see it finally being translated.

You all rock!

Prime32

  • Munch-Munch Demon
Re: Gensou Shoujo Taisen You - Translation (Decrypted .dat files)
« Reply #29 on: April 26, 2015, 02:32:21 PM »
As far as I know, it translates everything pertinent to gameplay that was actually changeable without access to the .exe. In other words, no spirit names.
While it's a crude way to do it, and you'd need to shorten them, would it be possible to change the spirit names by altering the font?