Json parser

[Tasiro]

Hello, first time here.

I created a proof of concept json parser (link to gist). It was originally intended as a decoder for persistent storage of achievements or something similar, but apparently the file API makes a line based format rather more advisable.

Anyway, due to the way the only composite data structure with automatic deconstruction (the array) works in Danmakufu, I decided to put Json strings, arrays and objects into separate arrays.

A value is represented by an array containing two elements, the type represented by a number (e. g. 1 for a boolean) and a number representing either the value (e. g. 1 for true) or the index in the corresponding array containing the actual entry. This is done because Danmakufu does not allow an array of elements with different types (unless you cheat).

Json arrays are represented by arrays containing Json values:

Code: [Select]

[11, 22]
-->
jsonArrays = [..., [[JsonTypeNumber, 11], [JsonTypeNumber, 22]], ...]
//     at index 12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
//                                vvvv
jsonValueOfArray = [JsonTypeArray, 12]

Json objects are represented by arrays containing their values prefixed by the index of the key in the string table:

Code: [Select]

{ "key1": 33, "key2": [11, 22] }
-->
jsonStrings = [..., "key1", "key2", ...]
// at index 8 and 9 ^^^^^^  ^^^^^^
//                   vvv                      vvv
jsonObjects = [..., [[8, JsonTypeNumber, 33], [9, JsonTypeArray, 12]], ...]
//      at index 31 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonValueOfObject = [JsonTypeObject, 31]

Since it is impossible to put all of #1 the result value, #2 the string array, #3 the array array and #4 the object array into one Danmakufu array (again, without cheating), global variables are used for those instead.
Calling JsonParse("{\"key\": [11]}") returns [JsonTypeObject, 0], and the values of the global variables after the call are:

Code: [Select]

jsonResultStrings = ["key"]
jsonResultArrays = [[JsonTypeNumber, 11]]
jsonResultObjects = [[0, JsonTypeArray, 0]]
jsonParseErrorIndex = -1 // the index of the first wrong char otherwise; might be off by one

The graceful failure in invalid inputs was, as always, the most difficult part. 

As implied, it is possible to "cheat" the somewhat limited type checking which the Danmakufu Executor enforces:

Code: [Select]

let array = [[], []];
array [0] = [11];
array [1] = ["str"];
I could have put all those troublesome global variables into one array using this, but decided not to, not least because I kind of want to create an external type checker which is supposed to catch all those vexing type errors before testing, which would also enable better tooling support...

Since I found no practical way to convert \uXXXX escape sequences to Danmakufu chars, those are not implemented and considered a syntax error. (Is there any way to create a character with a given code point?)
While not actually part of the Json standard, C like comments are quite useful, but were also not implemented. Changing that is likely to require little effort.

The stringify operation could be implemented for this representation, but I decided not to, because the actual representation of data is likely to be quite different from this. I consider a recursive approach to be better suited. This also avoids having to check for circles and other silly things. 


The Json parser can be found as a gist on my GitHub account; if you find any bugs or have any suggestions, please leave a comment or notify me otherwise. 

Any recommended format for saving non-trivial persistent data using Dnamakufu?


Edit: Thank you for fixing the links, Helepolis.

[Sparen]

The standard for storing data in Danmakufu is with Area Common Data. Of course any form of complex structure immediately becomes a pain to deal, and the support for multidimensional arrays is pretty much just... horrible.

If it's just achievements, they can be stored as a binary array or using Common Data (which is how I handled my unlock system for SeitenTouji)

[Tasiro]

The standard for storing data in Danmakufu is with Area Common Data.

Ah, missed the part about being able to save them to files. Thank you.

Of course any form of complex structure immediately becomes a pain to deal, and the support for multidimensional arrays is pretty much just... horrible.

Would you mind elaborating? The implementation of ScriptCommonData::_ReadRecord and ScriptCommonData::_WriteRecord do appear to be correct. Maybe I misunderstood you?

If it's just achievements, they can be stored as a binary array or using Common Data (which is how I handled my unlock system for SeitenTouji)

It is intended to contain a bit more information, such as progress towards the achievement or maybe the date. I hope you don't mind me looking at your unlock system implementation.

[Sparen]

Would you mind elaborating? The implementation of ScriptCommonData::_ReadRecord and ScriptCommonData::_WriteRecord do appear to be correct. Maybe I misunderstood you?

I wasn't referring to your code; Danmakufu just has terrible support for multidimensional arrays.

I hope you don't mind me looking at your unlock system implementation.

It's ugly and horrible to look at. I have to fix array sizes in order to initialize them so that they can be indexed later, and it's a mish-mash of various techniques.

[Tasiro]

I wasn't referring to your code; Danmakufu just has terrible support for multidimensional arrays.

Those two functions are part of the source code of Danmakufu.

There are many languages without direct support for multidimensional arrays; the only additional problem I found in Danmakufu concerns the type checks, specifically if the first subarray is empty:

Code: [Select]

// array is of type number[][]
let array = CreateNumberArray (/* size: */ 0, ...); // returns [[]] for this call, which always has the type char[][], probably because strings are just
// arrays in Danmakufu and the empty array, i. e. the empty string, should not require any more additional special handling
array = array ~ [[1, 2, 3]]; // error: cannot concatenate char[][] and number[][] (or string[] and number[][])
That kind of error was actually the reason why I added additional type information to my copy of Danmakufu; I didn't know why Danmakufu considered that to be wrong. 

Edit: As Drake mentioned, it should have been [[]], not []. Fixed that. Thank you, Drake.

[Drake]

Here's a relevant post on types
Here's a typeof implementation, and a follow-up explanation of some stuff

As I mention in the above posts, concatenation has no problem combining any type of empty array with a different type of array. The particular issue you're having is because the array you're returning there isn't [], it's [[]], which is an array containing char arrays (note something like [[],"a"] is valid). You can concatenate [] ~ [[1, 2, 3]], for example.

A few more maybe-related edge cases on empty arrays I didn't cover previously:

Code: [Select]

let a = [];
a = a ~ erase([1],0);    // error, because the resulting array is num-typed (type changing)
a = erase([1],0) ~ a;    // works, because the resulting array is char-typed
print(a ~ erase([1],0)); // works, because there's no type restriction here