Formatting JSON to allow mapping of data ingestion through Data Landing Zone detect the structure of object arrays
I am facing a problem where the structure of object arrays are not correctly detected when trying to ingest data into a Dataset from a JSON through Data Landing Zone.
Basically, we have object array fields where the size is not fixed, and in some (most) rows of data, the array is empty. Here is a simplified example.
When given the following JSON:
[{},{},{},{},{"cooperations": [{"a": 1,"b": 2}]},{"cooperations": [{"a": 1,"b": 2},{"a": 3,"b": 4}]}],
the mapping detects the object structure as so.
However, when I increase the number of empty objects at the beginning of the above JSON to 90 empty objects, the mapping does not detect the object structure.
When explicitly using empty arrays, like so:
[{"cooperations": []},{"cooperations": [{"a": 1,"b": 2}]},{"cooperations": [{"a": 1,"b": 2},{"a": 3,"b": 4}]}]
the object structure is not detected. Same thing goes for when I try to use null.
Moreover, an attempt to do object-to-object mapping returns an error saying that there are no compatible fields.
Please advise on the formatting of the JSON to be able to cater for my use case.
