• ZZartin - Thursday, March 7, 2019 7:42 AM

    One super annoying trend I'm seeing more of is people wanting to use their real time and or EAV interfaces for bulk data transfer.  I mean I'm sorry but when you 90% of your fields are defined in EAV or your real time interface is some bloated JSON/XML and you want to transfers hundreds of thousands or more records every day through it you're in a narrow window not going to have a good time.

    I feel your pain. I haven't done anything with JSON in SQL Server but I know that JSON Path, unlike XPATH does not have a getParent() equivalent function which means that bulk ingestion of JSON containing arrays produces two unconnectable recordsets.
    The approach we've had to address it is import a file containing many JSON documents of the same document type and loop over each document triggering multiple extractions.  In effect reinventing RBAR.
    I am currently experimenting with ways of bulk ingesting the non-array part of the document, which is very fast, and then submitting those documents containing arrays to the RBAR process.

    I've had some luck with a library called YAJL but no son of mine will every be called JASON.