cat test.json | jq -c ‘.’ > testNDJSON.json
Our personal data superhero Alina Zahng has hacked a way to load JSON into BigQuery successfully. That, without the pain of debugging JSON parsing error in row starting at position 0 or unexpected end of string.
Alina uncovered the power of using Newline delimited JSON (ndjson) instead of standard json.
NDJSON is a convenient format for storing or streaming structured data that may be processed one record at a time.In order to load JSON files into BigQuery successfully each time, all you need is to Convert the file into NDJSON and load it from BigQuery UI. Sounds simple? It is. Click here to get the exact flow and example screenshots on Alina’s medium profile. And, what about that line of code on top? Still wondering what it is? With this simple line of code, you can convert and save files in NDJSON format. Query on 🙂