Skip to content

jsonrescue: every whole value out of JSON no parser will open

The problem

Much of the "JSON" on disk is not one document: objects written back to back with no array around them, a JSON Lines log whose writer died mid-line, a download cut off half way through a big array, a file that opens half way through an object. json.load refuses it all at the first stray byte. jq reads back-to-back values but stops at the first broken one: on a real 505-value stream with three values torn, it returned 10.

The method

  • Decode one value, note where it stopped, go on. json.JSONDecoder.raw_decode(text, i) parses a single value and returns where it ended. That reads {..}{..}, JSON Lines, RFC 7464 record separators and comma-joined values alike.
  • After a tear, skip only to a real top-level start. In valid JSON a nested { or [ always follows :, , or [. A bracket after anything else (cut-off text, a closing bracket) starts the next value. A bracket opening a line counts only at the column top-level values sit at, so the members of a pretty-printed value are never lifted to the top.
  • Enter a cut-off array, one level, and keep its whole elements; a half element is dropped. A whole array stays one value.
  • Report every tear by line and column, and the share parsed. Complete objects inside torn spans go to a separate file with --fragments.
def starts_value(text, p):
    """After a tear: does the bracket at p open a top-level value?"""
    q = p - 1
    while q >= 0 and text[q] in " \t\r":
        q -= 1
    return q < 0 or text[q] not in ":,["   # after these it is nested

jsonrescue export.json > values.jsonl         # one value per line
jsonrescue --array cut.json > fixed.json      # one valid array
jsonrescue --fragments inner.jsonl torn.json  # keep pieces of torn values

Exit 0 when every byte was accounted for, 1 when something was torn, 2 when the input could not be read.

Limits

  • A cut right after :, , or [ makes the next value look nested; it is skipped with the torn one (--fragments keeps it).
  • A tear inside a minified array loses the rest of that array.
  • Only objects, arrays and record separators are resync points.
  • NaN/Infinity become null, invalid UTF-8 becomes U+FFFD; both counted.
  • The whole input is read into memory.

Code

methods/jsonrescue/ in the methods repository: one Python standard-library file and a 39-check test, including 40,000-value streams with tears that must come back whole and in linear time.