JSONTokenParser

[Source]

Streaming, incremental JSON token parser.

Feed bytes as they arrive with feed(); the parser walks the JSON structure to any depth and pushes tokens to a JSONTokenNotify callback as they complete. It builds no tree — its working memory is the container-depth stack, the single string or number it is part-way through, and the fed bytes it has not yet consumed. A value split across a chunk boundary is held and stitched onto the next feed(), so you never manage leftover bytes.

let parser = JSONTokenParser(
  object is JSONTokenNotify
    fun ref apply(p: JSONTokenParser, token: JSONToken) =>
      match token
      | let k: JSONTokenKey    => // k.value
      | let s: JSONTokenString => // s.value
      | let n: JSONTokenNumber => // n.value
      | JSONTokenObjectStart   => // ...
      end
  end)
parser.feed(chunk)?

It parses a stream of top-level values: after one value's tokens end it continues to the next, so newline-delimited JSON or a socket delivering messages back to back works with no per-value setup. Feeding a whole document at once is just the case where every byte is already in hand.

A number is the one value with no self-delimiter — the parser cannot know a number is finished until a following non-number byte arrives, so a number at the very end of the fed bytes is not emitted until more bytes come or you call finish(). finish() says "no more bytes are coming" and completes it.

Malformed input raises from feed()/finish(); read describe_error(), token_end(), and line() for the location. After a raise the parser latches: every later feed()/finish() raises again. abort() (called from the notifier) stops the parse the same way. For untrusted input, JSONParseLimits caps nesting depth and the length of a single string or number.

To turn a token stream back into a JSONValue, use JSONReassembler. For a whole in-memory document, JSONParser.parse() is simpler.

class ref JSONTokenParser

Constructors

create

[Source]

new ref create(
  notify: JSONTokenNotify ref,
  limits: JSONParseLimits val = reference)
: JSONTokenParser ref^

Parameters

• notify: JSONTokenNotify ref
• limits: JSONParseLimits val = reference

Returns

• JSONTokenParser ref^

---

Public Functions

feed

[Source]

Append a chunk of bytes and emit every token now derivable from the bytes in hand. Holds any partial value to be finished by a later feed(). Raises on malformed input or abort(); after that the parser is done and every later feed()/finish() raises. Do not call feed() or finish() from within the notifier — the parse is running and a re-entrant call raises.

fun ref feed(
  data: ByteSeq)
: None val ?

Parameters

• data: ByteSeq

Returns

• None val ?

---

finish

[Source]

Signal that no more bytes will be fed. Completes a pending number (the one value that otherwise waits for a following byte) and emits its token. Raises if that number's text is not valid, or if aborted/already done. A value left structurally incomplete (an open container, an unterminated string) is simply not completed — the consumer sees the truncation via incomplete(). After finish() the parser is done; feeding more raises.

fun ref finish()
: None val ?

Returns

• None val ?

---

abort

[Source]

Signal the parser to stop. The current feed()/finish() then raises, and the parser is done. Call this from the notifier when you have seen enough.

fun ref abort()
: None val

Returns

• None val

---

token_start

[Source]

Byte offset where the current token starts, absolute across all feeds. Valid only during the notify callback.

fun box token_start()
: USize val

Returns

• USize val

---

token_end

[Source]

Byte offset just past the current token, absolute across all feeds. Valid only during the notify callback.

fun box token_end()
: USize val

Returns

• USize val

---

line

[Source]

Current line number (1-based).

fun box line()
: USize val

Returns

• USize val

---

incomplete

[Source]

True while a value is part-way through — either mid-scan on a leaf or with a container still open. After feeding a complete document and calling finish() this is false; if it is still true, the input ended in the middle of a value. This is the byte-level truncation check; JSONReassembler.mid_value() reports only whether the reassembler is holding a partial value's tokens.

fun box incomplete()
: Bool val

Returns

• Bool val

---

describe_error

[Source]

Human-readable description of the most recent error.

fun box describe_error()
: String val

Returns

• String val

---

parse_error

[Source]

The most recent error as a JSONParseError (message, byte offset, line). Meaningful after a raise from feed()/finish().

fun box parse_error()
: JSONParseError val

Returns

• JSONParseError val

---