Limitations

The 64 MiB input-size limit

Every read of a Document from text refuses an input of more than 64 MiB (64 * 1024 * 1024 = 67,108,864 bytes) with document.limit.input-size at $ (omnist-spec D-23 to D-26). This is a behaviour change in v0.8.0-alpha: it replaces an older guard of 256 MiB counted in UTF-16 code units (issue #110), so an input between 64 and 256 MiB that used to parse is now refused until you raise the limit.

• Bytes, not characters. The UTF-8 size is counted, so an e with an acute accent is two, and a leading byte-order mark is three: the count is taken before the mark is stripped and before anything is decoded or parsed.
• Checked first. An oversized input is refused even if it is also malformed. An input of exactly the maximum is accepted.
• Library. readJson, readYaml, readToml, readXml and readOml take maxInputBytes: an integer of at least 1, otherwise a RangeError. There is no "unlimited". readFormat(name, text, opts) applies the same limit before a plugin reader runs.
• CLI. --max-input-bytes N on format, convert, check, validate and infer. The file or stdin read stops at N + 1 bytes, and the refusal says how to raise the limit.

import { readJson } from "@omnist-dev/omnist";

readJson('{"a":"xxxxxxxxxxxx"}', { maxInputBytes: 20 }); // 20 bytes: accepted
readJson('{"a":"xxxxxxxxxxxxx"}', { maxInputBytes: 20 }); // 21 bytes: ParseError,
// code "document.limit.input-size", path "$"

What the limit does and does not bound

The spec gives no reference default (D-24): an implementation may set any finite value, must document it, and should measure its slowest codec on a worst-case input of that size before raising it. A byte cap bounds the cost of a parsing library's worst case; it does not make any parse fast. Measured on this port (WSL2, Node 20, yaml 2.9, smol-toml, fast-xml-parser):

Input	Size	Read time
JSON, array of records	10 MiB / 64 MiB	0.5 s / 2.9 s
OML, repeated edges	10 MiB / 64 MiB	0.7 s / 4.5 s
XML, repeated elements	10 MiB / 64 MiB	0.9 s / 5.4 s
YAML, block sequence of records	2 MiB; 0.14 / 0.28 / 0.58 / 1.2 MiB of short records	0.7 s; 0.7 / 0.8 / 1.1 / 2.3 s (the first includes warm-up; linear)
TOML, array of tables	1 / 2 / 4 / 10 MiB	0.5 / 0.8 / 1.6 / 3.8 s (v0.10.1-alpha; before it 0.3 / 1.0 / 5.0 / 46.7 s)
YAML, one block mapping	91 / 183 / 388 / 1,004 KiB (5,000 / 10,000 / 20,000 / 50,000 keys)	0.3 / 0.3 / 0.5 / 1.2 s (v0.8.1-alpha; before it 0.8 / 2.4 / 10.7 s, and minutes at 50,000)

So 64 MiB is a sound bound for JSON, OML, XML and YAML mappings, and not for one parser:

• YAML (omnist-ts#157): the yaml library's duplicate-key check compares each key with every key before it, so its parse of one large block (or flow) mapping was quadratic in the keys: a 1 MB mapping of 50,000 keys took minutes (195 s on one machine, 74 s for a smaller text on another), far below 64 MiB. Fixed in v0.8.1-alpha: readYaml turns the library's check off and makes the same test in one linear pass, so that mapping now reads in about 1.2 s. A duplicate key is still refused with the same error and position, with one exception: in a mapping of a document whose total quadratic cost (the sum over its mappings of keys squared) exceeds that of one 4,096-key mapping, the error names the duplicate key's own position. The library's position is the end of the preceding token, so it can be the previous line when the previous value is empty (a: then a: 1 after 4,500 keys: library 4501:3, port 4502:1). Re-running the library's quadratic check on such a document would reopen the denial of service, whether it is one huge mapping or many large ones (40 mappings of 4,095 keys with one duplicate took 24 s to re-parse; now 5.4 s, most of it the parse itself). The aliases inside !!pairs and !!omap are a separate, still-open part of that issue.
• TOML (omnist-ts#170, ledger DIV-12): smol-toml looks for the next . after every key with a plain string search, so a key with no . later in the file scans to the end of the text, and a file of many keys or tables was quadratic: 60,000 array tables took 2.0 s, 200,000 took 21.5 s and 10 MiB took 46.7 s. Fixed in v0.10.1-alpha: a text over 65,536 characters is handed to the library with a . in a comment at the end of every line (a comment cannot change a document; readToml maps a syntax error back to its position in your input), so the library finds a . at once. Now 60,000 / 200,000 array tables read in 0.5 / 1.5 s, and 1 / 2 / 4 / 10 MiB in 0.5 / 0.8 / 1.6 / 3.8 s: linear. A text of 65,536 characters or less is not prepared, and its quadratic cost is negligible at that size.
• XML (omnist-ts#169):fast-xml-parser is superlinear in the nesting depth (8,000 deep 1.2 s, 16,000 deep 6.0 s, 32,000 deep 35 s, so minutes at 100,000), and its validator exhausts the call stack on an attribute value of about 5 MB or more. Fixed in v0.10.1-alpha: readXml finds the nesting in one linear pass over the text and refuses a document nested past maxDepth with document.limit.depth before the parser sees it (1,000 / 10,000 / 100,000 deep: 7 / 32 / 112 ms), and an attribute value too long to validate is parse.codec-syntax (1 / 2 / 4 MB are read in 0.3 / 0.5 / 1.0 s, 8 MB is refused in 1.9 s).
• YAML and TOML depth: the yaml library composes by recursion and fails at about 800 levels. readYaml measures the nesting on the library's token stream and refuses a document past twice maxDepth with document.limit.depth (100,000 levels of flow collections: 1.3 s). A maxDepth raised far above the default is bounded by the call stack, as the safety-limits section says.

If you read untrusted YAML, pass a maxInputBytes that matches what you expect to receive (for example 1024 * 1024), and run the parse in a process you can time out; the !!pairs part of #157 is still open.

Safety limits and how to set them

Depth, node count and integer digits (spec D-9) are options on every reader, beside maxInputBytes:

Option	Default	Refusal	Counts
maxDepth	200	document.limit.depth at $	levels of node nesting from the Document root, which is level 1; a chain of N edges from the root to a scalar leaf is N levels
maxNodes	1,000,000	document.limit.nodes at $	the nodes of one Document, the root included, a scalar leaf not
maxIntDigits	4,300	document.limit.int-digits at the literal's Document path	the decimal digits of an integer's value, sign excluded; a hex, octal, binary or underscored literal is counted by the value it denotes (D-28)

Each is a safe integer of at least 1, else a RangeError: a limit is never "none" (D-10). They are passed per call (readJson(text, { maxDepth: 50 }), readFormat("yaml", text, { maxNodes: 10_000 })), and the same options reach the OML reader. A depth well above the default is bounded by the JavaScript call stack. The CLI has no flags for them.

Why 64 MiB

A document within MAX_NODES (1,000,000 nodes) is never refused by it: a 1,000,000-leaf XML document is well under 50 MiB, and the other formats are more compact. It is also the value the Python implementation uses. A smaller default would refuse legitimate large documents and would still not make YAML or TOML safe, which is why the limit is configurable rather than lower.

Writers refuse a string with no UTF-8 encoding

A JavaScript string can hold a UTF-16 lone surrogate, which has no UTF-8 encoding. No writer will write one (spec C-9): writeJson, writeYaml, writeToml, writeXml, writeOml and their check* functions throw WriteError with code write.unsupported-value, at the Document path of the node holding the string (for an edge label, the node holding the edge), whatever strict is. A valid surrogate pair, an astral character, writes normally.