mirror of
https://github.com/nlohmann/json.git
synced 2026-07-21 19:23:03 +04:00
Niels Lohmann
190f4b6a7b
The bulk string scanner validates UTF-8 straight from a contiguous buffer. The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8 validation (easy to get subtly wrong - a from-scratch SSE attempt rejected valid CJK), wire in the vetted simdutf library behind an opt-in switch. simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes <simdutf.h> and routes the bulk validator through simdutf::validate_utf8; the project supplies and links simdutf. Undefined (the default), nothing external is included and the portable C++11 scalar path is used, so the library stays header-only and its baseline behavior is unchanged. Design keeps behavior identical either way: - scan_string_bulk() now finds the run up to the next quote/escape/control byte (non-ASCII allowed) and validates it in one shot; on the rare validation failure it recomputes the exact valid prefix with the scalar helper, so ill-formed input still falls through to the byte path and is reported at the same position with the same message. - the per-sequence scalar path is factored into scalar_string_bulk_run() and is the default backend; the refactor is behavior-preserving and does not change scalar throughput. Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse identically across 2,000,000 arbitrary-byte documents and 1,000,000 mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte path); lexer/parser/diagnostic-position/deserialization suites pass under both configurations (20,188 assertions with the backend enabled); warning-clean on g++ and clang, C++11 and C++20, both configurations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>