1
0
mirror of https://github.com/nlohmann/json.git synced 2026-07-21 19:23:03 +04:00

Add SWAR bulk fast path to string serialization (dump_escaped)

When ensure_ascii is false, dump_escaped previously ran every byte of
every string and object key through the UTF-8 DFA decoder, even for the
common case of ordinary text with nothing to escape. This mirrors the
per-byte cost the parser had before the contiguous fast paths.

At a character boundary, bulk-copy the longest run of bytes that need no
escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk
validator the lexer's contiguous path uses - and only fall back to the
byte-at-a-time DFA loop for the first byte that needs individual handling
(a quote, backslash, control character, or ill-formed/truncated UTF-8).
Because every "hard" or invalid byte is still processed by the unchanged
byte path, escaping output and error handling (including strict-mode
error 316 position and message) are byte-identical to before.

The ensure_ascii=true path is unchanged: it must escape non-ASCII and
0x7F, which string_bulk_run does not stop on, so a separate predicate
would be needed for it.

Verified byte-for-byte identical dump output against the pre-change
implementation across ~20k randomized byte strings plus curated edge
cases (all escapes, control chars, valid multibyte, surrogates,
overlong, truncated sequences) for both ensure_ascii settings and all
three error handlers, in C++11/17/20 at -O2/-O3.

Throughput (g++ -O3, ensure_ascii=false, vs pre-change):
  long ASCII strings   4.2x
  twitter-like objects 2.3x
  dense CJK            1.4x  (further headroom with JSON_USE_SIMDUTF)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
2026-07-20 21:11:02 +00:00
parent 41833047cc
commit 32c2b317af
2 changed files with 67 additions and 0 deletions
@@ -24,6 +24,7 @@
#include <nlohmann/detail/conversions/to_chars.hpp>
#include <nlohmann/detail/exceptions.hpp>
#include <nlohmann/detail/input/string_scan.hpp>
#include <nlohmann/detail/macro_scope.hpp>
#include <nlohmann/detail/meta/cpp_future.hpp>
#include <nlohmann/detail/output/binary_writer.hpp>
@@ -400,6 +401,38 @@ class serializer
for (std::size_t i = 0; i < s.size(); ++i)
{
// Fast path: when not escaping non-ASCII characters and sitting on a
// character boundary (state == UTF8_ACCEPT), bulk-copy the longest
// run of bytes that need no escaping. string_bulk_run() (shared with
// the lexer's contiguous scanner) stops exactly at the first byte
// that dump_escaped would handle individually - a quote, a backslash,
// a control character (< 0x20), or an ill-formed/truncated UTF-8
// sequence - so that byte is left to the byte-at-a-time path below,
// keeping error handling and diagnostics unchanged.
if (!ensure_ascii && state == UTF8_ACCEPT)
{
const auto* const data = reinterpret_cast<const unsigned char*>(s.data());
const std::size_t run = string_bulk_run(data + i, s.size() - i);
if (run != 0)
{
// emit any bytes still pending in string_buffer first to
// preserve output order, then write the run directly
if (bytes != 0)
{
o->write_characters(string_buffer.data(), bytes);
bytes = 0;
}
o->write_characters(s.data() + i, run);
bytes_after_last_accept = 0;
undumped_chars = 0;
i += run;
if (i >= s.size())
{
break;
}
}
}
const auto byte = static_cast<std::uint8_t>(s[i]);
switch (decode(state, codepoint, byte))
+34
View File
@@ -20548,6 +20548,8 @@ NLOHMANN_JSON_NAMESPACE_END
// #include <nlohmann/detail/exceptions.hpp>
// #include <nlohmann/detail/input/string_scan.hpp>
// #include <nlohmann/detail/macro_scope.hpp>
// #include <nlohmann/detail/meta/cpp_future.hpp>
@@ -20930,6 +20932,38 @@ class serializer
for (std::size_t i = 0; i < s.size(); ++i)
{
// Fast path: when not escaping non-ASCII characters and sitting on a
// character boundary (state == UTF8_ACCEPT), bulk-copy the longest
// run of bytes that need no escaping. string_bulk_run() (shared with
// the lexer's contiguous scanner) stops exactly at the first byte
// that dump_escaped would handle individually - a quote, a backslash,
// a control character (< 0x20), or an ill-formed/truncated UTF-8
// sequence - so that byte is left to the byte-at-a-time path below,
// keeping error handling and diagnostics unchanged.
if (!ensure_ascii && state == UTF8_ACCEPT)
{
const auto* const data = reinterpret_cast<const unsigned char*>(s.data());
const std::size_t run = string_bulk_run(data + i, s.size() - i);
if (run != 0)
{
// emit any bytes still pending in string_buffer first to
// preserve output order, then write the run directly
if (bytes != 0)
{
o->write_characters(string_buffer.data(), bytes);
bytes = 0;
}
o->write_characters(s.data() + i, run);
bytes_after_last_accept = 0;
undumped_chars = 0;
i += run;
if (i >= s.size())
{
break;
}
}
}
const auto byte = static_cast<std::uint8_t>(s[i]);
switch (decode(state, codepoint, byte))