Skip to content

Commit 62ff207

Browse files
syoyoCopilotclaude
authored
Add optimized parser with multithreading, SIMD, and custom allocator support (squashed take-over of tinyobjloader#424) (tinyobjloader#428)
* Add optimized parser with multithreading, SIMD, and custom allocator support Squashed take-over of tinyobjloader#424 (branch copilot/optimize-parser-for-huge-meshes) so CI runs on a maintainer-pushed branch. Highlights: - New optimized OBJ loader (LoadObjOpt / LoadObjOptTyped) targeting huge meshes, with fast_float-based ASCII float/double parsing. - Optional compile-time features: TINYOBJLOADER_USE_MULTITHREADING (threaded parse + merge), TINYOBJLOADER_USE_SIMD (SSE2/AVX2/NEON newline scanning), and TINYOBJLOADER_ENABLE_EXCEPTION (opt-in C++ exceptions). - ArenaAllocator / arena_adapter for bulk allocation and a TypedArray API. - Experimental stream-based OBJ loader under experimental/stream/. - Fuzzing harnesses (tests/obj-fuzz, tests/llvm-fuzz, tests/fuzz_common.h) and extensive parity tests against the legacy loader (tests/opt/loadobjopt_multithread.inc, tester.cc). Original work by Copilot coding agent; reviewed and validated locally (make check passes under clang++ with -std=c++11 -fsanitize=address). Co-Authored-By: Copilot <198982749+Copilot@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Harden optimized/stream parser per Copilot review Address the three review comments from Copilot on the take-over PR, plus a related robustness fix uncovered while doing so. - experimental/stream: fix `v x y z w r g b` (7-component) vertex parsing. The stream loader previously treated the weight `w` as red and shifted the colors. It now mirrors LoadObjOpt, keyed on the component count beyond xyz: +1 -> weight, +3 -> color (weight = r, legacy compat), +4 -> weight + color. - tests/tester.cc: include the canonical experimental/stream/stream_obj_loader.h instead of re-declaring StreamLoadConfig / LoadObjStreamExperimental, so the test cannot drift from the real API. - tests/tester.cc: make test_arena_adapter_overflow_guard conditional on TINYOBJLOADER_ENABLE_EXCEPTION (allocate() throws std::bad_alloc when exceptions are enabled, returns nullptr otherwise) so the test no longer terminates when built with exceptions. - tiny_obj_loader.h: guard the IMPLEMENTATION block against double inclusion within a single translation unit (TINYOBJLOADER_IMPLEMENTATION_DEFINED). Including the stream header after defining TINYOBJLOADER_IMPLEMENTATION now pulls in tiny_obj_loader.h a second time; without the guard this caused redefinition errors. - Add test_stream_loader_weighted_color_vertex_7_components regression test. Verified: `make check` passes under clang++ and g++, both with and without -DTINYOBJLOADER_ENABLE_EXCEPTION. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document 7-component vertex (v x y z w r g b) as a non-standard extension Web research against the official Wavefront spec, Paul Bourke's vertex-color reference, and Wikipedia confirms there is no standard 7-component `v` line: the format defines only xyz / xyzw / xyzrgb, and weight and color are mutually exclusive. Per maintainer decision, keep the weight + color interpretation (4th value = weight, next three = RGB) in LoadObjOpt and the experimental stream loader, but document clearly that it is a tinyobjloader-specific extension and that the classic LoadObj / LoadObjWithCallback path instead caps at 6 components, so the loaders intentionally diverge for 7+ token vertices. Comment-only change; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Make no-SIMD/no-threading/no-exception the tested default; polish dead code Ensure the library's default configuration (no SIMD, no multithreading, no exceptions) is the baseline that CI exercises, and clean up warnings/dead code introduced by the optimized parser. Default configuration: - tester.cc no longer force-defines TINYOBJLOADER_USE_MULTITHREADING, so the default `make check` build now runs with the library defaults (scalar, single-threaded, exception-free). The header already had no auto-#define of any feature macro, so consumers were already defaulting to all-off; this just makes the test build match. - tests/Makefile builds two binaries: `tester` (defaults) and `tester_features` (TINYOBJLOADER_USE_MULTITHREADING + _USE_SIMD + ENABLE_EXCEPTION). `make check` runs both, so CI covers the default and feature-enabled code paths. New `check_default` / `check_features` targets run a single config. - Document the opt-in macros (all disabled by default) in the header and tests/README. Polish / hardening: - Remove dead function opt_requires_legacy_fallback (a never-called duplicate of opt_line_start_is_legacy). - opt_tryParseDouble: drop the dead inner fast_float branch (unreachable in every configuration) and guard the whole function with TINYOBJLOADER_DISABLE_FAST_FLOAT, since it is only called from that fallback path. Removes two -Wunused-function warnings in the default build. - Guard the now-unused `cache` parameter in opt_parseLineToThreadData under TINYOBJLOADER_DISABLE_FAST_FLOAT (-Wunused-parameter). Verified: default, features, and TINYOBJLOADER_DISABLE_FAST_FLOAT builds all compile (clang++ and g++, C++11 and C++17) and pass the full suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add TINYOBJLOADER_DISABLE_FAST_FLOAT config to the test matrix `make check` now builds and runs a third configuration, `check_nofastfloat`, which defines TINYOBJLOADER_DISABLE_FAST_FLOAT to exercise the hand-written fallback float parser (otherwise library defaults). CI invokes `make check`, so this fallback path — previously only verified by hand — is now covered automatically alongside the default and feature-enabled builds. Verified: all three configs (default, features, nofastfloat) build and pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * README: document the optimized LoadObjOpt loader Replace the stale "Optimized loader" section (which referenced an old experimental loader and v0.9/v1.0 benchmark numbers) with a short usage guide for the in-header LoadObjOpt API: a compilable example, the opt-in multithreading/SIMD macros (off by default), and a note on the arena-backed LoadObjOptTyped variant and the experimental stream loader. Also add a "What's new" entry. Example verified to compile (-Wall -Wextra) and run against models/cornell_box.obj. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 1fe01b9 commit 62ff207

19 files changed

Lines changed: 11234 additions & 35 deletions

‎README.md‎

Lines changed: 38 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Old version is available as `v0.9.x` branch https://github.com/syoyo/tinyobjload
1818

1919
## What's new
2020

21+
* 22 May, 2026 : Added an optimized in-header loader `LoadObjOpt` with optional multithreading/SIMD (see [Optimized loader](#optimized-loader)).
2122
* 29 Jul, 2021 : Added Mapbox's earcut for robust triangulation. Also fixes triangulation bug(still there is some issue in built-in triangulation algorithm: https://github.com/tinyobjloader/tinyobjloader/issues/319).
2223
* 19 Feb, 2020 : The repository has been moved to https://github.com/tinyobjloader/tinyobjloader !
2324
* 18 May, 2019 : Python binding!(See `python` folder. Also see https://pypi.org/project/tinyobjloader/)
@@ -352,16 +353,45 @@ for (size_t s = 0; s < shapes.size(); s++) {
352353

353354
## Optimized loader
354355

355-
Optimized multi-threaded .obj loader is available at `experimental/` directory.
356-
If you want absolute performance to load .obj data, this optimized loader will fit your purpose.
357-
Note that the optimized loader uses C++11 thread and it does less error checks but may work most .obj data.
356+
For large `.obj` files, tinyobjloader ships an optimized in-header loader,
357+
`LoadObjOpt` (C++11 required). It parses the whole buffer in one pass and can
358+
optionally use multiple threads and SIMD line scanning. It does fewer error
359+
checks than the standard loader but handles most real-world `.obj` data.
358360

359-
Here is some benchmark result. Time are measured on MacBook 12(Early 2016, Core m5 1.2GHz).
361+
The result reuses the standard `attrib` / `shapes.mesh.*` layout, so the
362+
iteration code shown above works unchanged.
360363

361-
* Rungholt scene(6M triangles)
362-
* old version(v0.9.x): 15500 msecs.
363-
* baseline(v1.0.x): 6800 msecs(2.3x faster than old version)
364-
* optimised: 1500 msecs(10x faster than old version, 4.5x faster than baseline)
364+
```c++
365+
#define TINYOBJLOADER_IMPLEMENTATION
366+
// Optional speed-ups (all OFF by default; require C++11):
367+
//#define TINYOBJLOADER_USE_MULTITHREADING // multi-threaded parsing
368+
//#define TINYOBJLOADER_USE_SIMD // SIMD (SSE2/AVX2/NEON) newline scan
369+
#include "tiny_obj_loader.h"
370+
371+
tinyobj::basic_attrib_t<> attrib;
372+
std::vector<tinyobj::basic_shape_t<>> shapes;
373+
std::vector<tinyobj::material_t> materials;
374+
std::string warn, err;
375+
376+
tinyobj::OptLoadConfig config;
377+
config.triangulate = true;
378+
config.num_threads = -1; // -1 = hardware_concurrency, 0/1 = single-threaded.
379+
// Effective only with TINYOBJLOADER_USE_MULTITHREADING.
380+
381+
bool ok = tinyobj::LoadObjOpt(&attrib, &shapes, &materials, &warn, &err,
382+
"large_scene.obj", /* mtl_basedir */ nullptr,
383+
config);
384+
if (!warn.empty()) std::cout << warn;
385+
if (!ok) { std::cerr << err; return -1; }
386+
// attrib.vertices/.normals/.texcoords and shapes[s].mesh.* match the standard
387+
// loader — iterate exactly as in the example above.
388+
```
389+
390+
A `LoadObjOptTyped` variant is also available; it returns an `OptResult` whose
391+
arrays are backed by a single arena allocator and only allocates optional
392+
arrays (vertex weights, texcoord `w`, colors, ...) when the input contains
393+
them — handy when minimizing allocations matters. An experimental stream-based
394+
loader lives under `experimental/stream/`.
365395
366396
## Python binding
367397

‎experimental/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Experimental code for .obj loader.
22

33
* Multi-threaded optimized parser : tinyobj_loader_opt.h
4+
* Streaming experimental parser : stream/stream_obj_loader.h
45

56
## Requirements
67

‎experimental/stream/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Experimental Stream OBJ Parser
2+
3+
This directory contains an experimental line-by-line OBJ parser intended to
4+
reduce peak input buffering compared with the whole-buffer optimized parser.
5+
6+
The design is split into two layers:
7+
8+
- `StreamHandler`: callback-style incremental parser interface.
9+
- `LoadObjStreamExperimental(...)`: convenience wrapper that builds
10+
`tinyobj::attrib_t`, `tinyobj::shape_t`, and `tinyobj::material_t`.
11+
- Ordered multithreaded chunk mode: read bounded batches of lines, parse those
12+
chunks in parallel, then replay parsed events in original order.
13+
14+
## Goals
15+
16+
- Parse OBJ from `std::istream` without reading the whole file into one buffer.
17+
- Preserve support for relative face indices.
18+
- Allow applications to consume faces incrementally without materializing a
19+
full mesh.
20+
21+
## Current Scope
22+
23+
Implemented records:
24+
25+
- `v`
26+
- `vn`
27+
- `vt`
28+
- `f`
29+
- `g`
30+
- `o`
31+
- `usemtl`
32+
- `mtllib`
33+
- `s`
34+
35+
Ignored for now:
36+
37+
- `l`
38+
- `p`
39+
- free-form curves/surfaces
40+
- tags and skinning extensions
41+
- advanced vertex color fallback behavior matching the legacy loader exactly
42+
43+
## Notes
44+
45+
This parser is intentionally separate from `LoadObjOpt`.
46+
`LoadObjOpt` is built around random-access whole-buffer processing and
47+
multithreaded partitioning, while this module is focused on low intermediate
48+
buffer usage and incremental consumption.
49+
50+
The current multithreaded design uses a bounded in-flight chunk window rather
51+
than a true LRU chunk cache. That matches OBJ's mostly sequential access
52+
pattern better and keeps ordering/state management simple.

0 commit comments

Comments
 (0)