Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
7f545a8
Normalize the words "position" and "end"
jkeiser Mar 20, 2021
a334303
Don't step off the end of the index buffer
jkeiser Mar 21, 2021
02762fa
Split ondemand_object_tests into smaller files
jkeiser Mar 21, 2021
09601eb
Add __SIMDJSON_CHECK_EOF feature flag
jkeiser Mar 22, 2021
1fbd604
This adds "guarded" (i.e., memory-safe) number parsing routines.
lemire Mar 23, 2021
081f954
Adding potentially useless definition.
lemire Mar 23, 2021
42c8541
Adding the functions that I meant to add earlier.
lemire Mar 23, 2021
4d0f102
Merge pull request #1516 from simdjson/dlemire/guarded_number_parsing
jkeiser Mar 31, 2021
d7b08c8
Merge branch 'master' into jkeiser/no-padding
lemire Apr 23, 2021
76f7e4a
Merge branch 'master' into jkeiser/no-padding
lemire May 8, 2021
595c628
Merge branch 'master' into jkeiser/no-padding
lemire May 27, 2021
dadfb51
Minor correction.
lemire May 27, 2021
930b0be
Missing #ifdef SIMDJSON_DEVELOPMENT_CHECKS
lemire May 27, 2021
c0fb5f7
Another merge issue.
lemire May 28, 2021
0c14707
Merge branch 'master' into jkeiser/no-padding
lemire May 28, 2021
0d8f6f5
Merging again. Fixing issue.
lemire May 28, 2021
10a7b9b
Merge branch 'master' into jkeiser/no-padding
lemire Jun 4, 2021
633784a
Merge issue
jkeiser Jun 6, 2021
eda0f19
Don't allocate when parsing root number
jkeiser Jun 6, 2021
da4de2e
Don't read padding at all for root booleans and numbers
jkeiser Jun 6, 2021
b02aedb
Don't use padding parsing booleans
jkeiser Jun 6, 2021
ba0406f
Minor changes to compilation test.
lemire Jun 7, 2021
f45c59f
I suspect that sanitizers might increase stack usage, to combat that …
lemire Jun 8, 2021
bab51e5
Remove unlikely from null / bool parsing
jkeiser Jun 19, 2021
6ba9ad3
Remove need for padding from SIMD string parsing
jkeiser Jun 19, 2021
45fbd8c
Make len available to any dom_parser_implementation user
jkeiser Jun 21, 2021
ac647e6
Don't change the order of fields to avoid perf changes
jkeiser Jun 21, 2021
42a31ad
Fix ARM compile errors
jkeiser Jun 23, 2021
6a1f8fc
Don't step off the end of the index buffer
jkeiser Mar 21, 2021
78d1cd9
Split ondemand_object_tests into smaller files
jkeiser Mar 21, 2021
9040989
Add __SIMDJSON_CHECK_EOF feature flag
jkeiser Mar 22, 2021
5a6894c
This adds "guarded" (i.e., memory-safe) number parsing routines.
lemire Mar 23, 2021
40f522a
Adding potentially useless definition.
lemire Mar 23, 2021
b3b108b
Adding the functions that I meant to add earlier.
lemire Mar 23, 2021
e5d0e92
Minor correction.
lemire May 27, 2021
4b118bd
Verifying issue.
lemire Jul 18, 2021
267e70d
Adding another case.
lemire Jul 18, 2021
3300d84
Adding type test.
lemire Jul 19, 2021
3c5c2d2
This backports (isolate) a fix by jkeiser of issue 1632.
lemire Jul 19, 2021
f364e35
Minor fixes.
lemire Jul 19, 2021
e8e731a
Merge branch 'master' into dlemire/backport_jkeiser_fix_for_issue1632
lemire Jul 19, 2021
a349969
Adding a duplicate of the existing test.
lemire Jul 19, 2021
8b2bff9
Messy temporary work.
lemire Jul 19, 2021
df24da3
Cleaning.
lemire Jul 19, 2021
4501aeb
Disabling a couple of asserts.
lemire Jul 19, 2021
2e688a2
Removing two other asserts.
lemire Jul 19, 2021
9cace04
Adding more tests.
lemire Jul 19, 2021
d0e2a7f
Minor fixes
lemire Jul 19, 2021
c3f7eff
Adding some commentary
lemire Jul 19, 2021
7dcd43e
Merge branch 'jkeiser/no-padding' into dlemire/nopadding
lemire Jul 21, 2021
bbeec66
Merge branch 'jkeiser/no-padding-scalar' into dlemire/nopadding
lemire Jul 21, 2021
081c3ca
Merged.
lemire Jul 22, 2021
91b8fa4
Fixing a CI issue.
lemire Jul 22, 2021
ca9391c
Let us run more CI tests.
lemire Jul 22, 2021
e4bba2d
Do not needlessly do perf checks.
lemire Jul 22, 2021
24068a0
Merge branch 'master' into dlemire/backport_jkeiser_fix_for_issue1632
lemire Jul 22, 2021
57ca29e
Merge branch 'dlemire/backport_jkeiser_fix_for_issue1632' into dlemir…
lemire Jul 22, 2021
f2d419d
Basic guarding.
lemire Jul 22, 2021
1fe50cb
Trimming.
lemire Jul 22, 2021
0e8b686
Merge branch 'dlemire/backport_jkeiser_fix_for_issue1632' into dlemir…
lemire Jul 22, 2021
d7770e9
Disabling 'json[N-1]' test.
lemire Jul 22, 2021
bb0ed22
Removing some of the padding in the tests.
lemire Jul 22, 2021
70a0d62
Merge branch 'dlemire/backport_jkeiser_fix_for_issue1632' into dlemir…
lemire Jul 22, 2021
b1a3943
We should now be able to iterate over a string_view.
lemire Jul 22, 2021
910686f
Merge branch 'dlemire/nopadding' of github.com:simdjson/simdjson into…
lemire Jul 22, 2021
dc86daa
Removing the padding.
lemire Jul 22, 2021
21845f1
Documenting iterate_many + removing remnants of padding in On Demand.
lemire Jul 23, 2021
1a433fc
Enabling a broad range of tests.
lemire Jul 23, 2021
b12014f
Let us try some setups in debug mode (for faster builds).
lemire Jul 23, 2021
8ce1e52
Merge branch 'master' into dlemire/nopadding
lemire Jul 23, 2021
f6fc786
Disabling some change made to the DOM kernel which was evidently made…
lemire Jul 23, 2021
7175731
Removing the special-case
lemire Jul 23, 2021
757db1b
Removing padding from DOM API.
lemire Jul 23, 2021
d72cba2
Minor tweaks.
lemire Jul 23, 2021
a7ff410
Various fixes for the number parsing in the DOM front-end. Basically,…
lemire Jul 23, 2021
13055a4
Using an alternative instrinsic.
lemire Jul 23, 2021
dc66d61
Fixing comment.
lemire Jul 23, 2021
68df233
Removing a couple of tests from circle ci.
lemire Jul 23, 2021
bbc9bd0
Some tweaking.
lemire Jul 24, 2021
d8b6a8c
Merge branch 'dlemire/nopadding' of github.com:simdjson/simdjson into…
lemire Jul 24, 2021
2ceca3e
More tweaking to fix problems exposed by the fuzzers.
lemire Jul 24, 2021
60b6a16
Let us stress the memory overflow in more tests.
lemire Jul 24, 2021
657489e
More fixes.
lemire Jul 24, 2021
7c8bc65
Turning on developer mode explicitly throughout.
lemire Jul 24, 2021
ce11f41
Merge branch 'dlemire/nopadding' of github.com:simdjson/simdjson into…
lemire Jul 24, 2021
b6aff44
Syncing the singleheader files.
lemire Jul 24, 2021
2907912
Guarding the peek/advance.
lemire Jul 24, 2021
450cb7b
Merge branch 'dlemire/nopadding' of github.com:simdjson/simdjson into…
lemire Jul 24, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
I suspect that sanitizers might increase stack usage, to combat that …
…you may set SIMDJSON_NO_FORCE_INLINING (#1614)
  • Loading branch information
lemire authored Jun 8, 2021
commit f45c59f27ca0fba2ddaea324a69d7a720e1f70e5
12 changes: 6 additions & 6 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,22 +191,22 @@ jobs:
sanitize-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake sanitize build
executor: gcc10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
sanitize-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake sanitize build
executor: clang10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
threadsanitize-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake sanitize build
executor: gcc10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE_THREADS=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE_THREADS=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
threadsanitize-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake sanitize build
executor: clang10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE_THREADS=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE_THREADS=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
nocheckeof-clang10:
description: Validate that when __SIMDJSON_CHECK_EOF=0, everything still succeeds
Expand Down Expand Up @@ -270,12 +270,12 @@ jobs:
sanitize-haswell-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake sanitize build
executor: gcc10
environment: { CXXFLAGS: -march=haswell, CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
environment: { CXXFLAGS: -march=haswell, CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
sanitize-haswell-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake sanitize build
executor: clang10
environment: { CXXFLAGS: -march=haswell, CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
environment: { CXXFLAGS: -march=haswell, CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]

workflows:
Expand Down
4 changes: 2 additions & 2 deletions .drone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ steps:
environment:
CC: clang-9
CXX: clang++-9
CMAKE_FLAGS: -DSIMDJSON_SANITIZE=ON -DSIMDJSON_IMPLEMENTATION=haswell;westmere;fallback
CMAKE_FLAGS: -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON -DSIMDJSON_IMPLEMENTATION=haswell;westmere;fallback
BUILD_FLAGS: -- -j
CTEST_FLAGS: -j4 --output-on-failure -LE explicitonly
commands:
Expand Down Expand Up @@ -329,7 +329,7 @@ steps:
environment:
CC: clang-6.0
CXX: clang++-6.0
CMAKE_FLAGS: -DSIMDJSON_SANITIZE=ON -DSIMDJSON_IMPLEMENTATION=arm64;fallback
CMAKE_FLAGS: -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON -DSIMDJSON_IMPLEMENTATION=arm64;fallback
BUILD_FLAGS: -- -j
CTEST_FLAGS: -j4 --output-on-failure -LE explicitonly
commands:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ubuntu18-threadsani.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_SANITIZE_THREADS=ON .. &&
cmake -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE_THREADS=ON .. &&
cmake --build . --target document_stream_tests --target parse_many_test &&
ctest --output-on-failure -R parse_many_test &&
ctest --output-on-failure -R document_stream_tests
2 changes: 1 addition & 1 deletion .github/workflows/ubuntu20-threadsani.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_SANITIZE_THREADS=ON .. &&
cmake -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE_THREADS=ON .. &&
cmake --build . --target document_stream_tests --target parse_many_test &&
ctest --output-on-failure -R parse_many_test &&
ctest --output-on-failure -R document_stream_tests
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ install:
- export CMAKE_C_FLAGS="${CMAKE_CXX_FLAGS}"
- export CMAKE_FLAGS="-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DSIMDJSON_IMPLEMENTATION=ppc64;fallback";
- if [[ "${SANITIZE}" == "on" ]]; then
export CMAKE_FLAGS="${CMAKE_FLAGS} -DSIMDJSON_SANITIZE=ON";
export CMAKE_FLAGS="${CMAKE_FLAGS} -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON";
export ASAN_OPTIONS="detect_leaks=0";
fi
- if [[ "${STATIC}" == "on" ]]; then
Expand Down
5 changes: 5 additions & 0 deletions cmake/developer-options.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
#
add_library(simdjson-internal-flags INTERFACE)

option(SIMDJSON_NO_FORCE_INLINING "Do not attempt to force function inlining" OFF)
if(SIMDJSON_NO_FORCE_INLINING)
add_compile_definitions(SIMDJSON_NO_FORCE_INLINING=1)
endif()

option(SIMDJSON_SANITIZE_UNDEFINED "Sanitize undefined behavior" OFF)
if(SIMDJSON_SANITIZE_UNDEFINED)
add_compile_options(-fsanitize=undefined -fno-sanitize-recover=all)
Expand Down
12 changes: 10 additions & 2 deletions include/simdjson/common_defs.h
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,12 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
#define SIMDJSON_ISALIGNED_N(ptr, n) (((uintptr_t)(ptr) & ((n)-1)) == 0)

#if defined(SIMDJSON_REGULAR_VISUAL_STUDIO)

#if SIMDJSON_NO_FORCE_INLINING
// forcing inlining can increase stack usage.
#define simdjson_really_inline inline
#else
#define simdjson_really_inline __forceinline
#endif
#define simdjson_never_inline __declspec(noinline)

#define simdjson_unused
Expand Down Expand Up @@ -106,8 +110,12 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
#define SIMDJSON_POP_DISABLE_WARNINGS __pragma(warning( pop ))

#else // SIMDJSON_REGULAR_VISUAL_STUDIO

#if SIMDJSON_NO_FORCE_INLINING
// forcing inlining can increase stack usage.
#define simdjson_really_inline inline
#else
#define simdjson_really_inline inline __attribute__((always_inline))
#endif
#define simdjson_never_inline inline __attribute__((noinline))

#define simdjson_unused __attribute__((unused))
Expand Down