Skip to content

Commit 7a2fda8

Browse files
lemirelemire
authored
Merge branch 'master' into jkeiser/cmake-fuzz-noexceptions
2 parents db314bc + e50c239 commit 7a2fda8

69 files changed

Lines changed: 886 additions & 597 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.appveyor.yml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,19 @@ environment:
1212
matrix:
1313
- SIMDJSON_BUILD_STATIC: ON
1414
SIMDJSON_ENABLE_THREADS: OFF
15+
SIMDJSON_PLATFORM: x64
16+
- SIMDJSON_BUILD_STATIC: ON
17+
SIMDJSON_ENABLE_THREADS: OFF
18+
SIMDJSON_PLATFORM: Win32
1519
- SIMDJSON_BUILD_STATIC: OFF
1620
SIMDJSON_ENABLE_THREADS: ON
21+
SIMDJSON_PLATFORM: x64
1722

1823
build_script:
1924
- set
2025
- mkdir build
2126
- cd build
22-
- cmake -DSIMDJSON_BUILD_STATIC=%SIMDJSON_BUILD_STATIC% -DSIMDJSON_ENABLE_THREADS=%SIMDJSON_ENABLE_THREADS% -DCMAKE_BUILD_TYPE=%Configuration% -DCMAKE_GENERATOR_PLATFORM=x64 ..
27+
- cmake -DSIMDJSON_BUILD_STATIC=%SIMDJSON_BUILD_STATIC% -DSIMDJSON_ENABLE_THREADS=%SIMDJSON_ENABLE_THREADS% -DCMAKE_BUILD_TYPE=%Configuration% -DCMAKE_GENERATOR_PLATFORM=%SIMDJSON_PLATFORM% ..
2328
- cmake -LH ..
2429
- cmake --build . --config %Configuration% --verbose
2530

@@ -29,7 +34,7 @@ test_script:
2934
matrix:
3035
fast_finish: true
3136
exclude:
32-
# Don't build all variants on 2019, just running it to make sure readme_tests succeed
33-
- image: Visual Studio 2019
37+
# Don't build all variants on 2017, just running it to make sure readme_tests succeed
38+
- image: Visual Studio 2017
3439
SIMDJSON_BUILD_STATIC: ON
3540
SIMDJSON_ENABLE_THREADS: OFF

‎.circleci/config.yml‎

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ executors:
2525
CXX: g++
2626
BUILD_FLAGS: -j
2727
CTEST_FLAGS: -j4 --output-on-failure
28+
clang9:
29+
docker:
30+
- image: conanio/clang9
31+
environment:
32+
CXX: clang++-9
33+
BUILD_FLAGS: -j
34+
CTEST_FLAGS: -j4 --output-on-failure
2835

2936
clang6:
3037
docker:
@@ -50,10 +57,13 @@ commands:
5057
- run: make test
5158
- run: make checkperf
5259

53-
cmake_prep:
60+
install_cmake:
5461
steps:
5562
- run: apt-get update -qq
5663
- run: apt-get install -y cmake
64+
65+
cmake_prep:
66+
steps:
5767
- checkout
5868

5969
cmake_build:
@@ -78,6 +88,7 @@ commands:
7888
- run: SIMDJSON_FORCE_IMPLEMENTATION=fallback ctest $CTEST_FLAGS -L per_implementation
7989
- run: ctest $CTEST_FLAGS -LE "acceptance|per_implementation" # Everything we haven't run yet, run now.
8090

91+
8192
# we not only want cmake to build and run tests, but we want also a successful installation from which we can build, link and run programs
8293
cmake_install_test: # this version builds, install, test and then verify from the installation
8394
steps:
@@ -91,60 +102,65 @@ jobs:
91102
description: Build and run tests on GCC 7 and AVX 2 with a cmake static build
92103
executor: gcc7
93104
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=ON }
94-
steps: [ cmake_test_all, cmake_install_test ]
105+
steps: [ install_cmake, cmake_test_all, cmake_install_test ]
95106
clang6:
96107
description: Build and run tests on clang 6 and AVX 2 with a cmake static build
97108
executor: clang6
98109
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=ON }
99-
steps: [ init_clang6, cmake_test_all, cmake_install_test ]
100-
110+
steps: [ init_clang6, install_cmake, cmake_test_all, cmake_install_test ]
111+
# libcpp
112+
libcpp-clang9:
113+
description: Build and run tests on clang 6 and AVX 2 with a cmake static build and libc++
114+
executor: clang9
115+
environment: { CMAKE_FLAGS: -DSIMDJSON_USE_LIBCPP=ON }
116+
steps: [ cmake_test_all, cmake_install_test ]
101117
# sanitize
102118
sanitize-gcc9:
103119
description: Build and run tests on GCC 9 and AVX 2 with a cmake sanitize build
104120
executor: gcc9
105121
environment: { CMAKE_FLAGS: -DSIMDJSON_BUILD_STATIC=OFF -DSIMDJSON_SANITIZE=ON, BUILD_FLAGS: "", CTEST_FLAGS: -j4 --output-on-failure -E checkperf }
106-
steps: [ cmake_test_all ]
122+
steps: [ install_cmake, cmake_test_all ]
107123
sanitize-clang6:
108124
description: Build and run tests on clang 6 and AVX 2 with a cmake sanitize build
109125
executor: clang6
110126
environment: { CMAKE_FLAGS: -DSIMDJSON_BUILD_STATIC=OFF -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: -j4 --output-on-failure -E checkperf }
111-
steps: [ init_clang6, cmake_test_all ]
127+
steps: [ init_clang6, install_cmake, cmake_test_all ]
112128

113129
# dynamic
114130
dynamic-gcc7:
115131
description: Build and run tests on GCC 7 and AVX 2 with a cmake dynamic build
116132
executor: gcc7
117133
environment: { CMAKE_FLAGS: -DSIMDJSON_BUILD_STATIC=OFF }
118-
steps: [ cmake_test, cmake_install_test ]
134+
steps: [ install_cmake, cmake_test, cmake_install_test ]
119135
dynamic-clang6:
120136
description: Build and run tests on clang 6 and AVX 2 with a cmake dynamic build
121137
executor: clang6
122138
environment: { CMAKE_FLAGS: -DSIMDJSON_BUILD_STATIC=OFF }
123-
steps: [ init_clang6, cmake_test, cmake_install_test ]
139+
steps: [ init_clang6, install_cmake, cmake_test, cmake_install_test ]
124140

125141
# unthreaded
126142
unthreaded-gcc7:
127143
description: Build and run tests on GCC 7 and AVX 2 *without* threads
128144
executor: gcc7
129145
environment: { CMAKE_FLAGS: -DSIMDJSON_ENABLE_THREADS=OFF }
130-
steps: [ cmake_test, cmake_install_test ]
146+
steps: [ install_cmake, cmake_test, cmake_install_test ]
131147
unthreaded-clang6:
132148
description: Build and run tests on Clang 6 and AVX 2 *without* threads
133149
executor: clang6
134150
environment: { CMAKE_FLAGS: -DSIMDJSON_ENABLE_THREADS=OFF }
135-
steps: [ init_clang6, cmake_test, cmake_install_test ]
151+
steps: [ init_clang6, install_cmake, cmake_test, cmake_install_test ]
136152

137153
# noexcept
138154
noexcept-gcc7:
139155
description: Build and run tests on GCC 7 and AVX 2 with exceptions off
140156
executor: gcc7
141157
environment: { CMAKE_FLAGS: -DSIMDJSON_EXCEPTIONS=OFF }
142-
steps: [ cmake_test, cmake_install_test ]
158+
steps: [ install_cmake, cmake_test, cmake_install_test ]
143159
noexcept-clang6:
144160
description: Build and run tests on GCC 7 and AVX 2 with exceptions off
145161
executor: clang6
146162
environment: { CMAKE_FLAGS: -DSIMDJSON_EXCEPTIONS=OFF }
147-
steps: [ init_clang6, cmake_test, cmake_install_test ]
163+
steps: [ init_clang6, install_cmake, cmake_test, cmake_install_test ]
148164

149165
#
150166
# Misc.
@@ -174,7 +190,10 @@ workflows:
174190
# full multi-implementation tests
175191
- gcc7
176192
- clang6
177-
193+
194+
# libc++
195+
- libcpp-clang9
196+
178197
# full single-implementation tests
179198
- sanitize-gcc9
180199
- sanitize-clang6

‎.cirrus.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
task:
2+
timeout_in: 120m
3+
freebsd_instance:
4+
matrix:
5+
- image_family: freebsd-13-0-snap
6+
7+
env:
8+
ASSUME_ALWAYS_YES: YES
9+
setup_script:
10+
- pkg update -f
11+
- pkg install bash
12+
- pkg install cmake
13+
- pkg install git
14+
build_script:
15+
- mkdir build
16+
- cd build
17+
- cmake ..
18+
- make -j2
19+
test_script:
20+
- cd build
21+
- ctest -j2 --output-on-failure -E checkperf

‎.dockerignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
*
22
!.git
33
!Makefile
4-
!amalgamation.sh
4+
!amalgamate.sh
55
!benchmark
66
!dependencies
77
!include

‎.drone.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ platform:
4747
steps:
4848
- name: build
4949
image: gcc:8
50-
commands: [ make, make amalgamate ]
50+
commands: [ make, make amalgamate_test ]
5151
---
5252
kind: pipeline
5353
name: x64-slowtests
@@ -87,7 +87,7 @@ steps:
8787
image: gcc:8
8888
environment:
8989
EXTRA_FLAGS: -fno-exceptions
90-
commands: [ make, make amalgamate ]
90+
commands: [ make, make amalgamate_test ]
9191
---
9292
kind: pipeline
9393
name: x64-noexceptions-slowtests
@@ -139,7 +139,7 @@ platform:
139139
steps:
140140
- name: build
141141
image: gcc:8
142-
commands: [ make, make amalgamate ]
142+
commands: [ make, make amalgamate_test ]
143143
---
144144
kind: pipeline
145145
name: arm64-slowtests

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,11 @@ objs
133133
/tests/allparserscheckfile
134134
/tests/basictests
135135
/tests/checkimplementation
136+
/tests/compilation_failure_tests/dangling_parser_load_should_compile
137+
/tests/compilation_failure_tests/dangling_parser_parse_padstring_should_compile
138+
/tests/compilation_failure_tests/dangling_parser_parse_stdstring_should_compile
139+
/tests/compilation_failure_tests/dangling_parser_parse_uchar_should_compile
140+
/tests/compilation_failure_tests/dangling_parser_parse_uint8_should_compile
136141
/tests/compilation_failure_tests/example_compiletest_should_compile
137142
/tests/errortests
138143
/tests/extracting_values_example
@@ -213,3 +218,7 @@ _deps
213218
# We check in a custom version of root Makefile that is not generated by CMake
214219
!/Makefile
215220

221+
singleheader/amalgamation_demo
222+
singleheader/amalgamation_demo.cpp
223+
singleheader/simdjson.cpp
224+
singleheader/simdjson.h

‎CMakeLists.txt‎

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,13 @@ if(MSVC)
2525
else()
2626
option(SIMDJSON_BUILD_STATIC "Build a static library" OFF) # turning it on disables the production of a dynamic library
2727
option(SIMDJSON_COMPETITION "Compile competitive benchmarks" ON)
28+
option(SIMDJSON_USE_LIBCPP "Use the libc++ library" OFF)
2829
endif()
2930
option(SIMDJSON_GOOGLE_BENCHMARKS "compile the Google Benchmark benchmarks" OFF)
3031

3132
set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/tools/cmake")
3233

34+
3335
# We compile tools, tests, etc. with C++ 17. Override yourself if you need on a target.
3436
set(CMAKE_CXX_STANDARD 17)
3537
set(CMAKE_CXX_STANDARD_REQUIRED ON)
@@ -38,6 +40,7 @@ set(CMAKE_MACOSX_RPATH OFF)
3840
set(CMAKE_THREAD_PREFER_PTHREAD ON)
3941
set(THREADS_PREFER_PTHREAD_FLAG ON)
4042

43+
4144
# LTO seems to create all sorts of fun problems. Let us
4245
# disable temporarily.
4346
#include(CheckIPOSupported)
@@ -52,12 +55,23 @@ set(THREADS_PREFER_PTHREAD_FLAG ON)
5255
add_library(simdjson-flags INTERFACE)
5356
if(MSVC)
5457
target_compile_options(simdjson-flags INTERFACE /nologo /D_CRT_SECURE_NO_WARNINGS)
55-
target_compile_options(simdjson-flags INTERFACE /WX /W3 /wd4005 /wd4996 /wd4267 /wd4244 /wd4113)
58+
target_compile_options(simdjson-flags INTERFACE /WX /W3)
5659
else()
5760
target_compile_options(simdjson-flags INTERFACE -fPIC)
58-
target_compile_options(simdjson-flags INTERFACE -Werror -Wall -Wextra -Wsign-compare -Wshadow -Wwrite-strings -Wpointer-arith -Winit-self)
61+
target_compile_options(simdjson-flags INTERFACE -Werror -Wall -Wextra -Wsign-compare -Wshadow -Wwrite-strings -Wpointer-arith -Winit-self -Wconversion -Wno-sign-conversion)
5962
endif()
6063

64+
65+
if(SIMDJSON_USE_LIBCPP)
66+
target_link_libraries(simdjson-flags INTERFACE -stdlib=libc++ -lc++abi)
67+
# instead of the above line, we could have used
68+
# set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -stdlib=libc++ -lc++abi")
69+
# The next line is needed empirically.
70+
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -stdlib=libc++")
71+
# we update CMAKE_SHARED_LINKER_FLAGS, this gets updated later as well
72+
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -lc++abi")
73+
endif(SIMDJSON_USE_LIBCPP)
74+
6175
# Optional flags
6276
option(SIMDJSON_IMPLEMENTATION_HASWELL "Include the haswell implementation" ON)
6377
if(NOT SIMDJSON_IMPLEMENTATION_HASWELL)
@@ -138,6 +152,15 @@ if (NOT MSVC)
138152
add_custom_target(simdjson-user-cmakecache ALL DEPENDS ${SIMDJSON_USER_CMAKECACHE})
139153
endif()
140154

155+
#
156+
# Set up test data
157+
#
158+
enable_testing()
159+
add_subdirectory(jsonchecker)
160+
add_subdirectory(jsonexamples)
161+
add_library(test-data INTERFACE)
162+
target_link_libraries(test-data INTERFACE jsonchecker-data jsonexamples-data)
163+
141164
#
142165
# Create the top level simdjson library (must be done at this level to use both src/ and include/
143166
# directories) and tools
@@ -146,16 +169,11 @@ add_subdirectory(include)
146169
add_subdirectory(src)
147170
add_subdirectory(windows)
148171
add_subdirectory(tools)
172+
add_subdirectory(singleheader)
149173

150174
#
151175
# Compile tools / tests / benchmarks
152176
#
153-
enable_testing()
154-
155-
add_library(test-data INTERFACE)
156-
target_compile_definitions(test-data INTERFACE SIMDJSON_TEST_DATA_DIR="${CMAKE_CURRENT_SOURCE_DIR}/jsonchecker/")
157-
target_compile_definitions(test-data INTERFACE SIMDJSON_BENCHMARK_DATA_DIR="${CMAKE_CURRENT_SOURCE_DIR}/jsonexamples/")
158-
set(EXAMPLE_JSON ${CMAKE_CURRENT_SOURCE_DIR}/jsonexamples/twitter.json)
159177

160178
add_subdirectory(dependencies)
161179
add_subdirectory(tests)

‎CONTRIBUTORS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,5 +31,6 @@ Paul Dreik
3131
Jeremie Piotte
3232
Matthew Wilson
3333
Dušan Jovanović
34+
Matjaž Ostroveršnik
3435
# if you have contributed to the project and your name does not
3536
# appear in this list, please let us know!

‎HACKING.md‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ simdjson's source structure, from the top level, looks like this:
2020
implementations).
2121
* simdjson.cpp: A "master source" that includes all implementation files from src/. This is
2222
equivalent to the distributed simdjson.cpp.
23-
* arm64/|fallback/|haswell/|westmere/: Architecture-specific implementations. All functions are
23+
* arm64/|fallback/|haswell/|westmere/: Architecture-specific implementations. All functions are
2424
Each architecture defines its own namespace, e.g. simdjson::haswell.
2525
* generic/: Generic implementations of the simdjson parser. These files may be included and
2626
compiled multiple times, from whichever architectures use them. They assume they are already
@@ -37,7 +37,7 @@ Other important files and directories:
3737
* **.drone.yml:** Definitions for Drone CI.
3838
* **.appveyor.yml:** Definitions for Appveyor CI (Windows).
3939
* **.circleci:** Definitions for Circle CI.
40-
* **amalgamation.sh:** Generates singleheader/simdjson.h and singleheader/simdjson.cpp for release.
40+
* **amalgamate.sh:** Generates singleheader/simdjson.h and singleheader/simdjson.cpp for release.
4141
* **benchmark:** This is where we do benchmarking. Benchmarking is core to every change we make; the
4242
cardinal rule is don't regress performance without knowing exactly why, and what you're trading
4343
for it. If you're not sure what else to do to check your performance, this is always a good start:
@@ -56,7 +56,7 @@ Other important files and directories:
5656
* **tools:** Source for executables that can be distributed with simdjson
5757

5858
> **Don't modify the files in singleheader/ directly; these are automatically generated.**
59-
>
59+
>
6060
> While we distribute those files on release, we *maintain* the files under include/ and src/.
6161
6262
While simdjson distributes just two files from the singleheader/ directory, we *maintain* the code in
@@ -73,7 +73,7 @@ you can regenerate them by running this at the top level:
7373
make amalgamate
7474
```
7575
76-
The amalgamator is at `amalgamation.sh` at the top level. It generates singleheader/simdjson.h by
76+
The amalgamator is at `amalgamate.sh` at the top level. It generates singleheader/simdjson.h by
7777
reading through include/simdjson.h, copy/pasting each header file into the amalgamated file at the
7878
point it gets included (but only once per header). singleheader/simdjson.cpp is generated from
7979
src/simdjson.cpp the same way, except files under generic/ may be included and copy/pasted multiple
@@ -157,6 +157,19 @@ make
157157
make test
158158
```
159159
160+
linux way:
161+
162+
```
163+
mkdir build # if necessary
164+
cd build
165+
cmake .. -DCMAKE_CXX_COMPILER=g++ # or
166+
cmake .. -DCMAKE_CXX_COMPILER=clang++
167+
make -j
168+
```
169+
170+
171+
172+
160173
### Usage (CMake on 64-bit Windows using Visual Studio)
161174
162175
We assume you have a common 64-bit Windows PC with at least Visual Studio 2017 and an x64 processor with AVX2 support (2013 Intel Haswell or later) or SSE 4.2 + CLMUL (2010 Westmere or later).
@@ -189,7 +202,7 @@ On Windows (64-bit):
189202
will build and install `simdjson` as a shared library.
190203
191204
```
192-
.\vcpkg.exe install simdjson:x64-windows-static
205+
.\vcpkg.exe install simdjson:x64-windows-static
193206
```
194207
195208
will build and install `simdjson` as a static library.

0 commit comments

Comments
 (0)