Skip to content

Isolated jkeiser fix for issue 1632: make it so that INCORRECT_TYPE is a recoverable condition in On Demand #1663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lemire merged 18 commits into from
Jul 23, 2021
Merged

Isolated jkeiser fix for issue 1632: make it so that INCORRECT_TYPE is a recoverable condition in On Demand #1663

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4b118bd
Verifying issue.
lemire Jul 18, 2021
267e70d
Adding another case.
lemire Jul 18, 2021
3300d84
Adding type test.
lemire Jul 19, 2021
3c5c2d2
This backports (isolate) a fix by jkeiser of issue 1632.
lemire Jul 19, 2021
f364e35
Minor fixes.
lemire Jul 19, 2021
e8e731a
Merge branch 'master' into dlemire/backport_jkeiser_fix_for_issue1632
lemire Jul 19, 2021
a349969
Adding a duplicate of the existing test.
lemire Jul 19, 2021
8b2bff9
Messy temporary work.
lemire Jul 19, 2021
df24da3
Cleaning.
lemire Jul 19, 2021
4501aeb
Disabling a couple of asserts.
lemire Jul 19, 2021
2e688a2
Removing two other asserts.
lemire Jul 19, 2021
9cace04
Adding more tests.
lemire Jul 19, 2021
d0e2a7f
Minor fixes
lemire Jul 19, 2021
c3f7eff
Adding some commentary
lemire Jul 19, 2021
24068a0
Merge branch 'master' into dlemire/backport_jkeiser_fix_for_issue1632
lemire Jul 22, 2021
f2d419d
Basic guarding.
lemire Jul 22, 2021
1fe50cb
Trimming.
lemire Jul 22, 2021
d7770e9
Disabling 'json[N-1]' test.
lemire Jul 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into dlemire/backport_jkeiser_fix_for_issue1632
  • Loading branch information
@lemire
lemire committed Jul 22, 2021
commit 24068a089e0cdb5abd77cca92bbfed9dadf2d502
15 changes: 8 additions & 7 deletions include/simdjson/generic/ondemand/json_iterator-inl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,12 +82,12 @@ simdjson_warn_unused simdjson_really_inline error_code json_iterator::skip_child
// ] or } means we just finished a value and need to jump out of the array/object
case ']': case '}':
logger::log_end_value(*this, "skip");
_depth--;
if (depth() <= parent_depth) { return SUCCESS; }
#if __SIMDJSON_CHECK_EOF
// If there are no more tokens, the parent is incomplete.
if (at_end()) { return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "Missing [ or { at start"); }
#endif // __SIMDJSON_CHECK_EOF
_depth--;
if (depth() <= parent_depth) { return SUCCESS; }
break;
/*case '"':
if(*peek() == ':') {
Expand Down Expand Up @@ -143,16 +143,17 @@ simdjson_really_inline bool json_iterator::at_root() const noexcept {
return position() == root_position();
}

simdjson_really_inline bool json_iterator::streaming() const noexcept {
return _streaming;
}

simdjson_really_inline token_position json_iterator::root_position() const noexcept {
return parser->implementation->structural_indexes.get();
return _root;
}

simdjson_really_inline void json_iterator::assert_at_root() const noexcept {
SIMDJSON_ASSUME( _depth == 1 );
// Visual Studio Clang treats unique_ptr.get() as "side effecting."
#ifndef SIMDJSON_CLANG_VISUAL_STUDIO
SIMDJSON_ASSUME( token._position == parser->implementation->structural_indexes.get() );
#endif
SIMDJSON_ASSUME( token.position() == _root );
}

simdjson_really_inline void json_iterator::assert_more_tokens(uint32_t required_tokens) const noexcept {
Expand Down
4 changes: 1 addition & 3 deletions include/simdjson/generic/ondemand/token_iterator.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,6 @@ class token_iterator {
*
* @param position The position of the token.
*
* TODO consider a string_view, assuming the length will get stripped out by the optimizer when
* it isn't used ...
*/
simdjson_really_inline const uint8_t *peek(token_position position) const noexcept;
/**
Expand All @@ -72,7 +70,7 @@ class token_iterator {
simdjson_really_inline uint32_t peek_length(token_position position) const noexcept;

/**
* Save the current index to be restored later.
* Return the current index.
*/
simdjson_really_inline token_position position() const noexcept;
/**
Expand Down
12 changes: 10 additions & 2 deletions include/simdjson/generic/ondemand/value_iterator-inl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,11 @@ simdjson_warn_unused simdjson_really_inline simdjson_result<bool> value_iterator
}

simdjson_warn_unused simdjson_really_inline simdjson_result<bool> value_iterator::started_root_object() noexcept {
if (*_json_iter->peek_last() != '}') {
// When in streaming mode, we cannot expect peek_last() to be the last structural element of the
// current document. It only works in the normal mode where we have indexed a single document.
// Note that adding a check for 'streaming' is not expensive since we only have at most
// one root element.
if (! _json_iter->streaming() && (*_json_iter->peek_last() != '}')) {
return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing } at end");
}
return started_object();
Expand Down Expand Up @@ -396,7 +400,11 @@ simdjson_warn_unused simdjson_really_inline simdjson_result<bool> value_iterator
}

simdjson_warn_unused simdjson_really_inline simdjson_result<bool> value_iterator::started_root_array() noexcept {
if (*_json_iter->peek_last() != ']') {
// When in streaming mode, we cannot expect peek_last() to be the last structural element of the
// current document. It only works in the normal mode where we have indexed a single document.
// Note that adding a check for 'streaming' is not expensive since we only have at most
// one root element.
if ( ! _json_iter->streaming() && (*_json_iter->peek_last() != ']')) {
return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing ] at end");
}
return started_array();
Expand Down
38 changes: 19 additions & 19 deletions tests/ondemand/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,25 @@
link_libraries(simdjson)
include_directories(..)
add_subdirectory(compilation_failure_tests)

add_cpp_test(ondemand_active_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_array_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_array_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_compilation_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_key_string_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_misc_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_number_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_object_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_object_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_object_find_field_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_object_index_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_ordering_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_parse_api_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_readme_examples LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_scalar_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_twitter_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_wrong_type_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_tostring_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_active_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_array_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_array_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_compilation_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_document_stream_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_json_pointer_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_key_string_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_misc_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_number_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_object_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_object_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_ordering_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_parse_api_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_readme_examples LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_scalar_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_twitter_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_wrong_type_error_tests LABELS ondemand acceptance per_implementation)

if(HAVE_POSIX_FORK AND HAVE_POSIX_WAIT) # assert tests use fork and wait, which aren't on MSVC
add_cpp_test(ondemand_assert_out_of_order_values LABELS assert per_implementation explicitonly ondemand)
Expand Down
2 changes: 1 addition & 1 deletion tests/ondemand/ondemand_array_error_tests.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ namespace array_error_tests {
bool array_iterate_incomplete_error() {
TEST_START();
ONDEMAND_SUBTEST("unclosed after array", R"([ [1] )", assert_iterate(doc.get_array().at(0), { int64_t(1) }, { INCOMPLETE_ARRAY_OR_OBJECT }));
ONDEMAND_SUBTEST("unclosed after array", R"([ [1,])", assert_iterate(doc.get_array().at(0), { int64_t(1) }, { INCORRECT_TYPE, INCOMPLETE_ARRAY_OR_OBJECT }));
ONDEMAND_SUBTEST("unclosed after array", R"([ [1,])", assert_iterate(doc.get_array().at(0), { int64_t(1) }, { INCORRECT_TYPE, TAPE_ERROR }));
ONDEMAND_SUBTEST("unclosed after array", R"([ [1])", assert_iterate(doc.get_array().at(0), { int64_t(1) }, { INCOMPLETE_ARRAY_OR_OBJECT }));
ONDEMAND_SUBTEST("unclosed after array", R"([ [])", assert_iterate(doc.get_array().at(0), { INCOMPLETE_ARRAY_OR_OBJECT }));
TEST_SUCCEED();
Expand Down
2 changes: 1 addition & 1 deletion tests/ondemand/ondemand_json_pointer_tests.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ namespace json_pointer_tests {
ASSERT_ERROR(doc.at_pointer(json_pointer).get(val), simdjson::STRING_ERROR);
std::cout << "\t- unclosed_object" << std::endl;
ASSERT_SUCCESS(parser.iterate(unclosed_object).get(doc));
ASSERT_ERROR(doc.at_pointer(json_pointer).get(val), simdjson::TAPE_ERROR);
ASSERT_ERROR(doc.at_pointer(json_pointer).get(val), simdjson::INCOMPLETE_ARRAY_OR_OBJECT);
std::cout << "\t- missing_bracket_before" << std::endl;
ASSERT_SUCCESS(parser.iterate(missing_bracket_before).get(doc));
ASSERT_ERROR(doc.at_pointer(json_pointer).get(val), simdjson::TAPE_ERROR);
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.