Replies: 4 comments 4 replies
-
|
Here is an example for POSIX systems. padded_memory_map map(myfile);
simdjson::padded_string_view view = map.view();I will be including this in a future release on simdjson. (We will not be supporting Windows which has a non-standard approach to memory file mapping.) Code: #include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
struct padded_memory_map {
padded_memory_map(const char *filename) {
int fd = open(filename, O_RDONLY);
if (fd == -1) {
return; // file not found or cannot be opened, data will be nullptr
}
struct stat st;
if (fstat(fd, &st) == -1) {
close(fd);
return; // failed to get file size, data will be nullptr
}
size = (size_t)st.st_size;
size_t total_size = size + simdjson::SIMDJSON_PADDING;
void *anon_map =
mmap(NULL, total_size, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (anon_map == MAP_FAILED) {
close(fd);
return; // failed to create anonymous mapping, data will be nullptr
}
void *file_map =
mmap(anon_map, size, PROT_READ, MAP_SHARED | MAP_FIXED, fd, 0);
if (file_map == MAP_FAILED) {
munmap(anon_map, total_size);
close(fd);
return; // failed to mmap file, data will be nullptr
}
data = (const char *)file_map;
close(fd); // no longer needed after mapping
}
~padded_memory_map() { munmap((void *)data, size + simdjson::SIMDJSON_PADDING); }
// lifetime of the view is tied to the memory map, so we can return a view
// directly
simdjson::padded_string_view view() const {
if(!is_valid()) {
return simdjson::padded_string_view(); // return an empty view if mapping failed
}
return simdjson::padded_string_view(data, size, size + simdjson::SIMDJSON_PADDING);
}
bool is_valid() const { return data != nullptr; }
private:
const char *data{nullptr};
size_t size{0};
}; |
Beta Was this translation helpful? Give feedback.
-
Does it mean simdjson is not recommend to be used on windows with file mapped text? |
Beta Was this translation helpful? Give feedback.
-
|
May be I did not express correctly in the 1st message. Will the simdjson read beyond pages (and crash) if json is well formed and find ends on the very end of the page? |
Beta Was this translation helpful? Give feedback.
-
|
This is the way to correctly open fileMapping + padding on windows:
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello, thanks for amazing product, I have following question
If I use file mapping with page size = 4k (0x1000)
Can I use simdjson for such mapped files ?
Will simdjson read beyond the last page ?
If for example I have file with size 0x4FF1, the file mapping for this file will allocate 0x5000 bytes.
After that simdjson will report that I don't have sufficient padding for the file, what should I need to do ?
Beta Was this translation helpful? Give feedback.
All reactions