Skip to content

Add options to store files directly to the file system and to omit duplicating files in memory #257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
etr merged 50 commits into from
Mar 3, 2022
Merged

Add options to store files directly to the file system and to omit duplicating files in memory #257

Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
5f22dfd
Add options to store files directly to the file system and to omit du…
ctomahogh Feb 8, 2022
916610a
Delete accidentially added newline in http_resource.hpp
ctomahogh Feb 9, 2022
d9adfcb
Removed unused function, fixed some code style and typos in comments
ctomahogh Feb 10, 2022
220fdd8
Removed asprintf in favor or string concatenation
ctomahogh Feb 10, 2022
52cb06f
Made http_utils::generate_random_upload_filename plattform independant
ctomahogh Feb 10, 2022
71d1f20
Input and output of http_utils::generate_random_upload_filename are c…
ctomahogh Feb 10, 2022
1f37cfa
Added options and functions for file upload to README.md
ctomahogh Feb 10, 2022
fac086a
Use correct include and functions in generate_random_upload_filename …
ctomahogh Feb 10, 2022
5843f5f
Add missing includes for windows, fixed wrong argument for _mktemp_s
ctomahogh Feb 10, 2022
bc5a4fe
Removed accidentially pushed debug code
ctomahogh Feb 11, 2022
8d7ea8e
Use a static const char* for the random file system template
ctomahogh Feb 11, 2022
329c066
Make output of get_or_create_file_info const and introduce new setters
ctomahogh Feb 11, 2022
07e9f4d
Use const variables for the webserver options
ctomahogh Feb 11, 2022
29ba743
Added an example for file upload
ctomahogh Feb 11, 2022
2606d61
Use a prefixed operator in for loop
ctomahogh Feb 11, 2022
8ab2972
Added safety check for use of strlen and some missing free()s
ctomahogh Feb 14, 2022
3880d8d
Changed file map to be a map of keys with a nested map of files
ctomahogh Feb 14, 2022
2e99b1d
Adjusted file_upload example to use new files map
ctomahogh Feb 14, 2022
fd7bab9
Updated description of files map in README.md
ctomahogh Feb 14, 2022
ccc67e8
Removed strlen from generate_random_upload_filename
ctomahogh Feb 14, 2022
799c201
Use a pointer to std::ofstream in modded_request
ctomahogh Feb 15, 2022
f329061
Moved struct file_info_s to class file_info
ctomahogh Feb 16, 2022
8b5c172
Setters of class file_info are private and class webserver is a frien…
ctomahogh Feb 17, 2022
cb802d1
Use const reference as parameter for set_file_system_file_name
ctomahogh Feb 17, 2022
c6151c4
Don't create zero length files if no file is uploaded
ctomahogh Feb 17, 2022
4cb49c8
Updated README.md as file_info is now a class
ctomahogh Feb 17, 2022
7f2c353
Some code styling and small refactorings after review
ctomahogh Feb 18, 2022
27fc9a8
Replaced std::exception with custom generateFilenameException
ctomahogh Feb 18, 2022
1cf683e
Some more comments and code style fixes after review
ctomahogh Feb 18, 2022
88d14aa
Some more code style changes after review
ctomahogh Feb 18, 2022
b3a39dd
Added error string to generateFilenameException
ctomahogh Feb 18, 2022
cbca260
Added comment for c functions in generate_random_upload_filename
ctomahogh Feb 18, 2022
f896540
Removed const qualifier from file_info::get_file_size()
ctomahogh Feb 20, 2022
897eab9
Merge branch 'etr:master' into upload_to_file_system
ctomahogh Feb 21, 2022
7b73c48
Added unit test for generate_random_upload_filename
ctomahogh Feb 21, 2022
4f2cecf
Removed filesystem include from unit tests again, as it would require…
ctomahogh Feb 21, 2022
c1878f5
Close open upload file after post processor is done
ctomahogh Feb 22, 2022
f189014
Added content_type and transfer_encoding to file_info
ctomahogh Feb 22, 2022
8e16406
Fixed file_upload example
ctomahogh Feb 22, 2022
4bcc846
Made filename_template and path_separator part of class http_utils
ctomahogh Feb 22, 2022
f4fb34f
Added integration test for file upload
ctomahogh Feb 25, 2022
0e90742
Fixed integration test for file upload
ctomahogh Feb 25, 2022
ee91ea9
Merge branch 'etr:master' into upload_to_file_system
ctomahogh Feb 26, 2022
ac91c63
Removed empty AUTO_TEST from file_upload integration test
ctomahogh Feb 26, 2022
f13d1fe
Use internal values for http_ressource class in integration tests
ctomahogh Mar 2, 2022
91b90e5
Added further integration tests for file upload
ctomahogh Mar 3, 2022
4de82c9
Added test_content_2 to configure script
ctomahogh Mar 3, 2022
acebdfa
Fixed memory leak in post_iterator
ctomahogh Mar 3, 2022
6fff5f0
Some code style fixed on file upload integration test
ctomahogh Mar 3, 2022
dbba47c
Use map definition instead of auto in integration test
ctomahogh Mar 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use internal values for http_ressource class in integration tests 
Avoid using pointers in class print_file_upload_resource in favour of
internal values and according getter functions.
Additionally simplified the code by wrapping lines and using
the auto keyword instead of long map specifiers.
  • Loading branch information
@ctomahogh
ctomahogh committed Mar 2, 2022
commit f13d1fe0b098809061e4612ae8e18a28a42bb4fc
98 changes: 52 additions & 46 deletions test/integ/file_upload.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,26 +71,30 @@ static CURLcode send_file_to_webserver() {

class print_file_upload_resource : public http_resource {
public:
explicit print_file_upload_resource(std::map <std::string, std::string, httpserver::http::arg_comparator>* args,
std::map<std::string, std::map<std::string, httpserver::http::file_info>>* files,
stringstream* content) {
this->args = args;
this->content = content;
this->files = files;
}

const shared_ptr<http_response> render_POST(const http_request& req) {
(*content) << req.get_content();
(*args) = req.get_args();
(*files) = req.get_files();
content = req.get_content();
args = req.get_args();
files = req.get_files();
shared_ptr<string_response> hresp(new string_response("OK", 201, "text/plain"));
return hresp;
}

const map<string, string, httpserver::http::arg_comparator> get_args() const {
return args;
}

const map<string, map<string, httpserver::http::file_info>> get_files() const {
return files;
}

const string get_content() const {
return content;
}

private:
std::map <std::string, std::string, httpserver::http::arg_comparator>* args;
std::map<std::string, std::map<std::string, httpserver::http::file_info>>* files;
stringstream *content;
map <string, string, httpserver::http::arg_comparator> args;
map<string, map<string, httpserver::http::file_info>> files;
string content;
};

LT_BEGIN_SUITE(file_upload_suite)
Expand All @@ -102,9 +106,6 @@ LT_BEGIN_SUITE(file_upload_suite)
LT_END_SUITE(file_upload_suite)

LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_memory_and_disk)
stringstream content;
std::map<std::string, std::string, httpserver::http::arg_comparator> args;
std::map<std::string, std::map<std::string, httpserver::http::file_info>> files;
string upload_directory = ".";
webserver* ws;

Expand All @@ -116,42 +117,44 @@ LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_memory_and_disk)
ws->start(false);
LT_CHECK_EQ(ws->is_running(), true);

print_file_upload_resource resource(&args, &files, &content);
print_file_upload_resource resource;
ws->register_resource("upload", &resource);

CURLcode res = send_file_to_webserver();
LT_ASSERT_EQ(res, 0);

string actual_content = content.str();
string actual_content = resource.get_content();
LT_CHECK_EQ(actual_content.find(FILENAME_IN_GET_CONTENT) != string::npos, true);
LT_CHECK_EQ(actual_content.find(TEST_CONTENT) != string::npos, true);

auto args = resource.get_args();
Copy link
Copy Markdown
Owner

@etr etr Mar 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: let's avoid auto in this context - it makes it harder to understand the type used (same for all the others in this code).

LT_CHECK_EQ(args.size(), 1);
std::map<std::string, std::string, httpserver::http::arg_comparator>::iterator arg = args.begin();
auto arg = args.begin();
LT_CHECK_EQ(arg->first, TEST_KEY);
LT_CHECK_EQ(arg->second, TEST_CONTENT);

auto files = resource.get_files();
LT_CHECK_EQ(files.size(), 1);
std::map<std::string, std::map<std::string, httpserver::http::file_info>>::iterator file_key = files.begin();
auto file_key = files.begin();
LT_CHECK_EQ(file_key->first, TEST_KEY);
LT_CHECK_EQ(file_key->second.size(), 1);
std::map<std::string, httpserver::http::file_info>::iterator file = file_key->second.begin();
auto file = file_key->second.begin();
LT_CHECK_EQ(file->first, TEST_CONTENT_FILENAME);
LT_CHECK_EQ(file->second.get_file_size(), TEST_CONTENT_SIZE);
LT_CHECK_EQ(file->second.get_content_type(), httpserver::http::http_utils::application_octet_stream);

string expected_filename = upload_directory + httpserver::http::http_utils::path_separator + httpserver::http::http_utils::upload_filename_template;
LT_CHECK_EQ(file->second.get_file_system_file_name().substr(0, file->second.get_file_system_file_name().size() - 6), expected_filename.substr(0, expected_filename.size() - 6));
string expected_filename = upload_directory +
httpserver::http::http_utils::path_separator +
httpserver::http::http_utils::upload_filename_template;
LT_CHECK_EQ(file->second.get_file_system_file_name().substr(0, file->second.get_file_system_file_name().size() - 6),
expected_filename.substr(0, expected_filename.size() - 6));
unlink(file->second.get_file_system_file_name().c_str());

ws->stop();
delete ws;
LT_END_AUTO_TEST(file_upload_memory_and_disk)

LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_disk_only)
stringstream content;
std::map<std::string, std::string, httpserver::http::arg_comparator> args;
std::map<std::string, std::map<std::string, httpserver::http::file_info>> files;
string upload_directory = ".";
webserver* ws;

Expand All @@ -163,38 +166,40 @@ LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_disk_only)
ws->start(false);
LT_CHECK_EQ(ws->is_running(), true);

print_file_upload_resource resource(&args, &files, &content);
print_file_upload_resource resource;
ws->register_resource("upload", &resource);

CURLcode res = send_file_to_webserver();
LT_ASSERT_EQ(res, 0);

string actual_content = content.str();
string actual_content = resource.get_content();
LT_CHECK_EQ(actual_content.size(), 0);

auto args = resource.get_args();
LT_CHECK_EQ(args.size(), 0);

auto files = resource.get_files();
LT_CHECK_EQ(files.size(), 1);
std::map<std::string, std::map<std::string, httpserver::http::file_info>>::iterator file_key = files.begin();
auto file_key = files.begin();
LT_CHECK_EQ(file_key->first, TEST_KEY);
LT_CHECK_EQ(file_key->second.size(), 1);
std::map<std::string, httpserver::http::file_info>::iterator file = file_key->second.begin();
auto file = file_key->second.begin();
LT_CHECK_EQ(file->first, TEST_CONTENT_FILENAME);
LT_CHECK_EQ(file->second.get_file_size(), TEST_CONTENT_SIZE);
LT_CHECK_EQ(file->second.get_content_type(), httpserver::http::http_utils::application_octet_stream);

string expected_filename = upload_directory + httpserver::http::http_utils::path_separator + httpserver::http::http_utils::upload_filename_template;
LT_CHECK_EQ(file->second.get_file_system_file_name().substr(0, file->second.get_file_system_file_name().size() - 6), expected_filename.substr(0, expected_filename.size() - 6));
string expected_filename = upload_directory +
httpserver::http::http_utils::path_separator +
httpserver::http::http_utils::upload_filename_template;
LT_CHECK_EQ(file->second.get_file_system_file_name().substr(0, file->second.get_file_system_file_name().size() - 6),
expected_filename.substr(0, expected_filename.size() - 6));
unlink(file->second.get_file_system_file_name().c_str());

ws->stop();
delete ws;
LT_END_AUTO_TEST(file_upload_disk_only)

LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_memory_only_incl_content)
stringstream content;
std::map<std::string, std::string, httpserver::http::arg_comparator> args;
std::map<std::string, std::map<std::string, httpserver::http::file_info>> files;
string upload_directory = ".";
webserver* ws;

Expand All @@ -204,31 +209,30 @@ LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_memory_only_incl_content)
ws->start(false);
LT_CHECK_EQ(ws->is_running(), true);

print_file_upload_resource resource(&args, &files, &content);
print_file_upload_resource resource;
ws->register_resource("upload", &resource);

CURLcode res = send_file_to_webserver();
LT_ASSERT_EQ(res, 0);

string actual_content = content.str();
string actual_content = resource.get_content();
LT_CHECK_EQ(actual_content.find(FILENAME_IN_GET_CONTENT) != string::npos, true);
LT_CHECK_EQ(actual_content.find(TEST_CONTENT) != string::npos, true);

auto args = resource.get_args();
LT_CHECK_EQ(args.size(), 1);
std::map<std::string, std::string, httpserver::http::arg_comparator>::iterator arg = args.begin();
auto arg = args.begin();
LT_CHECK_EQ(arg->first, TEST_KEY);
LT_CHECK_EQ(arg->second, TEST_CONTENT);

LT_CHECK_EQ(files.size(), 0);
auto files = resource.get_files();
LT_CHECK_EQ(resource.get_files().size(), 0);

ws->stop();
delete ws;
LT_END_AUTO_TEST(file_upload_memory_only_incl_content)

LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_memory_only_excl_content)
stringstream content;
std::map<std::string, std::string, httpserver::http::arg_comparator> args;
std::map<std::string, std::map<std::string, httpserver::http::file_info>> files;
string upload_directory = ".";
webserver* ws;

Expand All @@ -238,20 +242,22 @@ LT_BEGIN_AUTO_TEST(file_upload_suite, file_upload_memory_only_excl_content)
ws->start(false);
LT_CHECK_EQ(ws->is_running(), true);

print_file_upload_resource resource(&args, &files, &content);
print_file_upload_resource resource;
ws->register_resource("upload", &resource);

CURLcode res = send_file_to_webserver();
LT_ASSERT_EQ(res, 0);

string actual_content = content.str();
string actual_content = resource.get_content();
LT_CHECK_EQ(actual_content.size(), 0);

auto args = resource.get_args();
LT_CHECK_EQ(args.size(), 1);
std::map<std::string, std::string, httpserver::http::arg_comparator>::iterator arg = args.begin();
auto arg = args.begin();
LT_CHECK_EQ(arg->first, TEST_KEY);
LT_CHECK_EQ(arg->second, TEST_CONTENT);

auto files = resource.get_files();
LT_CHECK_EQ(files.size(), 0);

ws->stop();
Expand Down