Skip to content

Adding parser for a set of data objects in consecutive memory pages (follow up #585) - #629

Merged
sawenzel merged 2 commits into
AliceO2Group:devfrom
matthiasrichter:dev-algorithm
Oct 23, 2017
Merged

Adding parser for a set of data objects in consecutive memory pages (follow up #585)#629
sawenzel merged 2 commits into
AliceO2Group:devfrom
matthiasrichter:dev-algorithm

Conversation

@matthiasrichter

Copy link
Copy Markdown
Collaborator

This is the basis for a generic raw reader of data split in consecutive DMA pages.

Memory pages have a fixed size and start with a page header. The parser
provides transparent iteration through data objects and handles page
boundaries.

Added in the update:

  • write functionality for parser iterator
  • bidirectional copy function
  • runtime check for the buffer const'ness
  • copy assignment and copy contructor for iterator
  • supporting iterator and const_iterator

@matthiasrichter

Copy link
Copy Markdown
Collaborator Author

Could not reopen #585 because the branch was updated with forced push

@alibuild

alibuild commented Oct 18, 2017

Copy link
Copy Markdown
Collaborator

Error while checking build/o2checkcode/o2 for 58f62f4efc38388d4ff7b8e24a7b9c020fa04031:

sw/BUILD/O2-latest/log
[ 10%] Built target Headers
[ 10%] Building CXX object CCDB/CMakeFiles/CCDB.dir/request.pb.cc.o
In file included from /mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.cc:5:0:
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
 #error This file was generated by an older version of protoc which is
  ^~~~~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
 #error incompatible with your Protocol Buffer headers.  Please
  ^~~~~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
 #error regenerate this file with a newer version of protoc.
  ^~~~~
In file included from /mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.cc:5:0:
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::clear_command()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:188:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ != &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_command(const string&)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:198:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_command(const char*)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:205:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_command(const char*, size_t)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:212:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'std::__cxx11::string* messaging::RequestMessage::mutable_command()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:219:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'std::__cxx11::string* messaging::RequestMessage::release_command()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:226:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:230:45: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
     command_ = const_cast< ::std::string*>(&::google::protobuf::internal::kEmptyString);
                                             ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_allocated_command(std::__cxx11::string*)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:235:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ != &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:243:45: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
     command_ = const_cast< ::std::string*>(&::google::protobuf::internal::kEmptyString);
                                             ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::clear_datasource()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/ca4d3389c9b13f737cf575a10c10472df378d576/O2/CCDB/request.pb.h:258:23: error: 'kEmptyString' is not a member of 'google::protobuf::internal'

Full log here.

@alibuild

alibuild commented Oct 18, 2017

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/o2 for 58f62f4efc38388d4ff7b8e24a7b9c020fa04031:

sw/BUILD/O2-latest/log
[INFO   ] ---User path for Configuration (g4Config.C) is used: /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/dev/0/Detectors/gconfig
[INFO   ] ---User path for Cuts and Processes (SetCuts.C) is used: /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/dev/0/Detectors/gconfig
Error in <TROOT::LoadMacro>: macro /g4libs.C not found in path .:/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/ROOT/v6-10-06+git_c54db1c10b-1/macros
input_line_60:2:3: error: use of undeclared identifier 'g4libs'
 (g4libs())
  ^

Full log here.

@alibuild

alibuild commented Oct 18, 2017

Copy link
Copy Markdown
Collaborator

Error while checking build/o2/macos for 58f62f4efc38388d4ff7b8e24a7b9c020fa04031:


Full log here.

@alibuild

alibuild commented Oct 18, 2017

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/o2-dev-fairroot for 58f62f4efc38388d4ff7b8e24a7b9c020fa04031:

sw/BUILD/O2-latest/log
Scanning dependencies of target CCDB
[  7%] Building CXX object CCDB/CMakeFiles/CCDB.dir/src/Backend.cxx.o
In file included from /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/0_O2_DEV_FAIRROOT/0/CCDB/src/Backend.cxx:16:0:
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
 #error This file was generated by an older version of protoc which is
  ^~~~~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
 #error incompatible with your Protocol Buffer headers.  Please
  ^~~~~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
 #error regenerate this file with a newer version of protoc.
  ^~~~~
In file included from /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/0_O2_DEV_FAIRROOT/0/CCDB/src/Backend.cxx:16:0:
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::clear_command()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:188:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ != &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_command(const string&)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:198:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_command(const char*)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:205:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_command(const char*, size_t)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:212:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'std::__cxx11::string* messaging::RequestMessage::mutable_command()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:219:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'std::__cxx11::string* messaging::RequestMessage::release_command()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:226:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ == &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:230:45: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
     command_ = const_cast< ::std::string*>(&::google::protobuf::internal::kEmptyString);
                                             ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::set_allocated_command(std::__cxx11::string*)':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:235:20: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
   if (command_ != &::google::protobuf::internal::kEmptyString) {
                    ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:243:45: error: 'kEmptyString' is not a member of 'google::protobuf::internal'
     command_ = const_cast< ::std::string*>(&::google::protobuf::internal::kEmptyString);
                                             ^~
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h: In member function 'void messaging::RequestMessage::clear_datasource()':
/mnt/mesos/sandbox/sandbox/sw/BUILD/2245aaa1bec318f0b8514031411a48895b4ec81d/O2/CCDB/request.pb.h:258:23: error: 'kEmptyString' is not a member of 'google::protobuf::internal'

Full log here.

@alibuild

alibuild commented Oct 19, 2017

Copy link
Copy Markdown
Collaborator

Error while checking build/o2/macos for e75735fe4fb8bdb29a12745f8264c79cb5448205:

sw/BUILD/O2-latest/log
[ 29%] Built target runQCViewerDevice
[ 30%] Built target Publishers
In file included from /build/macos_ci/build_o2_macos/sw/SOURCES/O2/629/0/Algorithm/test/pageparser.cxx:24:
/build/macos_ci/build_o2_macos/sw/SOURCES/O2/629/0/Algorithm/test/../include/Algorithm/PageParser.h:124:15: error: no viable overloaded '='
      mParent = rhs.mParent;
      ~~~~~~~ ^ ~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:1975:21: note: in instantiation of member function 'o2::algorithm::PageParser<PageHeader, 128, ClusterData, int>::Iterator<const ClusterData>::operator=' requested here

Full log here.

@alibuild

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/o2 for e75735fe4fb8bdb29a12745f8264c79cb5448205:

sw/BUILD/O2-latest/log
[INFO   ] ---User path for Configuration (g4Config.C) is used: /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/dev/0/Detectors/gconfig
[INFO   ] ---User path for Cuts and Processes (SetCuts.C) is used: /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/dev/0/Detectors/gconfig
Error in <TROOT::LoadMacro>: macro /g4libs.C not found in path .:/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/ROOT/v6-10-06+git_c54db1c10b-1/macros
input_line_60:2:3: error: use of undeclared identifier 'g4libs'
 (g4libs())
  ^

Full log here.

@matthiasrichter

Copy link
Copy Markdown
Collaborator Author

the CI now fails for both build/O2/o2 and build/o2/macos with

The following tests FAILED:
	 94 - tpcsim_G4 (SEGFAULT)
	 95 - tpcsim_G3 (SEGFAULT)
	121 - mcloggertest (SEGFAULT)

@ktf

ktf commented Oct 19, 2017

Copy link
Copy Markdown
Member

I think the issue on mac is a genuine problem. Can you check?

Memory pages have a fixed size and start with a page header. The parser
provides transparent iteration through data objects and handles page
boundaries.
- bidirectional copy function
- runtype check for the buffer const'ness
- supporting iterator and const_iterator
- fixing the assignment operator problem on xcode, now internally using
  pointer instead of reference. Using reference was resulting in making
  and assigning a copy.
@alibuild

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/o2-dev-fairroot for bc280d7:

sw/BUILD/FairRoot-latest/log
+ unset SIMPATH
+ case $ARCHITECTURE in
+ SONAME=so
+ [[ -n /mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/boost/v1.64.0-alice1-1 ]]
+ BOOST_NO_SYSTEM_PATHS=ON
+ cmake /mnt/mesos/sandbox/sandbox/sw/SOURCES/FairRoot/dev/0 -DMACOSX_RPATH=OFF '-DCMAKE_CXX_FLAGS=-fPIC -O2 -std=c++14' -DCMAKE_BUILD_TYPE=RELWITHDEBINFO -DROOTSYS=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/ROOT/v6-10-06+git_c54db1c10b-1 -DROOT_CONFIG_SEARCHPATH=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/ROOT/v6-10-06+git_c54db1c10b-1/bin -DNANOMSG_DIR=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/nanomsg/v1.0.0+git_c52f1bedca-1 -DPythia6_LIBRARY_DIR=/lib -DGeant3_DIR= -DDISABLE_GO=ON -DBUILD_EXAMPLES=OFF -DFAIRROOT_MODULAR_BUILD=ON -DDDS_PATH=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/DDS/1.7-1 -DZEROMQ_ROOT=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/ZeroMQ/v4.1.5-2 -DBOOST_ROOT=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/boost/v1.64.0-alice1-1 -DBOOST_INCLUDEDIR=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/boost/v1.64.0-alice1-1/include -DBOOST_LIBRARYDIR=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/boost/v1.64.0-alice1-1/lib -DBoost_NO_SYSTEM_PATHS=ON -DGSL_DIR=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/GSL/v1.16-2 -DGTEST_ROOT=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/googletest/1.8.0-2 -DProtoBuf_DIR=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/protobuf/v3.0.2-2 -DCMAKE_INSTALL_PREFIX=/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/FairRoot/dev-1
CMake Error at CMakeLists.txt:16 (CMAKE_MINIMUM_REQUIRED):
  CMake 3.9.4 or higher is required.  You are running version 3.8.2


-- Configuring incomplete, errors occurred!

Full log here.

@alibuild

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/o2 for bc280d7:

sw/BUILD/O2-latest/log
[INFO   ] ---User path for Configuration (g4Config.C) is used: /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/dev/0/Detectors/gconfig
[INFO   ] ---User path for Cuts and Processes (SetCuts.C) is used: /mnt/mesos/sandbox/sandbox/sw/SOURCES/O2/dev/0/Detectors/gconfig
Error in <TROOT::LoadMacro>: macro /g4libs.C not found in path .:/mnt/mesos/sandbox/sandbox/sw/slc7_x86-64/ROOT/v6-10-06+git_c54db1c10b-1/macros
input_line_60:2:3: error: use of undeclared identifier 'g4libs'
 (g4libs())
  ^

Full log here.

@alibuild

alibuild commented Oct 20, 2017

Copy link
Copy Markdown
Collaborator

Error while checking build/o2/macos for bc280d7:


Full log here.

@matthiasrichter

Copy link
Copy Markdown
Collaborator Author

Looks like FairRoot needs to stay cutting edge and requires a new boost and cmake ;-(

FairRootGroup/FairRoot@62e5354

@matthiasrichter

Copy link
Copy Markdown
Collaborator Author

@ktf, correct, there was a real problem when compiling on xcode, new version has been pushed. Now its ok apart from the already known problems in the simulation unit tests and the new CMake and boost required by FairRoot dev branch.

@sawenzel
sawenzel merged commit 2e537f0 into AliceO2Group:dev Oct 23, 2017
arvindkhuntia pushed a commit to arvindkhuntia/AliceO2 that referenced this pull request Jun 6, 2022
- simplify TEfficiency making
- add efficiency for primary and secondary particles
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants