Optimizations to the streaming of MCTruthContainer - #2151
Conversation
|
Hi Matthias,
This looks like a good approach, but I wonder if it wouldn't be better to do it in a more general way:
- we could add the flatten / restore functionality with polymorphic allocators to gpu/common/flatobject.h,and then make the mc container derive from it.
- That should make the mc cintainer compatible to gpu.
- And it should allow for sending the other gpu flat objects we already have via dpl?
What do you think? We can still merge this as is and extend it later, but eventually i'd prefer to base that functionality on the flatobject.
Kind Regards
David Rohr
Sent from my mobile. (Excuse the typos!)
…On July 1, 2019 11:03:32 AM GMT+02:00, Matthias Richter ***@***.***> wrote:
The MCLabelContainer has two vectors of messageable type which do not
need to be root serialized.
For the moment, flattening always requires a copy, but the data can be
directly copied
to the message resource. The prototype can be extended later to use
multiple parts as allowed by the
DataHeader.
Restore can be done using spectator memory resources for the two
regions of the input buffer, but this kind of memory resource needs to
be implemented. For the moment we have a copy also here.
The DPL I/O API will be extended to use the custom methods for
flatten/restore. In the end its not much different then the BOOST
serialization concept, but we have the option to optimize this by using
polymorphic memory resources and allocators.
TODO before merging:
- [ ] find names for flatten/restore methods (if not yet ok)
- [ ] check the necessary changes in the DPL I/O API (we might want to
sort out #1970 before)
- [ ] develop "spectator memory resouces", may be several of this kind
working over the memory of of the `o2::pmr::MessageResouce`
- [ ] alternatively check if the two vectors can be sent in individual
messages, `o2::pmr::MessageResouce` could be used directly
This concept can be extended to a generic tool later.
You can view, comment on, or merge this pull request online at:
#2151
-- Commit Summary --
* using vectors with polymorphic_allocator in the MCTruthContainer
* Adding flatten and restore functionality to the MCLabelContainer
-- File Changes --
M
DataFormats/simulation/include/SimulationDataFormat/MCTruthContainer.h
(110)
M DataFormats/simulation/test/testMCTruthContainer.cxx (36)
-- Patch Links --
https://github.com/AliceO2Group/AliceO2/pull/2151.patch
https://github.com/AliceO2Group/AliceO2/pull/2151.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#2151
|
|
Hi David, I'm thinking in a similar direction. For the moment I'm planning for a prototype to initiate the discussion. I had a look to the |
|
Hi @matthiasrichter : for examples, please check GPU/TPCFastTransformation/TPCFastTransform.h or or Detectors/Base/include/.../MatLayerCylSet.h The FlatObject is actually not an interface for polymorphic allocation. Its main purpose is exposing the setActualBufferPtr and setFutureBufferPtr methods, as well as the getFlatBufferSize and getFlatBufferPtr, which can be used to ship it into preallocated memory on the GPU. In addition, it can be root-serializable if one calls setActualBufferAddress after reading from file. My idea is to complement (not replace) this with the polymorphic allocator, which should be rather trivial. Then, the flat objects could do both: being shipped in DPL with polymorphic allocators, and be moved to GPU with the current interface. |
|
Is this still being worked on or do we rely on other solutions? |
Adding flatten_to and restore_from methods which allow to flatten to a container instance and restore the vector content from a buffer, both by copy. This can be used with a container with pmr allocator and underlying memory resource. Flattening always requires a copy, but the data can be directly copied to the message resource. Restore can be done using spectator memory resources for the two regions of the input buffer, but this kind of memory resource needs to be implemented.
Using custom streamer for MCTruthContainer<MCCompLabel> and flatten the two vectors into one additional char vector member before serialization. The original vectors are empty during serialization. When reading an object, the vectors are restored from the char vector which has been streamed. The reduced serialization time for one single char vector and two empty vectors outweighs the cost of flattening and restoring. There is a significant speedup in the serialization. As the next step, for transmitting messages the serialization can be skipped completely by a custom DPL IO interface implementation for MCTruthContainer. TODO: - add move assignment from a source vector, by that passing an object which has access to different underlying memory resources, until that, the pmr::MemoryResource has been removed again - add interface to access header and truth elements directly from the raw buffer, by that inflation can be postponed until new elements are added, with the effect that inflation can be avoided in most cases. This avoids copy for deserialized read-only objects.
c58efd2 to
2df94b5
Compare
|
I did many tests during the last week, the newly pushed update is a condensed result. Main results:
Details: Unfortunately ROOT schema evolution does not work correctly if the two original vectors are marked transient, which would be in principle possible. The ,from a read pragma, generated conversion function does not access the members of the old object correctly. The workaround is thus to keep the original members and leave them empty for serialization. More to come: During my tests I have also created a direct access to the flattened buffer, so that read-only access to a deserialized object can be even more efficient. Code has to get finalized though, leaving this for a later optimization. Once we have a pmr memory resource working as spectator over message buffer, we can extract a vector and pass it directly to MCTruthContainer, making serialization completely unnecessary. |
|
@shahor02 @davidrohr can you please check the tracking benchmarks with this update. Is there a procedure and some tests I can run myself to verify the changes? |
|
@davidrohr we can do the conversion to |
shahor02
left a comment
There was a problem hiding this comment.
I run the its and tpc tracking with your branch and dev and the results are exactly the same, both for the reconstruction and for assigned MClables.
Cheers,
Ruben
davidrohr
left a comment
There was a problem hiding this comment.
OK with me (Ruben was faster with his test), the minor comment can be addressed when we integrate FlatObject.
| #define ALICEO2_DATAFORMATS_MCTRUTH_H_ | ||
|
|
||
| #include <TNamed.h> | ||
| #include <TNamed.h> // to have the ClassDef macros |
There was a problem hiding this comment.
For later when we port to FlatObject and make it compatible to GPU, we should use GPUCommonRtypes.h for ClassDef macro, instead of the ROOT header.
|
thanks for the feedback! |
|
then we are ready to go and I leave the open items as task list for future extensions. |
|
@matthiasrichter concerning the custom streamers of |
|
Yes, I have this on the radar. We actually need to implement for every specialization. This is because Alternatively we can skip |
The MCLabelContainer has two vectors of messageable type which do not need to be root serialized.
For the moment, flattening always requires a copy, but the data can be directly copied
to the message resource. The prototype can be extended later to use multiple parts as allowed by the
DataHeader.
Restore can be done using spectator memory resources for the two regions of the input buffer, but this kind of memory resource needs to be implemented. For the moment we have a copy also here.
The DPL I/O API will be extended to use the custom methods for flatten/restore. In the end its not much different then the BOOST serialization concept, but we have the option to optimize this by using polymorphic memory resources and allocators.
TODO before merging:
o2::pmr::MessageResoucecould be used directly -> flattening can be done directly to buffer with underlying memory resourceThis concept can be extended to a generic tool later.
Next developments which will not be part of this PR
o2::pmr::MessageResouce