Architecture Overview Documentation #6248

Activity

1. AtnNn commented on Feb 7, 2017

   

   AtnNn

   on Feb 7, 2017

   Member

   More actions

   Thanks for asking. This type of documention does indeed belong in this repo. There is some incomplete information in src/README and in comments spread throughout the source. I don't have full answers to all your questions, but here are some pointers for where to look:

   • src/arch/io The lowest-level IO layer has implementations for each supported platform
   • src/serializer/log The log serializer fills file extents with logically-addressed blocks. They are garbage collected.
   • src/buffer_cache Those blocks are cached in memory
   • src/btree They contain btree nodes which contain documents and indexes.
   • src/clustering It is all abstracted with some sharding and replication
   • src/rdb_protocol The data is accessed using the ReQL query language

   We allocate memory using jemalloc. Some specific things like coroutine stacks are pooled.

   @danielmewes gave an overview of the garbage collection in a comment here:

   Internally, we divide the data file into extents of currently 512 KB each. At any time, there is a set of active extents. Data is written only to the end of an active extent. Once an extent is full, a different extent becomes active. Eventually, old extents get garbage collected on a per-extent basis. Still current data from the extent is copied over to one of the active extents. Once all current data has been copied out of the garbage collected extent, it can become active again and is overwritten with new data. In a sense, RethinkDB is append-only on a per-extent level.

   The current cache was implemented by @srh in #1642. There is some discussion about its design in #97.

   If you have any questions that are more specific, please come bug us in https://rethinkdb.slack.com/messages/engineering/.
2. AtnNn

   added this to the subsequent milestone on Feb 7, 2017
3. mglukhovsky commented on Feb 7, 2017

   

   mglukhovsky

   on Feb 7, 2017

   Member

   More actions

   We've also been thinking about having some of the core engineers record sessions where they describe key pieces of the architecture to help disseminate useful information. Creative ideas are welcome!