This is a high-performance alternative to using System.Data.
It is a set of interfaces for use with in-memory columnar databases or ECS (Entity Component System) architecture.
In traditional object-oriented approaches large amounts of data are stored as collections of objects or structs. This is also referred to as the array-of-structs approach. This can be thought of as a collection of rows in a table.
The alternative to lists of objects is a column-oriented layout, also known as struct-of-arrays. The column-oriented approach is more efficient for in-memory processing, and allows for extremely efficient packing of the columns.
For more information see:
- Apache Parquet - column oriented file format
- Apache Arrow - column oriented in memory database
Ara3D.DataTable is primarily a simple set of interfaces, that allows a unified method of working with columnar data in memory or serialized from disk.
It is the basis of other libraries that might do compression, (de-)serialization, queries, conversion to-from different formats, and transporting data to/from relational databases or object-relational mapping (ORM).
At Ara 3D we use this library to create efficient code for working with BIM data associated with large AEC models.
IDataSet- A collection ofIDataTableobjects.IDataTable- A collection ofIDataColumnobjects.IDataRow- A collection of values for a single row (usually created on demand)IDataColumn- A buffer of unmanaged values for a single column.IDataDescriptor- A set of column descriptors (type, name)
DataTable comes with a set of builder classes to make it easy to construct simple object tables in memory incrementally that satisfy the interfaces. These classes are not optimized for performance, but instead designed for simple use cases.
DataSetBuilderDataTableBuilderDataRowBuilderDataColumnBuilder
No type checking is done on objects passed into table builder.
This library has many parallels to System.Data, but there are several issues with using System.Data classes like DataSet and DataTable directly.
| Issue | Impact in a real‑time viewer |
|---|---|
| Memory overhead | Each DataRow is a full object with boxing for value types, versioning arrays, and two hash tables |
| GC pressure | Per‑row allocations plus frequent ColumnChanged events drive Gen‑0/1 churn; stalls become visible at frame‑times < 16 ms. |
| Weak typing | weakly‑typed columns (object cells) kill inlining, require casts, and move many errors to runtime. |
| No columnar access | Many analytics scan rows sequentially; CPUs love structure‑of‑arrays, not array‑of‑structures. |
| Thread‑safety locks | DataTable internally synchronises; fine for editor‑thread, harmful if your physics or analysis jobs touch the same table. |
| Limited relational modelling | Cross‑table constraints exist, but traversing n‑level object graphs is verbose and slow compared with a proper ORM or ECS. |
| Versioning you don’t need | Proposed/Current values cost memory even if you never call BeginEdit. Can’t switch them off. |
| Complexity | DataTable is a complex object with many properties and events. It’s hard to understand and use correctly. |
We created Ara3D.DataTable to handle large amounts of relational data in memory efficiently.
This project is built using .NET 8 and has no dependencies.