Date: 2022-07-30
+Accepted
+We need to record the architectural decisions made on this project.
+We will use Architecture Decision Records, as described by Michael Nygard.
+See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's adr-tools.
+ + + + + + +Date: 2023-09-03
+Draft
+We aim for having full support regarding
+Starting from SQLAlchemy 2.0, there's a native support for dataclasses.
+However, it's less seamless than one might initially imagine. It comes down to inheriting explicitly from another base class, namely sqlalchemy.orm.MappedAsDataclass and using it as a base for models:
+
from sqlalchemy.orm import DeclarativeBase
+from sqlalchemy.orm import MappedAsDataclass
+
+class Base(MappedAsDataclass, DeclarativeBase):
+ """subclasses will be converted to dataclasses"""
+
+class User(Base):
+ __tablename__ = "user_account"
+
+ id: Mapped[intpk] = mapped_column(init=False)
+As a result, User becomes a dataclass with attached SQLAlchemy's behaviour.
In terms of risk, this may cause compatibility issues. As a reminder, Event Sourcery's models are declared as bare classes (notice lack of inheritance from Base):
class Stream:
+ __tablename__ = "event_sourcery_streams"
+ __table_args__ = (
+ UniqueConstraint("uuid", "category"),
+ UniqueConstraint("name", "category"),
+ )
+
+ id = mapped_column(BigInteger().with_variant(Integer(), "sqlite"), primary_key=True)
+The assumption is that someone setting up a project with SQLAlchemy will have their own Base class and models. The library will let them attach our models to their declarative base later using event_sourcery_sqlalchemy.models.configure_models, like:
@as_declarative()
+class Base:
+ pass
+
+
+# initialize Event Sourcery models, so they can be handled by SQLAlchemy and e.g. alembic
+configure_models(Base)
+The exception is raised in case MappedAsDataclass appears multiple time in model's MRO, for example:
+- let's say we have a base class for our models that inherits from MappedAsDataclass
+- someone has their own Base that also inherits from MappedAsDataclass
sqlalchemy.exc.InvalidRequestError: Class <class 'event_sourcery_sqlalchemy.models.Stream'> is already a dataclass; ensure that base classes / decorator styles of establishing dataclasses are not being mixed. This can happen if a class that inherits from 'MappedAsDataclass', even indirectly, is been mapped with '@registry.mapped_as_dataclass'
+To reproduce, add MappedAsDataclass as a base to any model and use this snippet:
from sqlalchemy import create_engine
+from sqlalchemy.orm import DeclarativeBase, MappedAsDataclass
+
+from event_sourcery_sqlalchemy.models import configure_models
+
+
+engine = create_engine(
+ "sqlite+pysqlite:///:memory:", echo=True, future=True
+)
+
+
+class Base(MappedAsDataclass, DeclarativeBase):
+ pass
+
+
+Base.metadata.create_all(bind=engine)
+
+
+# initialize Event Sourcery models, so they can be handled by SQLAlchemy and e.g. alembic
+configure_models(Base)
+There is a recipe for providing base class model with overriding __init__:
+
class ModelBase:
+ def __init__(self, *args: Any, **kwargs: Any) -> None:
+ return super().__init__(*args, **kwargs)
+This basically makes mypy unable to verify anything extra because such a signature allows everything to be passed in. Conversely, all operations are considered to be correct from type checker's perspective.
+This solution is a bit tricky and just gets rid of warnings, but doesn't give us any additional benefits.
+__init__ definitionWe can simply provide tailored __init__ for each model to get type safety out of the box.
This is safe & compatible with other ORM's features because SQLAlchemy uses other means to build an object when e.g. fetching them from the database.
+In other words, SQLAlchemy internally is not invoking __init__.
Writing custom __init__ makes the most sense.
Whenever someone changes fields of a model (this should be rare after release), __init__ needs to follow.
+However, this inconvenience can be potentially automated away in the future. At least, we could also have a custom static code check that would guard that.
Date: 2023-10-27
+Accepted
+Previous code layout haven't stood the test of time and wasn't really aligned with e.g. tests.
+The latter were testing particular functionalities, also being poor man's documentation (unless we write a proper one).
+Hence, instead of strictly technical code organization (e.g. interfaces, dtos etc.) we think code should be rather organized around different functionalities.
+Eventually, we agreed on a following subpackages of event_sourcery (for now):
+- aggregate
+- event_store
+- read_model
event_storeThis is a "core" part of the library, with the Event Store itself. It also includes serialization/deserialization based on Pydantic and proper base classes for events.
+Additionally, a couple of other, foundational things find place here, e.g. StreamId type.
aggregateKeeps a base class for Aggregate and Repository that can be used to implement Event Sourcing.
+read_model(experimental at the moment) This package contains code that tracks streams and can be used as a foundation to build read models.
+After changes, library is much more modular and oriented about particular features.
+However, the approach to importing code for a client of the library will be different - they are now supposed to import code not from top-level event_sourcery package, but from one of it's subpackages.
Do:
+- from event_sourcery.event_store import Event, EventStore
+- from event_sourcery.aggregate import Aggregate, Repository
Don't:
+- from event_sourcery import EventStore, Aggregate
Date: 2023-10-27
+Accepted
+The library at the early stages had a functionality of dispatching in-memory events.
+A user of the library could plug-in listeners to particular types of events (also - catch-all listener was supported).
+When events were published via EventStore with configured listeners, the latter were called synchronously one after another. This happened right after events were persisted.
This delivered a very basic mechanism of subscribing to events from e.g. other code modules, similar to e.g. Django signals.
+However, in-memory events implementation failed to conform to Liskov's Substitute Principle, because the handling and consequences of the errors are different for SQL-based backend and Event Store DB.
+With SQL-based, everything is wrapped in a transaction. That means if the one of listeners fails, then the whole transaction is rolled back. With Event Store DB (and possibly other stores as well) there are no transactions, so failing in listener would not reverted changes made in Event Store.
+Hence, some time ago we decided to remove that feature and reconsider adding it back when we understand more.
+After consideration, we decided to NOT include this feature in the library and instead suggest users that they implement it on their own if only they need it. For SQL-based storages, one can use SQLAlchemy's events, in particular after_insert to plug their own logic after event is persisted.
We shall also consider exposing deserialization, so that users can use event objects instead of their representations for persistence (i.e. SQLAlchemy models).
+The similar approach could be also used for future backend implementations, including Django.
+Our maintenance burden would be lower in expense for documenting the approach with leveraging persistence-layer mechanisms, e.g. SQLAlchemy's events.
+ + + + + + +Date: 2023-10-29
+Accepted
+Features of core event store requires integration tests with every backend implementation. +In this context first tests were prepared.
+InMemory event store strategy will be used as default for feature tests. Also, it's a +good starting point to experiment with library or start a project postponing +infrastructure decisions.
+Possible lack of testing backend-specific consequences for features. +This will require additional backend-specific tests.
+ + + + + + +Date: 2023-12-30
+Proposed
+Transactional databases have a chance to simplify projections by projecting +events in same transaction where event is created. It has limitations, but it's +a great start for most projects, and can be enough in most of them.
+Transactional backend will have it's own ability to use in transaction subscription. +It won't be in main API of EventStore, but in every backend API separately. Events +in this subscriptions won't be stored yet, so won't have EventStore position defined.
+Migration from in transaction subscription to async subscription will require some +additional work from developers to synchronize projections on event position. This +probably will need some migration strategy or a new projection from scratch.
+ + + + + + +Date: 2024-07-26
+Accepted
+Event Sourcery supports multiple backends, such as SQLAlchemy, EventStoreDB. More adapters will appear in the future.
+Each integration has its own dependencies, for example EventStoreDB relies on esdbclient which in turn depends on grpcio.
When someone wants to use the library for e.g. Django or SQLAlchemy, they don't need aforementioned packages.
+The approach to packaging and releases should make it possible for library clients to install only packages that are required in their setup.
+In this approach, we release a single python_event_sourcery package that includes all modules i.e. "core" + adapters for esdb, django and sqlalchemy (+ possibly more in the future).
+To control optional dependencies, we'll use optional dependencies + "extras" feature which is widely supported in popular Python packaging tools.
+We already have it implemented for Django - when someone installs our library using "pip install python-event-sourcery[django]", we'll also install Django framework.
+NOTE Real-world Django projects would already have Django installed FIRST and they'll be adding our library LATER. However, it still makes sense to have Django as an optional dependency and extra to cross-validate the supported version. If someone has Django installed and its version frozen, but it's not compatible with our library, they'll see an error.
+Extras propositions for the current project structure:
+- sqlalchemy - SQLAlchemy adapter
+- esdb - EventStoreDB adapter
For example, when someone wants to use our library with SQLAlchemy, they'll have to install it using pip install python-event-sourcery[sqlalchemy].
Downside: maintainers of the library are aware of some in-house solutions that doesn't support "extras". We acknowledge the existence of such a niche, but we won't be treating is a blocker.
+In this approach, python-event-sourcery would be split into multiple separately installable packages, i.e.:
+python_event_sourcery_core - core functionalitypython_event_sourcery_sqlalchemy - SQLAlchemy adapter, depending on python_event_sourcery_corepython_event_sourcery_esdb - EventStoreDB adapter, depending on python_event_sourcery_corepython_event_sourcery_django - Django adapter, depending on python_event_sourcery_coreWe could still have them all in a single repository and maintain a single test suite, but build & release process would be uploading separate packages to PyPI.
+This would require a work to create multiple pyproject.toml files, one per each package and put there the dependency on the core package.
+Downside: in the current setup we have only support for different backends. In the future, when e.g. support for Kafka or RabbitMQ will be added, installation would become more cumbersome. For example, if someone would be using SQLAlchemy and Kafka, they would have to install two packages. With extras, it'll still be a single package but with to extras.
+We decided to go with the "Single package with extras" approach.
+As a consequence, we'll have to put work into maintaining the extras feature in the pyproject.toml file.
+ + + + + + +Date: 2024-10-07
+Proposed
+One of the missing features for v1.0 is support for multitenancy, i.e. data isolation for separate customers (tenants) inside a single application.
+The basic rule is that data from one tenant must not be visible for other tenant.
+We also need to make sure library would still operate in tenant-less context. For example, one may define a subscription that builds read model. In such a context, we expect all data to be available. Also, some applications may wish to maintain tenant-less, "global" data.
+There are multiple approaches to multitenancy: +- row-level multitenancy - each row in a table has a tenant id +- schema-level multitenancy - each tenant has its own schema +- database-level multitenancy - each tenant has its own database.
+In the library, we'll natively support approach similar to row-level multitenancy.
+Other approaches will also be possible to implement with the library, but for now they remain of scope for this ADR or scope of row-level implementation. The blocker is Github issue #57 - Make database tables configurable. Once it's done, one will be able to create separate instances of EventStore, each connecting to other schema or database. Of course this would require e.g. doing schema migration for each tenant schema or database, but if this is a level of isolation required, it means it would have to be necessary anyway. The library itself has nothing to do with it, but should not prevent this pattern from applying.
Since all events are organized into streams, the problem can be reduced to stream visibility. Stream visible for tenant A should have "A" associated. Conversely, "global" stream should have no tenant or default tenant associated.
+Decisions to make: +- do we allow streams to coexist with the same name and ID in different tenants, as well as tenant-less context? +- what type of column should be used for tenant id? +- do we allow for empty (null) values for tenant id?
+To guarantee full data separation we'll allow streams with the same ID and/or name exist in any tenant, as well as in tenant-less context. For example, if we have 10 tenants, there might be 11 streams with the same ID or name.
+This is to avoid awkward API behaviour when trying to load a stream by id when the stream exists but in the current context we have no access to it.
+In an ideal world with full data separation, we'd return something like "stream not found" but with globally unique Stream IDs we should deny access, thus give away the fact that stream exists. If we'd returned "stream not found", we would suggest that one can create a stream with the same ID. When they would attempt to do so, we'd have to handle uniqueness violation, still giving away the fact that stream exists in some other context.
+This is a security risk for the library users and may be abused by attacker. We'd rather be on the safe side and make it impossible for users of the library to introduce vulnerabilities into their software.
+On the brightside, this makes exposing stream ids in URLs or other places kinda ok. UUIDs are still poor from UX perspective, but at least data separation is still guaranteed.
+| Stream tenant | +mode | +access attempt by | +response | +
|---|---|---|---|
| null | +tenant-less | +by id | +yes | +
| null | +tenant-less | +by name | +yes | +
| null | +as tenant1 | +by id | +no | +
| null | +as tenant1 | +by name | +no | +
| 1 | +tenant-less | +by id | +no | +
| 1 | +tenant-less | +by name | +no | +
| 1 | +as tenant1 | +by id | +yes | +
| 1 | +as tenant1 | +by name | +yes | +
| 1 | +as tenant2 | +by id | +no | +
| 1 | +as tenant2 | +by name | +no | +
In the past we were wondering whether we should use integer or UUID type for tenant id. We acknowledged the fact that UUID is actually a superset of integer type in many databases, so we could go with UUID.
+However, this is not the type used in other solutions.
+MartenDB uses varchar column for tenant_id.
+Varchar is definitely less performant than integer or UUID, however this shouldn't be a blocker. We'll add an index to the column because it will be used in all queries.
+In MartenDB, if some documents in are inserted in a tenant-less context they get default value of *DEFAULT*.
In our case we'll use similar value - *default*. This should be safer to use in wider range of databases. In PostgreSQL one could use partial indexes and workaround nulls in the column, but AFAIR in other popular SQL databases a workaround is needed, which would make the whole thing more complex.
Implementing row-level multitenancy in the library entails adding a column "tenant_id" of type varchar to the streams table. This column will be indexed.
Since many streams can co-exist with the same ID or name, we have to start passing tenant_id to any library code that is responsible handling streams, e.g. in projections. This is quite huge change in the library, yet is required to get complete data separation and allows to build the best API for the users of the library.
Because in all contexts we'll always have a tenant (whether it's *default* or a user-defined one) associated with a stream and there might be multiple streams with the same ID and name, EventStore will no longer be able to unambiguously load a stream by ID or name.
Outbox and subscriptions will work "globally" simply by getting tenant_id along with the stream id.
+For now, we don't see a need for an interface to iterate over all streams in all tenants, e.g. getting all streams with the same ID or name. Should such a need arises, we'll consider adding such a thing.
+ + + + + + +Date: 2025-04-11
+Accepted
+In ADR 20231027 #1 the decision was made to package the project in the following way:
+aggregateevent_storeread_modelDuring documentation writing it became obvious that aggregate name is not accurate because inside this package there are more things than just Aggregate class.
aggregate will be renamed to event_sourcing to more accurately reflect its contents.
Date: 2025-07-29
+Proposed
+We want to provide a way to protect privacy-sensitive data in events, inspired by crypto-shredding patterns. This includes marking fields as private, encrypting them, and being able to irreversibly remove access (shred) by deleting encryption keys. The design should be simple for users of the library and support masking of shredded data.
+Annotated[str, Encrypted(...)].DataSubject which identifies the privacy subject.DataSubject field is used by default for all encrypted fields in the event.subject_field if needed.mask_value parameter.mask_value must match the type of the field being encrypted.Encryption class works on entire events, not individual fields.encrypt(event, stream_id) returns a dictionary with encrypted fields.decrypt(event_type, data, stream_id) returns a dictionary with decrypted fields.event_store/interfaces.py:KeyStorageStrategy - interface for key managementEncryptionStrategy - interface for encryption operationsevent_store/event/dto.py:Encrypted, DataSubject)event_store/encryption.py:Encryption class that processes entire eventsevent_store/in_memory.py)event_sourcery_fernet)event_store/exceptions.py:EventStoreException@dataclass
+class Encryption:
+ strategy: EncryptionStrategy
+ key_storage: KeyStorageStrategy
+
+ def encrypt(self, event: Event, stream_id: StreamId) -> dict:
+ """Encrypt an event and return the data dictionary with encrypted fields."""
+
+ def decrypt(self, event_type: type[Event], data: dict, stream_id: StreamId) -> dict:
+ """Decrypt event data and return the processed data dictionary."""
+
+ def shred(self, subject_id: str) -> None:
+ """Remove access to data by deleting the encryption key."""
+from typing import Annotated
+from event_sourcery.event_store.event.dto import Event, Encrypted, DataSubject
+
+class UserRegistered(Event):
+ user_id: Annotated[str, DataSubject]
+ email: Annotated[str, Encrypted(mask_value="[REDACTED]")]
+ public_info: str # unencrypted field
+
+# Configuration
+factory = (
+ SQLAlchemyBackendFactory(session)
+ .with_encryption(
+ key_storage=InMemoryKeyStorage(),
+ strategy=FernetEncryptionStrategy(),
+ )
+)
+
+# Usage (automatic through serialization)
+encryption = Encryption(strategy=strategy, key_storage=key_storage)
+encrypted_data = encryption.encrypt(user_event, stream_id) # Returns dict with encrypted fields
+decrypted_data = encryption.decrypt(UserRegistered, encrypted_data, stream_id) # Returns dict with decrypted fields
+Date: 2025-10-04
+Accepted
+We want to keep the project structure clear and intuitive. Import paths should reflect the functionality provided by each package, regardless from the code evolution. That's why implementation of our core event_store package will be kept as private _event_store subpackage.
+In root package we will keep public API structure with proxy imports to implementation in _event_store.
+Proxy imports will be organized in modules reflecting functionality (e.g.encryption, event, types etc.).
Implementation of the event_store will be moved into _event_store subpackage. Root package will keep public API structure with proxy imports to implementation in _event_store.
\n {translation(\"search.result.term.missing\")}: {...missing}\n
\n }\n \n