An Introduction to Event Sourcing

In this article, we would like to present a design pattern of microservices architecture called Event Sourcing. The core concept of Event Sourcing revolves around capturing every modification made to an application’s state as an event object and preserving these events in the order they occurred throughout the application’s lifespan.

Deleting or modifying existing events is prohibited in this system. You are not only storing the events but also keeping the context of the events. This ensures a comprehensive understanding of what occurred and why in a given business operation.
There can be many domains that Event Sourcing can be applied to, such as E-commerce and retail, financial systems, and the Internet of Things. In the financial systems domains, it can be valuable in the systems that are dealing with financial transactions, where the traceability of the events is crucial. In IoT applications, event sourcing can capture events from devices and sensors which enables the analysis of the behavior of the device, detecting patterns if necessary or helps with the data-driven decisions.

So, what is an “Event”?

In the context of Event Sourcing, an event is the data that indicates a specific occurrence that took place in the domain. These events are like the definitive records, this means everything that happened is encapsulated in events. So the current state is derived from the sequence of these immutable events. For now, we will show example events of an e-commerce app which is also easier to understand.

As is shown here, they are named in the past tense and events also contain metadata such as the timestamp when the event took place.

Event Database

The event database stores the events that have happened. Naturally, they are different from traditional databases (document, relational, etc.). An event database is designed for storing the history. Events are stored in an append-only log. Instead of replacing or overwriting existing data, Event Sourcing involves adding new events to the end of the event log. In this way, a sequence of immutable events is created chronologically.

As it is mentioned earlier in the context of event-sourced systems, there is no concept of updating and deleting. Every change made is defined as an event and you can only append it to the event log. This works in the same way as an accountant’s ledger. Event Sourcing is not something new, it has been used in production by some of the world’s biggest companies (including your favorite streaming service, Netflix)[1].

It is easy to assume that Event Sourcing is mainly beneficial for auditing purposes. However, this view is limited. Undoubtedly an append-only system is good for auditing, but Event Sourcing offers more power than that. An audit log is simply a record of changes in chronological order but it doesn’t provide any additional context. The “why” and “when” of the event are also stored. This information can provide a wealth of valuable information and context that can benefit the business in various ways.

The most famous event store is EventStoreDB which is a state-transition database.[2] As mentioned above, these databases not only store your data but also the changes that happen to your data over time. Thus we can record the changes that occurred while providing insights into the changes that had happened in chronological order. Eventstore lets you store data in JSON format.

Benefits of Event Sourcing

Auditing

The immutability and comprehensive storage of events within Event Sourcing make it quite easy to verify any changes that have occurred within the system. If any unexpected changes in the data state arise due to bugs or malicious activities, the specific event that caused the issue can be easily identified. The storage of all state changes within Event Sourcing makes it feasible to replay these changes and move the system back and forth in time, a capability that is immensely useful for debugging and analyzing various business scenarios. So it works like a time machine.

Fault Tolerance

Event streams are similar to logs, but with added backup and recovery capabilities, making them a valuable tool for enabling system recovery through downstream projections. Furthermore, the use of an append-only log as the write model makes it easy to partition the data, providing scalability.

Analysis of Data

Since an event represents a fact that took place within the business, any business-related event can be traced back to the events that have been stored, making Event Sourcing an effective means of providing traceability. This makes data analysis the most important advantage of Event Sourcing

Drawbacks of Event Sourcing

Storage

Since Event Sourcing relies on storing the entire history of the events, it can result in increased storage requirements. The event log can grow significantly over time. It may require effective strategies for storage while maintaining performance.

Event Schema Evolution

As application requirements change over time, event schema may change. As a result of the evolution, you may end up introducing changes to your event schema structure. One should keep in mind that most of these changes have to be backward compatible and the system should handle different event versions. Schema evolution can be complex and requires careful planning.

Learning Curve

Event Sourcing introduces a different programming paradigm compared to traditional state-centric approaches. It may require developers to learn new concepts, leading to an initial learning curve. Moreover, implementing a solution on a legacy codebase may require significant effort.

Benefits of the IoT

Event Sourcing and the Internet of Things are two technologies that can be combined to create powerful applications. Utilizing Event Sourcing establishes a solid base for handling massive data flow from IoT devices.

• Since Event Sourcing ensures data integrity, the data captured by IoT devices will always be consistent. In smart manufacturing, the data coming from the sensors can be used to track the real-time status, and if a sensor detects a fault in the machine, Event Sourcing can be used to reconstruct the sequence of the events that lead up to the fault to identify the root cause and take action
• In addition to the smart manufacturing context, Event Sourcing can also be used in connected car applications like live location and speed tracking. Additionally, it can provide a complete list of changes of the vehicle’s data which can be used for pro-active maintenance.
• In smart building applications, Event Sourcing can be used to track the status of various systems such as lighting, heating, ventilation, and air conditioning. Usage of Event Sourcing can lead to optimized energy consumption. If an outage occurs, Event Sourcing can be used to replay events to reconstruct the state of the building and restore it as quickly as possible.

In a nutshell, event sourcing and IoT are powerful technologies that can be combined to create innovative applications. By leveraging the strength of both technologies, the organization can benefit from data integrity, real-time analytics, fault tolerance, and error recovery.

[1] Scaling Event Sourcing for Netflix Downloads, Netflix Technology Blog, https://netflixtechblog.com/scaling-event-sourcing-for-netflix-downloads-episode-1-6bc1595c5595

[2]EventStoreDB: A State-Transition Database, https://www.eventstore.com/