About Forest Bus

An introduction to Forest Bus

What is Forest Bus?

Forest Bus is a standalone message bus / distributed commit log that allows clients to send messages to topics and receive messages from topics. This simple mechanism allows for a number of uesful design patterns:

Loosely coupled integration	Software components can trigger activities in other sub-systems by publishing a message to a Forest Bus topic. Any errors or downtime in other sub-systems are isolated from the publisher.
Real time reporting	Publish business events to a Forest Bus topic as they happen, rather than producing multiple hourly or daily extracts. This isolates operational systems from analytic reports, while enabling real time reports to be created.
Operational Reslience	Clients specify which message to start reading from, allowing them to catch-up after downtime and reprocess data after bug fixes or feature enhancements

How to use Forest Bus

Once a Forest Bus cluster is up and running (see the quick start guide) use forest-bus-admin to create a topic that will hold messages. Producers use one of the Client libraries to connect to the server(s) that form a particular cluster, and send batches of messages to the topic. Multiple clients can send messages to the same topic at once.

When sending messages to a cluster, Clients can choose to wait for the cluster to confirm the messages are committed before returning. If message delivery isn't critical then messages can be sent without waiting for the commit to take place. The Forest Bus cluster will return the list of IDs created for a batch of messages - these IDs uniquely identify the message within the topic.

Subscribers that are interested in messages in this topic also connect to the cluster using a Client library. Each subscriber is responsible for recording the last message index (ID) that they processed. Multiple clients can read from the same topic and can choose to re-read the same messages again if required. Clients can either wait for new messages to arrive on a topic and read them in real time, or they can connect periodically and read those messages that are available.

How does it work?

Forest Bus can be run as a single server or as a cluster of nodes that provide resliency. When multiple servers are used they coordinate the distribution of the commit log using the RAFT algorithm, essentially this means that:

Each server knows about the other and between them they elect a leader for a given topic
Topics can therefore be spread between different servers
If a server becomes unresponsive, elections will be held and a new leader elected
The client libraries connect to each server and remember where they found the leader for each topic
Leaders send messages to the other nodes and mark messages as committed once a majority of them have confirmed receipt

Sections

Software

The full list of my published Software