Configuration and monitoring of Forest Bus
The following topics are covered by this administration guide:
The forest-bus-server command supports the following options:
|The network name and port (e.g :3000) for the server to listen for incoming connections on. Both other server connections (when configured as a cluster) and Go client connections are served on this port. Required
|The location of the configuration and data storage path to be used by this forest-bus-server. Currently all data for all topics is stored under this path, in future releases distributing the topic data across different paths will be supported. Required
|The Cluster ID that this server is part of. Once set this cannot be easily changed. If the server is started up with a different Cluster ID to the one recorded under -path from a previous run, forest-bus-server will not start. The use of Cluster ID is optional but recommended.
|This is the network name and port (e.g. :5000) that forest-bus-server will listen on for incoming CBOR RPC connections. Java and Python clients connect to forest-bus-server using CBOR RPC. If you are only using Go clients and / or the forest-send and forest-get clients then this is not required.
|The network name and port to publish server metrics on (e.g. :8000). A single URL,
/debug/vars, is published at this address and contains a JSON object with server variable and performance information. This is used by the Forest Bus Monitor and can be used for other monitoring tool integration.
|An optional alternative network name and port (e.g. :6000) for Go clients to connect to using GOB RPC. This is optional and only required if you want to separate forest-bus-server peer connections from client connections.
The forest-admin command is used to distribute Forest Bus configuration information to one or more servers.
The overall usage of the forest-admin command is:
forest-admin [-id <clusterID>] [-node node1[,...]] command [arguments]
The general usage of the -id and -node flags is:
|The Cluster ID of the server(s) that the configuration change is going to be made on. This needs to match the Cluster ID given on the command line to the forest-bus-server.
|A comma separated list of forest-bus-server nodes (e.g. localhost:3000,localhost:3001) that the sub-command should be sent to.
forest-admin -id <clusterID>] [-node node1[,...]] peers peer1[,...]
The peers subcommand is used to tell a server the list of nodes that makes up a cluster. It takes one parameter, a . Even if a cluster only has one memeber, it needs to have the forest-admin peers command run against it. If the -id option is used with peers, a file will be created called ClusterID.nodelist containing the list of peers passed to forest-admin.
There are two special cases of using forest-admin:
forest-admin -id <clusterID>] [-node node1[,...]] topic <topicName> \
[-segmentSize=<target size in MB>] \
[-sgementCleanupAge=<age in hours>]
The topic subcommand allows the administrator to:
Introduced in version 1.1
forest-admin -id <clusterID>] [-node node1[,...]] removetopic <topicName>
The removetopic subcommand allows the administrator to remove topics from the nodes. The topic will be shutdown gracefully and the configuration removed. The data can then be deleted from the file system.
If the data has not been deleted, the topic can be restored using the
forest-admin topic command.
Each topic in Forest Bus elects a leader node from within the cluster. This leader node accepts messages from clients, writes them to disk and then sends them to all the other nodes in the cluster. When a follower node has recieved a message, it also writes the message to disk and then sends back an acknowledment to the leader node.
The RAFT algorithm depends on a majority of the nodes in a cluster acknowledging the receipt of a new message for the message to be deemed committed. This means that for a cluster of two nodes, both need to be up and running for writes to be committed. Reads of already committed messages can be supported by any number of nodes.
As a consquence of this, three nodes are recommended as the minimum set for clustering. This allows any one node to fail while the remaining two provide write service. Four nodes extends the number of replicas of data made, but doesn't increase write resilience as three nodes need to be working for a message to be committed. Five nodes allows for up to two failures while still providing write service, but at the cost of the leader node having to send messages to four other servers and needing two acknowledgments for it to be committed.
When all nodes in a Forest Bus cluster is restarted it needs to determine which messages are committed across the nodes. If all of the nodes in the cluster are running and are up to date, the commit index will be determined automatically. If one or more of the nodes is down at startup the commit index can only be determined once the first message has been sent to each topic.
Forest Bus Monitor connects to a forest-bus-server's http interface and presents the internal state as a web page. The rest of this section will describe the key elements displayed and their meaning.
The Server section on the right displays overall statistics for this server.
This section shows the status of this forest-bus-server's connections to the other peers in the cluster. For each peer the forest-bus-server establishes a connection pool of 10 connections which are shared by all of the topics on the server. The table shows for each peer:
The Current value shows the snapshot absolute figure and the TPS value is the transactions per second over the last 5s interval.
Each topic on the server has it's own section that can expanded by clicking on the name of the topic. A number of tabs are shown depending on whether this server is currently the leader for a topic or a follower.
High level information regaring the topic is shown:
If this server is the leader of this topic, additional stats are shown here.
This section shows how many messages are being received for this topic and how effectively the server is being able to batch them together for maximum through-put.
This section shows how the follower nodes are performing from a leaders perspective.
This section shows how well the server is performing for a given topic.
The Head Cache is a small cache of messages that have been recently appended to a topic. It is used to service internal requests for information about the latest messages, leaders sending recent messages to followers and clients requesting messages.
This section shows details of how the topic is being stored to disk.
Hover the mouse over each row to see the path to the segment file.