Wishbone modules process and transport messages in one way or the other. Obviously, this needs to happen as reliable as possible. Wishbone has a particular way of dealing with exceptions. In this article we cover the role unhandled code exceptions can play and how we can take advantage of them by just allowing them to happen.
Failed and successful queues
Messages travel between modules via queues. By default, a module has an inbox, successful and failed queue. Typically, a module has a function registered against the inbox queue. This function will then process each message from the associated queue. After the function successfully processes the message it will be automatically submitted to the module's successful queue. On the contrary, whenever the function generates an exception while processing the message, it will be submitted to the failed queue.
If a queue is not connected to another queue it will drop the messages it receives. So by connecting another module to the failed queue, we can forward the offending messages to another module in order to construct a failover strategy. Therefor it is important not to trap any exceptions inside the registered function which allows the Wishbone framework to submit the message to the module's failed queue.
The TCPOut module
There is no error handling code inside this function. If submitting a message to the socket fails an exception will be raised. Whenever that's the case, the the message will be submitted to the module's failed queue.
A failover strategy
The described behavior can be demonstrated with a test setup using following bootstrap file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
--- modules: dictgenerator: module: wishbone.input.dictgenerator arguments: interval: 0.5 output_1: module: wishbone.output.tcp arguments: port: 10001 output_2: module: wishbone.output.tcp arguments: port: 10002 output_3: module: wishbone.output.tcp arguments: port: 10003 diskout: module: wishbone.output.disk arguments: directory: ./buffer diskin: module: wishbone.input.disk arguments: directory: ./buffer idle_trigger: true idle_time: 20 funnel: module: wishbone.flow.funnel routingtable: - dictgenerator.outbox -> funnel.one - funnel.outbox -> output_1.inbox - output_1.failed -> output_2.inbox - output_2.failed -> output_3.inbox - output_3.failed -> diskout.inbox - diskin.outbox -> funnel.two ...
This bootstrap file creates a Wishbone server which generates a random dictionary every half a second. This dictionary is submitted to the output_1 module instance which will try to submit the message to localhost tcp/10001. If this fails then the message will be forwarded to module instance output_2 which tries to submit the message to tcp/10002. When that fails then the message will be forwarded to the module instance output_3 which tries to submit the message to tcp/10003. Whenever the third destination fails then there message is forwarded to the disk buffer module, which feeds back into the TCP destinations in order to retry submitting the messages again to one of the TCP destinations.
$ wishbone debug --config failovertest.yaml
In three separate terminals start following socat instances:
$ socat tcp4-listen:10001,fork stdout
$ socat tcp4-listen:10002,fork stdout
$ socat tcp4-listen:10003,fork stdout
At this point, messages should be arriving to the socat instance listening on port 10001. Interrupting that socat instance makes the messages arrive to the instance listening on port 10002. When the first socat instance is restored then the messages should be arriving back there again. If all three TCP destination are unavailable then the messages are submitted to a disk buffer. You should see Wishbone log error messages when submitting data to a TCP destination fails.