Exceptions as a feature

Failed and successful queues

Messages travel between modules via queues. By default, a module has an inbox, successful and failed queue. Typically, a module has a function registered against the inbox queue. This function will then process each message from the associated queue. After the function successfully processes the message it will be automatically submitted to the module's successful queue. On the contrary, whenever the function generates an exception while processing the message, it will be submitted to the failed queue.

If a queue is not connected to another queue it will drop the messages it receives. So by connecting another module to the failed queue, we can forward the offending messages to another module in order to construct a failover strategy. Therefor it is important not to trap any exceptions inside the registered function which allows the Wishbone framework to submit the message to the module's failed queue.

The TCPOut module

The wishbone.output.tcp module submits messages to a TCP socket. The consume function has been registered to consume all messages from the inbox queue.

There is no error handling code inside this function. If submitting a message to the socket fails an exception will be raised. Whenever that's the case, the the message will be submitted to the module's failed queue.

A failover strategy

The described behavior can be demonstrated with a test setup using following bootstrap file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

  ---
  modules:

    dictgenerator:
      module: wishbone.input.dictgenerator
      arguments:
        interval: 0.5

    output_1:
      module: wishbone.output.tcp
      arguments:
        port: 10001

    output_2:
      module: wishbone.output.tcp
      arguments:
        port: 10002

    output_3:
      module: wishbone.output.tcp
      arguments:
        port: 10003

    diskout:
      module: wishbone.output.disk
      arguments:
        directory: ./buffer

    diskin:
      module: wishbone.input.disk
      arguments:
        directory: ./buffer
        idle_trigger: true
        idle_time: 20

    funnel:
      module: wishbone.flow.funnel

  routingtable:
    - dictgenerator.outbox  -> funnel.one
    - funnel.outbox         -> output_1.inbox
    - output_1.failed       -> output_2.inbox
    - output_2.failed       -> output_3.inbox
    - output_3.failed       -> diskout.inbox
    - diskin.outbox         -> funnel.two
  ...

This bootstrap file creates a Wishbone server which generates a random dictionary every half a second. This dictionary is submitted to the output_1 module instance which will try to submit the message to localhost tcp/10001. If this fails then the message will be forwarded to module instance output_2 which tries to submit the message to tcp/10002. When that fails then the message will be forwarded to the module instance output_3 which tries to submit the message to tcp/10003. Whenever the third destination fails then there message is forwarded to the disk buffer module, which feeds back into the TCP destinations in order to retry submitting the messages again to one of the TCP destinations.

To start Wishbone execute following command [1]:

$ wishbone debug --config failovertest.yaml

In three separate terminals start following socat instances:

$ socat tcp4-listen:10001,fork stdout

$ socat tcp4-listen:10002,fork stdout

$ socat tcp4-listen:10003,fork stdout

At this point, messages should be arriving to the socat instance listening on port 10001. Interrupting that socat instance makes the messages arrive to the instance listening on port 10002. When the first socat instance is restored then the messages should be arriving back there again. If all three TCP destination are unavailable then the messages are submitted to a disk buffer. You should see Wishbone log error messages when submitting data to a TCP destination fails.

Final words

By chaining the failed queues into a sequential list of destinations it's fairly easy to create a fail-over strategy in a Wishbone setup. Questions and suggestions always welcome!