Coding is Actually Simple!! - Dramatiq & Rabbitmq - Baby Steps towards a Production Model
Let’s say you want to build an enterprise scale application. Clearly, it can’t sit isolated, right? There should be some kind of integration with the outside world. Let’s assume you are building a CRM or SCM or ERP. All three of them, for instance, deal with some kind of EDI based message structure. Whether you are directly using the EDI standards or whether you are writing your own, the reality is that you will depend upon someone else to run the message orchestra. For instance, consider an order. What do you see as the statuses?
Order Create, Order Change, Delivery Confirmation, Shipment, Proof of Delivery, Invoice.
Now, in the real world, it is not possible that a single system has all this information (for example, if it is doing a shipment, how can it do a Proof of Delivery? And that means you need at least two systems doing this. In it’s crudest possible form, such a setup looks something like this.
Even if you call EDI Broker as a middleware system, the reality is that you need at least three entities to do this work. Now, ERPs like SAP have inbuilt SCM(which you may or may not want to use) but still, you have a supplier to communicate with. That’s where the concept of middleware comes from. It’s that ether which floats everywhere and acts as a bridge between systems. So, the problem statement is something like this.
I want a system which creates orders and sends it as XML every x minutes. This data is passed through a message broker which feeds into a database: The initial load is into a staging layer from which the data is written into the end tables.
Let’s use Dramatiq as a background task processor and Rabbitmq as the message broker. The concept is this. Dramatiq will run a process which generates XMLs and puts into a Rabbitmq channel.
All you are doing is declare a broker and queue a task using dramatiq decorator and run the service through dramatiq CLI(dramatiq tasks where tasks.py is your dramatiq file).
The question is this. We have a service which runs to push the data to a queue. But what triggers it and what is sent? Consider the below code snippet. There are only three parts in it.
Run a loop
Capture a function output as a variable
Run the function declared with dramatiq function with additional options as send or send_with_options. Common pitfall: Pass the function output as a variable. That’s what the receiving system expects to read.
Now, the message is sent out, all you need is another Rabbitmq service to read this message. This is also equally basic.
There is only one word which is of our interest - callback. This is the function which will be called when a message is consumed. In other words, it’s the entry point for your code which reads the XML, parses it, writes to a staging table and does everything.
My structure here is this. Have four tables as staging:
A raw dump: This contains the complete XML
A header table: This is my standard pattern to load the schema. Now, this will have to be split into two tables: If this data comes as an integration endpoint, you don’t need to store the schema for every message. Maintaining it centrally is more than enough. But, if you are using a file upload, you really need the schema - both for lazy load and to ensure user mistakes are captured when trying to rebuild the data.
A data table: Data joined through a delimiter
A table to link the raw XML to the end data. For instance, if I am processing orders as is this case, the primary link is order number. This table should hold that information to easily fetch the XML on demand.
This setup can be used for anything - I chose orders, you can choose traffic management.
What next? A front-end to make this look complete, and a dashboard like grafana to track the metrics.