Send to a stream or queue

Status

Variants

Publish rows to a message broker

Send each row as a message to a topic, queue or stream on a broker such as Kafka, JMS, MQTT or Kinesis.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Reach for a stream or queue when another system has to react to each row as it arrives, rather than pick up a finished file later. (Meridian's view) In Apache Kafka, producers are the client applications that publish, or write, events to Kafka, and consumers are those that subscribe to and read them.^[3] In Kafka, producers and consumers are fully decoupled, and producers never need to wait for consumers.^[3] In Kafka, an event, also called a record or message, has a key, a value, a timestamp and optional metadata headers.^[3] Kafka organises and durably stores events in topics, and a Kafka topic can have zero, one or many producers writing to it.^[3]

Message queuing systems often offer options for delivery, such as whether a message must be delivered at least once or no more than once.^[1] Settle which of those promises the receiving system needs before sending; a receiver that may see a row twice needs a key it can deduplicate on. (Meridian's view)

Kafka topics are partitioned, which means a topic is spread over a number of buckets held on different Kafka brokers.^[3] Kafka writes events with the same event key to the same partition, and guarantees that a consumer reads a partition's events in the order they were written.^[3] In Apache Hop's Kafka producer, a message with no key is distributed to a partition at random.^[4] When the order of rows for one customer or account matters downstream, send that identifier as the message key. (Meridian's view) Apache Hop's Kafka producer can read each row's topic from a field, and a row whose topic field is empty makes the Kafka producer fail.^[4]

The method

In a typical message-queueing setup, a system administrator installs and configures a queue manager or broker and defines a named message queue, or registers with a message queuing service.^[1] A queue manager stores messages until a receiving application connects, and then calls the software routine that application registered.^[1]

Three standards are used in open-source message queue implementations: Advanced Message Queuing Protocol (AMQP), Streaming Text Oriented Messaging Protocol (STOMP) and MQTT.^[1] AMQP is a feature-rich message queue protocol, approved as ISO/IEC 19464 since April 2014.^[1] MQTT, formerly MQ Telemetry Transport, is a lightweight message queue protocol especially for embedded devices.^[1]

Compared with the alternatives

The Hypertext Transfer Protocol (HTTP), used in the World Wide Web and in web services, operates synchronously: a user sends a request and then waits for a reply.^[1] In publish/subscribe systems, an application publishes information for any number of clients to read.^[1] In a publish/subscribe system it would not make sense for the sender to have to wait if, for example, one of the recipients had crashed.^[1] Prefer a queue or stream over a direct request to an API when the sender should not stall on a slow or absent receiver. (Meridian's view)

Unlike traditional messaging systems, Kafka does not delete events after they are read; a per-topic setting says how long Kafka retains them.^[3] Events in a Kafka topic can be read as often as needed.^[3]

Sources

1. [1]Message queuesecondary source
2. [2]Message brokersecondary source
3. [3]Introduction | Apache Kafkadocumentation
4. [4]Kafka producer | Apache Hopdocumentation