Publish / stream · All operations

Send rows over a socket or pipe

A variant of Send to a stream or queue

Write rows straight to a listening network socket, a UDP or syslog endpoint, a named pipe or another instance, with no broker in between.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache NiFi (3)
Pentaho Data Integration (1)
  • Socket writerread 2026-09-27

    Serve rows over a socket to a Socket reader step

Talend (2)
  • tNamedPipeOutputread 2026-09-27

    Write rows into an open named pipe

  • tSocketOutputread 2026-09-27

    Send the rows of the flow to a listening socket port

Background

A network socket is a software structure in a network node that serves as an endpoint for sending and receiving data across the network.[1] In Talend, the tSocketOutput component sends out the data from the incoming flow to a listening socket port.[2] Apache NiFi's UDPEventRecordSink formats records and sends them as User Datagram Protocol (UDP) datagram packets to a configurable destination.[3] A named pipe is one of the methods of inter-process communication (IPC) on Unix and Unix-like systems.[4]

When to use it

A named pipe can be used to transfer information from one application to another without an intermediary temporary file.[4] Without a named pipe, loading a compressed file into MySQL would mean first writing out the whole uncompressed file, which takes time, more I/O and disk space.[4] PostgreSQL's command-line utility, psql, also supports loading data from named pipes.[4] In Talend, tNamedPipeOutput writes data into an existing open named pipe, which tNamedPipeOpen opens and tNamedPipeClose closes at the end of a process.[5]

UDP is connectionless, does not keep track of what it has sent, and gives no guarantee of delivery, ordering or duplicate protection.[6] If transmission reliability is wanted over UDP, it must be implemented in the user's application.[6]

The Transmission Control Protocol (TCP) provides reliable, ordered and error-checked delivery of a stream of bytes between applications on hosts communicating over an Internet Protocol (IP) network.[7] TCP is connection-oriented, and the server must be listening for connection requests from clients before a connection is established.[7]

Syslog is a standard for message logging, and over a network a syslog server listens for and logs messages coming from clients.[8] Historically the most common transport for network syslog has been UDP with the server on port 514, and because UDP lacks congestion control, TCP port 6514 is used.[8]

The method

In the Internet protocol suite, a socket is identified to other hosts by its socket address: the transport protocol, the IP address and the port number.[1] The term socket is also used for the endpoint of IPC within one node, which often uses the same application programming interface as a network socket.[1] In Talend, the tSocketInput component opens the socket port and listens for incoming data.[2]

On Unix, a named pipe is created with mkfifo() or mknod(); one process opens it for reading and another opens it for writing.[4] A traditional pipe is unnamed and lasts only as long as the process, whereas a named pipe can last as long as the system is up.[4] In the Windows operating system, a named pipe can be accessed much like a file.[4]

UDP was designed by David P. Reed in 1980 and formally defined in Request for Comments (RFC) 768.[6] Syslog was developed in the 1980s by Eric Allman as part of the Sendmail project.[8] The Internet Engineering Task Force documented the status quo of syslog in RFC 3164 in August 2001, and syslog was standardised by RFC 5424 in March 2009.[8] Apache NiFi's PutSyslog processor sends syslog messages to a given host and port over TCP or UDP, and checks each message it builds against the RFC 5424 and RFC 3164 formats.[9] In PutSyslog, a message that does not form a valid syslog message is routed as invalid, while valid messages are sent to the syslog server.[9]

Apache NiFi's SiteToSiteReportingRecordSink writes records over a Site-to-Site connection to a destination NiFi instance named by its URL.[10] In SiteToSiteReportingRecordSink, the Batch Size property sets how many records are sent in a single batch at most, with a default of 1000.[10]

Compared with the alternatives

UDP avoids the overhead of error checking and correction in the protocol stack.[6] Time-sensitive applications often use UDP because dropping packets is preferable to waiting for packets delayed by retransmission.[6] TCP's three-way handshake, retransmission and error detection add to reliability but lengthen latency.[7]

Sources

  1. [1]Network socketsecondary source
  2. [2]Socket components | Talend Components for Jobs Helpdocumentation
  3. [3]UDPEventRecordSink - Apache NiFidocumentation
  4. [4]Named pipesecondary source
  5. [5]NamedPipe components | Talend Components for Jobs Helpdocumentation
  6. [6]User Datagram Protocolsecondary source
  7. [7]Transmission Control Protocolsecondary source
  8. [8]Syslogsecondary source
  9. [9]PutSyslog - Apache NiFidocumentation
  10. [10]SiteToSiteReportingRecordSink - Apache NiFidocumentation