Control / exec · All operations
Run a job on a compute platform
A variant of Run an external command
Start a Spark, Hadoop, notebook, function or cloud service job on another platform, and wait for or poll its result.
Status
- In arcform
Not built yet
arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Apache Hop (2)
Databricks job runread 2026-09-27Start or submit a Databricks job run and wait for it to finish
Databricks job waitread 2026-09-27Poll an existing Databricks job run until it completes
Apache NiFi (2)
Cloud AI service job processors (family)read 2026-09-2712 processors, start and poll jobs on AWS Polly, Textract, Transcribe, Translate and Google Vision
PutLambdaread 2026-09-27Invoke an AWS Lambda function with the FlowFile's JSON content
Azure Data Factory (12)
Azure Functionread 2026-09-27Run an Azure Function from a pipeline
Databricks Jar Activityread 2026-09-27Run a Spark Jar on an Azure Databricks cluster
Databricks Job activityread 2026-09-27Run a Databricks job, including serverless jobs, in an Azure Databricks workspace
Databricks Notebookread 2026-09-27Run a Databricks notebook in an Azure Databricks workspace
Databricks Python Activityread 2026-09-27Run a Python file on an Azure Databricks cluster
Hadoop Streamingread 2026-09-27Run Hadoop Streaming programs on an HDInsight cluster
Hiveread 2026-09-27Run Hive queries on an HDInsight cluster
MapReduceread 2026-09-27Run MapReduce programs on an HDInsight cluster
Pigread 2026-09-27Run Pig queries on an HDInsight cluster
Sparkread 2026-09-27Run Spark programs on an HDInsight cluster
Synapse Notebook Activityread 2026-09-27Run a Synapse notebook in an Azure Synapse workspace
Synapse Spark job definition activityread 2026-09-27Run a Synapse Spark job definition in an Azure Synapse workspace
Bruin (1)
emr_serverless.sparkread 2026-09-27Trigger an existing PySpark script or JAR as an EMR Serverless job
RapidMiner Studio (2)
Call Endpoint(RTSA)read 2026-09-27Call the endpoint of a process deployed on a Real-Time Scoring Agent and return its result table
Check Job Status (OpenAI)read 2026-09-27Check the status of an OpenAI fine-tuning job
Background
Apache Hop's Databricks job run action triggers a job on a Databricks workspace through the Jobs REST API and can wait until the run finishes.[1] In Azure Data Factory, the Azure Databricks Job activity runs Databricks jobs, serverless jobs included, in an Azure Databricks workspace.[2] Azure Databricks is a managed platform for running Apache Spark.[2]
When to use it
Apache Hop's Databricks job run action can wait for completion, polling until the run is terminal or a timeout is reached, or fire and forget, returning as soon as the run is accepted.[1] In fire and forget mode, Apache Hop's Databricks job run action is paired with a Databricks job wait action later in the workflow, using the run's ID.[1] Apache Hop's Databricks job run action stores the job ID, run ID and status in workflow variables for downstream actions.[1] Fire and forget suits a run that other work can overlap, but whichever mode is used, something later in the pipeline should read the run's final state before its output is trusted. (Meridian's view)
Databricks has no dedicated wait endpoint, so Apache Hop's Databricks job wait action repeatedly requests the run's status, sleeps for the poll interval, and exits when the run finishes or the timeout is hit.[3] Apache Hop's Databricks job wait action has a default poll interval of 15 seconds, and a timeout of 0 means no timeout.[3] If the parent workflow is stopped while Apache Hop's Databricks job run action is waiting, the action attempts to cancel the Databricks run.[1] Decide whether stopping the pipeline should also cancel the remote run, and set the step to match. (Meridian's view)
Called from Azure Data Factory's Azure Function activity, Azure Functions time out after 230 seconds, regardless of the functionTimeout setting configured.[4] A longer-running Azure Function called from Azure Data Factory can follow an async pattern or use Durable Functions, which offer their own state-tracking mechanism.[4] A Durable Function's statusQueryGetUri returns HTTP status 202 while the function is running, so Azure Data Factory can poll the function's status with a Web Activity.[4]
The method
A job scheduler is a computer application for controlling unattended background program execution of jobs, commonly called batch scheduling.[5] Basic features expected of job scheduler software include interfaces to define workflows or job dependencies, automatic submission of executions, and interfaces to monitor the executions.[5] Polling is actively sampling the status of an external device by a client program as a synchronous activity.[6] In busy-wait polling, the computer does nothing other than check the status of the device until it is ready.[6] Polling also refers to checking a device repeatedly for readiness and, if it is not ready, returning to a different task, which is not as wasteful of CPU cycles as busy waiting.[6]
Apache Hop's Databricks job run action can run an existing job by its job ID, submit a one-time run from a raw JSON body, or deploy files and then create or update a job and run it.[1]
Compared with the alternatives
Job scheduling should not be confused with process scheduling, which is the assignment of currently running processes to CPUs by the operating system.[5]
Sources
- [1]Databricks job run | Apache Hopdocumentation
- [2]Transform data with Databricks Job - Azure Data Factory & Azure Synapse | Microsoft Learndocumentation
- [3]Databricks job wait | Apache Hopdocumentation
- [4]Azure Function Activity - Azure Data Factory & Azure Synapse | Microsoft Learndocumentation
- [5]Job scheduler - Wikipediasecondary source
- [6]Polling (computer science) - Wikipediasecondary source