Running a 2.5 hour job from a 60 minute pipeline: Azure Container Apps Jobs, kickoff and collect

I have an import job that takes more than 2.5 hours. It cleans up and re-imports a legacy dataset into Cosmos DB and Azure SQL. Azure DevOps hosted agents give a job 60 minutes, and even if they gave more, I don’t want an agent sitting idle for hours doing nothing but polling. So I split the work into two pipelines and let an Azure Container Apps Job do the actual lifting in between. This post is about that pattern, why the first version didn’t survive, and the details that made the second version work.

The workload

The job is a one-time migration from a WordPress based system into a new multi-tenant platform, the same project I described in my Claude Code and Azure DevOps post. The importer is a plain .NET console app. I considered a Python script, driving the import through the public API, and Azure Data Factory, and rejected all three: the console app reuses the existing infrastructure code for Cosmos and SQL, is type safe, and can run inside the VNet next to the databases.

How I use Claude Code with Azure DevOps

Optimize enterprise development with Claude Code and Azure DevOps. Use custom agents, rules, and skills to streamline your .NET, Angular, and Bicep monorepo.

Dariusz Parys

The numbers give a feel for the runtime:

The run goes through six stages: a preflight that validates and fingerprints the source files, a cleanup that deletes the previous import from Cosmos and then from SQL, the SQL aggregates written in one transaction per user, the measurements written to Cosmos with Change Feed syncing them to SQL, an Entra preflight that looks up every candidate email in Microsoft Graph, and finally the Entra mutation that creates a disabled account for each user and remaps the login identity in SQL.

That last pair of stages is what pushed the runtime past any hosted agent limit. Every user means at least one Graph call, and Graph isn’t in a hurry.

Version one: start the job and wait

The first pipeline did the obvious thing. It built the image, updated the Container Apps Job, started an execution, and then polled:

$maxAttempts = 720
$sleepSeconds = 5

for ($i = 1; $i -le $maxAttempts; $i++) {
  $execution = az containerapp job execution show `
    --name $jobName `
    --resource-group $resourceGroup `
    --job-execution-name $executionName `
    --output json | ConvertFrom-Json

  $status = $execution.properties.status
  if ($status -eq "Succeeded") { exit 0 }
  if ($status -eq "Failed") { exit 1 }
  Start-Sleep -Seconds $sleepSeconds
}
Write-Host "##vso[task.logissue type=error]Timeout waiting for job completion"
exit 1

720 attempts at 5 seconds is exactly 60 minutes. On success or failure it pulled the container output with az containerapp job logs show --execution. The job itself had a replica timeout of 30 minutes, which I raised to 50 minutes in January when the measurement volume grew.

This worked as long as the import fit in the hour. When the Entra stage arrived in March, the job needed a 3 hour timeout, and the pipeline needed a different shape.

Version two: kickoff and collect

The idea is simple. One pipeline starts the job and hands you an identifier. A second pipeline takes that identifier, asks Azure how the execution went, pulls the logs, and turns the result into a pipeline outcome you can read and gate on. No agent is blocked while the job runs.

sequenceDiagram
    participant K as Kickoff pipeline
    participant J as Container Apps Job
    participant L as Log Analytics
    participant C as Collect pipeline
    K->>J: job update (env vars), job start
    J-->>K: execution name
    K->>K: publish artifact with execution name
    Note over J: runs for 2.5 hours, no agent waiting
    J->>L: console and system logs
    C->>J: job execution show
    J-->>C: status, start time, end time
    C->>L: KQL query over the execution window
    L-->>C: console and system logs
    C->>C: report, summary, exit code

The job itself

The Container Apps Job is defined in Bicep with a manual trigger, parallelism of one, a completion count of one, and a replica timeout of 10,800 seconds. The retry limit is zero. If the import fails I want to look at it, not have the platform quietly run it again. It gets 1 vCPU and 2 GiB of memory.

A single user assigned managed identity handles everything: pulling the image from the container registry, talking to Cosmos and SQL, and reading the source CSVs from a blob container. The Bicep pins the image to latest, and the pipeline swaps in a build specific tag before each run.

One property on the Container Apps Environment matters more than everything else in this post:

appLogsConfiguration: {
  destination: 'log-analytics'
  logAnalyticsConfiguration: {
    customerId: workspace.properties.customerId
    sharedKey: workspace.listKeys().primarySharedKey
  }
}

That’s what makes the ContainerAppConsoleLogs and ContainerAppSystemLogs tables exist in Log Analytics. Without it, the second pipeline has nothing to query.

Kickoff

The kickoff pipeline has two jobs. The first one optionally builds and pushes the image, tagged with the build id. The second one prepares and starts the Container Apps Job.

Parameters travel as environment variables. The pipeline calls az containerapp job update --set-env-vars with the dry run flag, the stage filter, the deployment environment, and a couple of safety switches. Then it verifies the job exists and starts it:

$startResult = az containerapp job start `
  --name $jobName `
  --resource-group $resourceGroup `
  --output json | ConvertFrom-Json

$executionName = $startResult.name

The execution name is the whole handshake. It looks like the job name with a random suffix, and it’s the only thing the second pipeline needs. The kickoff writes it into a small JSON file together with the parameters and a UTC start timestamp, and publishes that as a pipeline artifact. The whole pipeline finishes in a few minutes.

One thing to be aware of: job update --set-env-vars changes the job definition, not the execution. If you kick off two runs with different parameters in quick succession, the second update can land before the first execution has read its configuration. For a manually triggered migration job that is acceptable. For anything that runs concurrently it isn’t, and you would want to pass parameters as container arguments on start instead.

Collect

The collect pipeline takes the environment and the execution name as inputs. It starts by asking for the execution metadata:

$execution = az containerapp job execution show `
  --name $jobName `
  --resource-group $resourceGroup `
  --job-execution-name $executionName `
  --output json | ConvertFrom-Json

$status = $execution.properties.status
$startTime = $execution.properties.startTime
$endTime = $execution.properties.endTime   # null while still running

If the job is still running, the end time falls back to now, and the pipeline collects whatever logs exist so far.

Then it resolves the Log Analytics workspace. The resource id comes from the platform deployment outputs, and az monitor log-analytics workspace show --query customerId turns it into the workspace id that the query command wants.

Why Log Analytics and not the CLI log command

In version one I used az containerapp job logs show. That command streams from a live or recently finished replica. When the collect pipeline runs hours later, on a fresh agent, the replica is long gone and there’s nothing to stream from. Log Analytics is the only durable sink for container output, the environment is already wired to it, and the logs carry the job name and the execution name as columns. So the second pipeline queries it.

The console query looks like this:

let executionName = '<execution name>';
let jobName = '<job name>';
let startTime = todatetime('<execution start>');
let endTime = todatetime('<execution end>');
union isfuzzy=true ContainerAppConsoleLogs, ContainerAppConsoleLogs_CL
| extend JobNameValue = coalesce(
    tostring(column_ifexists('JobName', '')),
    tostring(column_ifexists('JobName_s', '')))
| extend ContainerGroupNameValue = coalesce(
    tostring(column_ifexists('ContainerGroupName', '')),
    tostring(column_ifexists('ContainerGroupName_g', '')))
| extend StreamValue = coalesce(
    tostring(column_ifexists('Stream', '')),
    tostring(column_ifexists('Stream_s', '')))
| extend LogValue = coalesce(
    tostring(column_ifexists('Log', '')),
    tostring(column_ifexists('Log_s', '')))
| where TimeGenerated between (startTime - 15m .. endTime + 15m)
| where JobNameValue == jobName or ContainerGroupNameValue contains executionName
| order by TimeGenerated asc
| project strcat(format_datetime(TimeGenerated, 'yyyy-MM-dd HH:mm:ss.fff'), 'Z ',
    iff(isempty(StreamValue), '', strcat('[', StreamValue, '] ')), LogValue)

Three details in there earned their place:

• union isfuzzy=true over both tables. Container Apps logs exist either in the Azure managed ContainerAppConsoleLogs table or in the older custom log table with the _CL suffix, depending on when and how the environment was set up. Fuzzy union means the query doesn’t fail if one of them doesn’t exist.
• column_ifexists for every column. The custom log schema suffixes columns with their type, so JobName becomes JobName_s. Coalescing both spellings makes the same query work against either schema.
• A 15 minute pad around the execution window. Ingestion into Log Analytics lags, and the first log line doesn’t arrive at the exact execution start time. 15 minutes on both sides has been enough.

The query is executed with az monitor log-analytics query --timespan P3D. A second, near identical query pulls the system logs, which is where you see image pulls, replica scheduling and the timeout, if there was one.

A failed query is a warning, not a failure. The report then says that logs could not be retrieved, but the pipeline still produces the execution status.

Turning it into a pipeline result

The collect pipeline writes three files and publishes them as an artifact with condition: always(): the raw execution JSON, a report with the metadata followed by console and system logs, and a short summary. The import tool itself prints its full report to stdout when the pipeline asks it to, so the report ends up inside the console logs and comes along for free. Blob upload of the report is still in the code as a fallback.

The last step maps the execution status onto the pipeline outcome:

switch ($status) {
  "Succeeded" { exit 0 }
  "Failed" {
    Write-Host "##vso[task.logissue type=error]Legacy import execution failed."
    exit 1
  }
  default {
    Write-Host "##vso[task.logissue type=warning]Execution not finished yet (status: $status)."
    exit 0
  }
}

That gives you a pipeline you can re-run until it goes green, and one that downstream stages can depend on.

What to keep in mind

• The execution name is the contract. Store it somewhere you will find it again. The kickoff artifact is one option, a work item comment is another.
• Collect within the retention window. The query uses a 3 day timespan, and the workspace uses the default 30 day retention. Neither is a problem for a manual migration, but both are worth knowing.
• Give the job headroom. The replica timeout is 3 hours against an observed 2.5. A 20 percent margin has been enough so far.
• No retries for non trivial jobs. A failed import that silently restarts is worse than a failed import. Zero retries and a manual re-run keep a human in the loop.
• The pattern isn’t specific to imports. Anything that outgrows a hosted agent and needs to run inside your network fits: schema migrations, backfills, reconciliation runs. Kick it off, get the execution name, collect later.

The hosted agent limit turned out to be a useful constraint. It forced the long running work into a place that already had the network access, the identity, and the durable logging it needed, and left the pipelines with the two jobs they are good at: starting things and reporting on them.