Is there a way to enforce FIFO for scheduled flow runs (firts scheduled, first to execute)? #19431
Unanswered
anton-daneyko-ultramarin
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Our research team uses Prefect to run long running (~hours) tasks. One of the recent complaints is that task submitted are not running in FIFO order. It was the case that one person submitted a bunch of workloads, then the other two did and the workloads of the first person are still in the queue while most of the workloads of the latter two are done. We do have several queues set up with different priorities, but all three people happen to use the same queue.
Some of our workloads run directly in K8s where we observe similar behavior. If Prefect could give us a reliable or close to reliable FIFO order (best effort), that would be a great argument in favor of migrating those workloads to Prefect.
Beta Was this translation helpful? Give feedback.
All reactions