Migrating Alteryx workflows to PySpark addresses the workflow conversion itself.
But what happens after the workflows leave Alteryx Server?
The organization still needs a way to schedule workflows, start jobs, prevent duplicate runs, monitor failures, and manage day-to-day operations.
For a production migration involving 48 Alteryx workflows and macros, the target environment used Amazon EMR Serverless with a shared execution pattern to handle these operational requirements.
The result was a cloud-based operating model that let the migrated workflows run without relying on Alteryx Server for scheduling and execution.
Alteryx Server and its Gallery provide more than a place to run workflows. They also support scheduled execution and operational management. Teams often stop noticing this because the schedule simply works and the failed-run email simply arrives. Even well-run estates built on Alteryx ETL best practices miss it when they plan a move.
When workflows move to PySpark, you have to rebuild each of those quiet responsibilities in the target environment. A good migration plan asks these questions early. How will workflows be scheduled and started? How will duplicate executions be prevented? How will failures be reported and workflows rerun? Who will manage the environment after go-live? For a larger migration, these decisions are part of the architecture, not an afterthought.
What this means for you: List what your team relies on Alteryx Server to do today, including the things nobody wrote down, before you design the replacement.
| Job | The Question to Answer | How This Migration Handled It |
|---|---|---|
| Scheduling | How does each workflow start on its required schedule? | Amazon EventBridge triggers each workflow on its own schedule |
| Execution | Where does the PySpark code run? | Amazon EMR Serverless, controlled by AWS Step Functions |
| Run control | How are duplicate or overlapping runs handled? | Workflow-level locks in Amazon DynamoDB |
| Failure management | How does the team find out when a workflow fails? | Failure notifications through Amazon SES |
| Reruns | How are failed workflows investigated and restarted? | Documented launch and troubleshooting procedures |
| Ownership | Who runs the environment after go-live? | The internal team took over, using the handover documentation |
| Future workflows | How are new workflows added without a new process? | A shared dispatcher and a common workflow package pattern |
These questions matter as much as the conversion itself, so assign an owner to each one before the first workflow goes live.
For this production migration, we deployed the PySpark workflows to Amazon EMR Serverless, and surrounding AWS services provided scheduling and controls. Amazon EventBridge triggers the schedules. AWS Step Functions controls execution. EMR Serverless does the processing. Amazon S3 stores workflow outputs, and Amazon DynamoDB holds run controls. Amazon SES sends failure notifications, and AWS Glue Connections handle database connectivity.
None of this is exotic. AWS documents the same pattern of EventBridge scheduling a Step Functions workflow that runs Spark on EMR, and Step Functions has a direct integration for starting EMR Serverless job runs. This migration added a deliberate operating model on top of it. This separates the schedule from the processing, so the processing can move to EMR Serverless while workflows keep their scheduled cycles. For similar work on AWS, see our Alteryx to AWS Glue and Alteryx vs AWS Glue articles. Outputs can land in a data lake. The same operating questions apply to other targets. See our Alteryx vs Databricks comparison, our Databricks migration services overview, and our Microsoft Fabric data pipeline migration article.
What this means for you: The services are the easy part. The design of who triggers what, and what happens when something goes wrong, is where the effort lies.
With dozens of production workflows, the goal was not to invent a separate operating model for each one. Instead, the migration used a shared execution pattern. Each workflow kept its own schedule, but all of them followed a common process for starting execution, controlling concurrent runs, handling failures, sending alerts, and managing the workflow package.
A central Python dispatcher was the entry point. It receives the workflow to be run and routes the request to the matching PySpark workflow, so there is one common execution path. Run control, failure handling, alerting, and shared logic live in one place, and you maintain them once. That cuts duplicated effort, and it keeps behavior consistent when something changes. Teams building this kind of framework in Python can draw on our Python ETL automation article, and our ETL migration solution overview shows where it fits in a wider program.
Scheduled production workflows can be triggered again before the previous run has finished. For workflows that must not overlap, this can cause data and processing problems.
The migration used workflow-level run locks in Amazon DynamoDB. Before a workflow runs, it obtains its lock. If another run of the same workflow is already active, the second run does not proceed at the same time. When the workflow finishes, it releases the lock, and cleanup keeps the process consistent across all migrated workflows. Because locks can occasionally be left behind after an unexpected failure, the handover documentation includes run-lock recovery, which is the kind of detail that decides whether operations stay calm at 3 AM.
What this means for you: Decide which workflows can safely run twice, and lock the ones that cannot.
A production environment needs a clear way to see failures. This design used Amazon SES, so the relevant operations team could receive an alert and investigate. Centralizing the behavior also meant every workflow followed the same failure-notification process, with no separate alerting logic to write and maintain for each one.
Scheduling followed the same spirit. Each migrated workflow kept its scheduling requirement. EventBridge started the workflows on their cycles, and Step Functions provided the control layer, so the schedule and the PySpark processing environment were no longer tied together.
A shared pattern has to work for small and large workflows alike. The migration included one of the most complex workflows, with 1,157 tools. The largest individual input held about 432 million rows per day, and the largest workflow processed more than 400 million input rows in a full run. All of it ran inside the same operating model, which is a better test of a shared design than a handful of small jobs.
Output checks matter just as much at this scale, and our automated ETL testing article describes how to prove the migrated workflows produce the same results.
Replacing Alteryx Server does not mean switching it off on day one. Organizations often need a transition period to validate workflows, hand over production operations, and clear any remaining dependencies.
In this migration, the internal team took over the target environment, including schedules, workflow execution, run controls, and alerts. The organization could then downsize or retire its Alteryx Server environment on its own timeline. That is a more practical path than forcing the old and new environments to change at the same moment. If reports or dashboards depend on the old schedules, plan their move at the same time, as our reports conversion service does.
A migration is not complete just because the PySpark workflows run. The operating team has to know how to manage the environment after go-live. The project included operational documentation covering workflow launches, bundle updates, run-lock recovery, failure troubleshooting, and general production operations. That documentation is what moved day-to-day ownership to the internal team.
What this means for you: Write the runbook while the migration team is still in the room, not after they leave.
This section is planning guidance, not a statement of DataTerrain’s authorization status or government work. Regulated and public sector teams usually add requirements around who can run, change, and see production jobs:
The costly mistake is treating the conversion as the whole project, then discovering at go-live that nobody owns scheduling, locking, or alerts. DataTerrain brings 17+ years and 400+ clients in data and reporting modernization, with workflow conversion, validation, operating-model design, and documented handover planned together.
Request a free ETL migration assessment
Alteryx to Microsoft Fabric Migration | Microsoft Fabric Migration Services | Traditional vs Modern ETL | Data Warehouse to Databricks | Legacy Systems to Databricks | Legacy Reports Customer Stories