Rethinking ETL : Why We Migrated Our Legacy Pipelines to Apache Spark

Overview

Many of the enterprises were transforming their legacy data systems by shifting to modern ETL (Extract, Transform, Load) architectures powered by tools like Apache Spark for agility, performance and access to the broader eco systems. The same happened when we joined a client project at Stryv. Their existing data pipeline was a patchwork of scheduled cron jobs, manually written SQL queries, and a single, rigid monolithic data warehouse. Data refreshing would take hours, and any schema change led to manual patchwork across a spaghetti of workflows.

What could be the reason for migrating a legacy application? Their outdated data architecture depends heavily on on-premises systems and legacy warehouses that are costly to maintain and hinder agility. Typically, these systems operate on relational databases and work well for structured data but struggle for real-time demands or under the pressure of high-concurrency workloads. The more our data grew, the more things started to break. Load times dragged, maintenance piled up, and scaling became a guessing game. Our team was spending more time fixing than innovating. At that point, it was obvious; this wasn’t just an aging system. It was a risk we couldn’t afford to carry it forward.

Why Apache Spark? What We Evaluated

We considered traditional tools (like Informatica), open-source orchestrators (Airflow + SQL), and modern ELT approaches. But Apache Spark stood out for a few key reasons:

Apache Spark being an open source, distributed processing system preferably used for bigger data workloads. No matter the size, it can run fast analytic queries against data. As it supports a variety of languages including Java, Scala, Python, and R, making it developer friendly with minimal code. Apache Spark comes with the ability to run multiple workloads, including interactive queries, real-time analytics, machine learning, and graph processing. One application can combine multiple workloads seamlessly.

• • Distributed Computing: We needed something that could handle massive joins and transformations across partitions.
• • Unified Framework: With Spark, we can handle batch and streaming within a common codebase.
• • Strong Ecosystem: PySpark, Delta Lake, MLlib all well-documented and extensible.

We deployed Spark on AWS EMR initially and later transitioned to Databricks for tighter operational control.

What the Legacy ETL Migration Actually Looked Like

There’s a big gap between “let’s use Spark” and “Spark is running in production.” Migrating from a legacy ETL system is not a plug-and-play process it requires careful planning, deep auditing, and foundational rewrites. At Stryv, we were involved in breaking apart monolithic workflows, redesigning data logic in a distributed paradigm, and building a resilient, scalable ETL framework from the ground up. Here’s how we approached it, step by step: 

1. Audited Legacy Pipelines

Before building anything new, we had to understand what we already had. This meant diving deep into the existing spaghetti of pipelines.

• • Identified every input-output mapping.
• • Analysed dependencies and scheduling.
• • Categorized pipelines into critical, redundant, and deprecated.

2. Refactored Logic

Once the landscape was mapped, we began the process of rewriting existing logic into Spark-native constructs, optimizing scalability and modularity.

• • SQL joins were translated into PySpark Data Frame logic.
• • Reused transformation logic across jobs using custom Python modules.
• • Schema changes were handled with metadata-driven ingestion.

3. Built Reusable ETL Framework

To avoid duplicating work and make onboarding easier, we built a configurable ingestion engine with operational guardrails.

• • Created a configuration-first ingestion engine.
• • Implemented retry + alert logic using Airflow + Spark-submit.
• • Built lineage reporting using audit tables and Glue catalog integration.

4. Monitoring & Scaling

A performance pipeline is only as good as its observability. We invested early in benchmarking, tuning, and setting up real-time alerts.

• • Spark jobs were benchmarked using executor and memory stats.
• • Auto-scaling and pre-emption strategies were tested on EMR clusters.
• • Logs were pushed into centralized dashboards for anomaly detection.

What We Gained

• • 90% reduction in manual job management.
• • 5x faster data transformation jobs.
• • Schema changes now require no code edits, —just metadata updates.
• • Unified pipeline framework that scales across multiple domains.
• • 90% reduction in manual job management, thanks to automation and metadata-driven design.
• • 5x faster data transformation jobs with optimized Spark execution plans.
• • Schema changes now require no code edits, just metadata updates.
• • Unified pipeline framework that scales across multiple domains.
• • Faster onboarding for new engineers with reusable, config-first ETL modules.
• • Improved data quality and consistency through centralized auditing and lineage tracking.
• • Flexibility to run batch and streaming jobs under a single, cohesive architecture.

What We Wish We Knew Earlier

• • Spark Shuffle is your bottleneck. Optimize your partitions early.
• • PySpark is powerful, but memory leaks can creep in if you don’t manage joins and caching properly.
• • For small datasets, Spark can be overkill. Not everything needs a distributed computing environment.

Final Thoughts

Spark is not a silver bullet. But when used correctly, it’s a workhorse that replaces a dozen legacy tools. At Stryv, we now treat Spark as the engine behind every serious data workload whether it’s batch, streaming, or model training.

If you’re currently dealing with slow, brittle, manually maintained ETL pipelines, you’re not alone, and you’re not stuck. Legacy systems can quietly become technical debt that stalls progress, eats up engineering time, and limits your organization’s ability to scale. Modernizing your data stack isn’t just a technical upgrade it’s a strategic investment in agility, visibility, and long-term resilience.

It might be time to rethink your foundation. Not just for better performance, but to unlock the full potential of your data infrastructure. Need Help Migrating? Talk to our engineering team at Stryv. We do this on a scale.