Tips and Tricks #218: Implement Idempotent ETL with Merge Statements
Use MERGE (upsert) for safe, rerunnable data pipelines that handle duplicates gracefully.
Designing intelligent systems, one layer at a time.
Use MERGE (upsert) for safe, rerunnable data pipelines that handle duplicates gracefully.
Three years ago, I inherited a distributed system that processed insurance claims across twelve microservices. The orchestration logic lived in a tangled web of message queues, retry handlers, and compensating…
Calculate running totals, rankings, and moving averages efficiently with SQL window functions.
Gradually replace legacy systems by routing traffic to new implementations incrementally.
Use domain events to decouple components and enable reactive architectures.
Separate read and write operations for better scalability and simpler code.