Tag: data-engineering

Implement semantic search using text embeddings for more relevant results than keyword matching.

Build modular, tested, documented data transformations with dbt.

Use table partitioning to dramatically speed up queries on large datasets.

Use MERGE (upsert) for safe, rerunnable data pipelines that handle duplicates gracefully.

Eliminate heap allocations when parsing strings by using Span for memory-efficient operations.

Handle rate limits and transient failures gracefully with exponential backoff.