Posts

Data Engineering Interview Preparation #4: Modular Transformations and Data Quality Testing with Python and DuckDB

Summary : Learn how to build maintainable data transformation workflows with Python, Pandas and DuckDB. This hands-on Data Engineering post covers composable functions, separation of transformation logic from orchestration, avoiding hidden state, schema standardization, SQL transformations, reference-data joins, deterministic calculations, unit testing and data-quality validation. Data engineering involves more than loading data from source systems and storing it in a database. A maintainable pipeline also needs well-structured transformation logic, predictable outputs, clear data contracts and tests that help detect problems early. In this hands-on exercise, we explore how to design smaller, reusable and testable transformations using Python, Pandas and DuckDB. We start with simple transaction calculations, move into schema standardization and SQL transformations, and then build deterministic functions and unit tests to validate expected behaviour. Run the hands-on Kaggle notebo...

Data Engineering Interview Preparation #3: Building a Financial Data Pipeline with Python, DuckDB and Parquet

Summary : Learn about building a repeatable financial data engineering pipeline using Python, Pandas, NumPy and DuckDB. This post covers deterministic data generation, CSV ingestion, SQL transformations, canonical schema design, data-quality checks, Parquet export, and CSV-versus-Parquet validation. Financial data engineering involves much more than loading records from a CSV file into a database. A reliable pipeline must handle source data consistently, maintain predictable schemas, validate records, and produce outputs that can be used confidently by downstream applications and analytical workflows. In this hands-on exercise, I created a small but practical financial data pipeline using Python, Pandas, NumPy and DuckDB. We start with supplied CSV files, inspect and clean the source data, transform raw trade records into a canonical structure, apply data-quality checks, and export validated data to Parquet. Run the hands-on notebook: DE Interview Prep #3: Financial Data Enginee...

Data Engineering Interview Questions and Answers: Part 1 | Python, Pandas, SQL and DuckDB

Summary : Prepare for data engineering interviews with practical questions and answers covering Python, Pandas, DuckDB SQL, data validation, aggregations, window functions, and reproducible data pipelines. This Part 1 guide includes two hands-on coding challenges: finding the top three users by monthly spending and verifying that Pandas and SQL return equivalent results. Introduction Preparing for a data engineering interview involves more than writing code that produces the correct output. You should also be able to explain your approach, validate assumptions, identify edge cases, and demonstrate that your results are reliable. This post will test essential data engineering concepts through practical questions and coding exercises. You will explore reusable Python functions, data-quality checks, Pandas transformations, DuckDB SQL queries, aggregations, and window functions. You will also solve two hands-on problems. The first identifies the top three users by total spending fo...