Posts

Showing posts with the label NumPy

Data Engineering Interview Preparation #3: Building a Financial Data Pipeline with Python, DuckDB and Parquet

Summary : Learn about building a repeatable financial data engineering pipeline using Python, Pandas, NumPy and DuckDB. This post covers deterministic data generation, CSV ingestion, SQL transformations, canonical schema design, data-quality checks, Parquet export, and CSV-versus-Parquet validation. Financial data engineering involves much more than loading records from a CSV file into a database. A reliable pipeline must handle source data consistently, maintain predictable schemas, validate records, and produce outputs that can be used confidently by downstream applications and analytical workflows. In this hands-on exercise, I created a small but practical financial data pipeline using Python, Pandas, NumPy and DuckDB. We start with supplied CSV files, inspect and clean the source data, transform raw trade records into a canonical structure, apply data-quality checks, and export validated data to Parquet. Run the hands-on notebook: DE Interview Prep #3: Financial Data Enginee...

You Do Not Really Know NumPy Until You Understand These Core Truths

Image
Summary : NumPy is the foundation of Python’s data science ecosystem, yet many Data Scientists and ML Engineers use it without understanding what makes it so powerful. This blog post explains core truths about NumPy that reveal why it is fast, memory-efficient, and essential for serious data work. Introduction: The Bedrock of Python Data Science If you work with data in Python, you have almost certainly used libraries like Pandas, Scikit-Learn, or TensorFlow. These tools power everything from data cleaning to machine learning. But have you ever stopped to think about what makes them so fast and efficient? At the foundation of this entire ecosystem is NumPy. Short for Numerical Python, NumPy is not just another library. It is the core engine that turned Python into a serious language for scientific computing. First view the NumPy tutorial for beginners. Then, read on. If you strip away the higher-level tools, you eventually reach NumPy. Understanding how it works chan...