The art of simplicity

Dimension tables don't stay still. A customer moves city, a product gets reclassified, a salesperson switches regions. The question is never whether this happens, it's what you do with the old value once it changes. That question has a name: Slowly Changing Dimensions (SCD). If you haven't already, it's worth reading my first post about what SCDs actually are and the full set of types first — this post assumes you already know the difference between Type 1 and Type 2 and want to get straight to implementing Type 2 in Fabric. Until recently, implementing Type 2 in Fabric meant either building a Dataflow Gen2 with a chain of merge steps, or writing a PySpark notebook against Delta tables. Both work. Both also mean you're maintaining custom logic per table, forever. Fabric's Copy job now has SCD Type 2 built in as a write method. No merge statements, no derived columns for surrogate keys, no alter-row logic. You pick a write method from a dropdown. This pos...

Being new to data warehousing, I never heard about the term "Slowly Changing Dimensions" before. It sounded like academic jargon. It isn't. It's one of those concepts that, once it clicks, explains half the weird design decisions you'll see in any reporting database. This post is the explainer I wish existed before I had to learn it the hard way: by breaking a report Facts and dimensions, the short version Data warehouses generally split data into two kinds of tables. Fact tables hold the things that happened: an order, a sale, a support ticket, a click. They're typically just numbers and foreign keys — quantity, amount, a reference to which customer, which product, which date. Dimension tables hold the descriptive context around those facts: who the customer is, what the product is called, which region a salesperson covers. A fact on its own is nearly meaningless. "47 units, customer 1182, product 309" tells you nothing until you jo...

Being new to Microsoft Fabric I noticed that you have multiple options when writing notebooks using Python: run your code with PySpark (backed by a Spark cluster) or with Python (running natively on the notebook's compute). Both options look almost identical on the surface — you're still writing Python syntax either way — but under the hood they behave very differently, and picking the wrong one can cost you time, money, and unnecessary complexity. In this post I try to identify the key differences and give you some heuristics for deciding which engine to reach for. Python vs PySpark: what's actually different? When you select PySpark in a Fabric notebook, your code runs on a distributed Apache Spark cluster. Fabric spins up a cluster, distributes your data across multiple worker nodes, and executes transformations in parallel. The core abstraction is the DataFrame (or RDD), and operations are lazy — nothing actually runs until you trigger an action like .show() ...

Yesterday I showed how we could query local parquet and delta files using pandas and deltalake. Although these libraries work, once you start loading big parquet files you see your system stall while your memory usage spikes. A colleague suggested to give DuckDB a try. I never heard about it, but let’s discover it together. What is DuckDB? DuckDB is an in-process analytical database — think SQLite, but built for OLAP workloads instead of transactional ones. It runs entirely inside your Python process (no server to spin up, no connection string to manage) and is optimized for the kinds of queries data engineers run every day: large scans, aggregations, joins, and window functions over columnar data. A few things that make it stand out: It reads files directly. You don't import data into DuckDB before querying it. You point it at a Parquet file, a folder of Parquet files, or a Delta table, and it queries them in place. No ETL step, no intermediate copy. It's c...

Last week I shared how you could use the  OneLake File Explorer to sync your Lakehouse tables to your local machine. It's a convenient way to get your Parquet and Delta Lake files off the cloud and onto disk — but what do you actually do with them once they're there? In this post, I’ll walk you through how to interact with your locally synced OneLake files using Python. We'll cover four practical approaches, with real code you can drop straight into a notebook. Where are your files? When OneLake File Explorer syncs your files, they land in a path that looks something like this: C:\Users\<you>\OneLake - <workspace name>\<lakehouse name>.Lakehouse\Tables\<table name> Keep that path in mind— you'll be passing it into every example below. Delta Lake tables are stored as folders containing multiple Parquet files plus a _delta_log/ directory, so make sure you're pointing at the table's root folder, not an individual file. Readin...

If you've spent any time working with Microsoft Fabric, you know that navigating to the web portal every time you need to inspect, upload, or tweak a file gets old fast. OneLake File Explorer is Microsoft's answer to that friction — a lightweight Windows application that mounts your entire Fabric data estate directly in Windows File Explorer, the same way OneDrive handles your documents. One..what? OneLake is the unified data lake underpinning every Microsoft Fabric tenant. Unlike traditional architectures where teams maintain separate data lakes per domain or business unit, every Fabric tenant gets exactly one OneLake — one place where Lakehouses, Warehouses, KQL databases, and other Fabric items store their data. There's no need to copy data between engines; Spark, SQL, and Power BI all read from the same underlying storage. The organizational hierarchy is straightforward: Tenant → Workspaces → Items (Lakehouses, Warehouses, etc.) → Files/Tables . This maps neat...

If you started using Microsoft Fabric, one of the first things you want to do is to get data in the platform. You could of course, start creating your own Data Factory pipelines, but there is a less complicated alternative to get started; the Microsoft Fabric's Copy Job feature. It offers a streamlined, no-code solution that eliminates the need for complex pipeline development. In this post, we'll explore what Copy Jobs are, why they matter, and how you can leverage them in your data workflows. What is a Copy Job? Copy Job is Microsoft Fabric Data Factory's answer to simplified data movement. It's a purpose-built solution designed to move data from various sources to multiple destinations without requiring you to build traditional data pipelines. Whether you're working with databases, cloud storage, or on-premises systems, Copy Job provides an intuitive, guided experience that handles the complexity for you. At its core, Copy Job addresses a common challenge: ...

The Microsoft Fabric Capacity Metrics app allows you to monitor your Microsoft Fabric capacities. You can use the app to monitor your capacity consumption and use these insights to decide when to scale (or setup autoscaling). After installing the Microsoft Fabric Capacity Metrics App , I noticed that no data was shown on the Health page: Clicking on the error details showed me the following info: The NoCapacitiesInRegion.Error message mentions that no capacity is available. So, let’s have a look at the assigned capacity for this workspace: The cause of this error is not related to having no capacity but to the fact that I had a Power BI Pro capacity assigned instead of a Fabric capacity. After changing to the Fabric capacity, the health page started to work as expected: Nice! More information What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft Learn Understanding Microsoft Fabric Capacity and Throttling–A first attempt

Last week I shared my understanding of how capacity works in Microsoft Fabric. I just hit publish when I got a message that I exceeded my Fabric capacity (again). Time to open the Fabrics Metrics dashboard I introduced in my last post about Fabric and see who is the culprit: I opened the Compute tab and hovered over the top item in the list.  Turns out that Copilot is eating up a lot of my capacity. Whoops! I decided to disable copilot for this data warehouse. Therefore, I opened up a query in the data warehouse and clicked on the Copilot completions item at the bottom of the screen: This opens the configuration settings for my data warehouse where I can disable the completions:   If you want to disable Copilot completely, you can do that at the tenant level :   Remark: If you are still confused on how Fabric capacity exactly works, I found this great post by Tom Keim where he compares CU usage to watts: Just like electricity, where we’re billed by k...

Being new to Microsoft Fabric, one of the topics that I found challenging, is how Fabric capacity and especially the throttling works. And what is a better way to structure my understanding than writing a blog post. Let’s give it a try! Remark : If I made some mistakes, please feel free to let me know so I can update this article. What is Microsoft Fabric Capacity? Microsoft Fabric capacity is the compute and storage resources you purchase to run Fabric workloads. Unlike traditional per-service pricing models, Fabric uses a unified capacity model where you purchase Capacity Units (CUs) that power all Fabric experiences including Data Engineering, Data Warehouse, Data Science, Real-Time Analytics, Power BI, and Data Factory. When you purchase a Fabric capacity, you're essentially reserving a pool of compute resources that can be shared across different workloads and users within your organization. This capacity is measured in Capacity Units, which represent the computational...