Tags: databricks, delta lake, big data, incremental refresh, machine learning Incremental refresh This paper describes what I also called temporal join in the past. Suppose you have 2 big data tables (e.g. delta tables). Because the tables are so big, they are incrementally refreshed. This means that every day we add or change only the […]
Why is DWA framework x better than DWA framework y ? How do you compare DWA frameworks ? To answer these questions we first have to define what a DWA framework exactly is. This article will lay a foundation to compare DWA frameworks. A DWA framework or DWA software generates the code for building your […]
When working with ADF I frequently have to copy paste json code into sql server management studio ( ssms). Json uses \n for specifying a newline so this is shown in ssms as one large string which is difficult to read. Solution: Use the program autohotkey to record a hotkey / macro to replace some […]
The following link will explain the reasons for using infra as code: My recent setup involved: Azure data factory, SSIS, Azure SQL server, Azure dev-ops, key vault and ARM templates. GitI’ve made two different repositories. One for ADF and one for SSIS and SQL Server. Why? Reason 1 quicker building and deployingChanges to ADF are […]
