BETL architecture

By seperating what needs to be done and how it is done we obtain a separation of concerns. You can easily change the how without affecting the what or vice versa.

ComponentDescriptionTooling/techniquePart of current BETL release?
ETL OrchestrationDefine jobs that run ETL task sequentially or in parallel. Schedule and run jobs.Custom build. Use the ADF triggering
Orchestration monitorMonitor running jobs and restart in case of an error. Also send notifications. Gather statistics.Custom build. Power-BIno
ETL ToolDefine and execute ETL tasks. Securely connect to source and target database and transfer data from source to targetAzure data factory (ADF)some demo pipelines.
ETL EngineGenerate ETL using meta data and templatesBETL for Azureyes
TemplatesDescribes for example how to update a historic table or create a ddl statement for creating a table. SQL code generation needs the engine meta data of courseHandlebarsyes
LoggingLog every step performed by the ETL engine, so that we can troubleshoot issues. Our logging requirements are extensive. loglevel can be set to verbose, debug, info, warning or errorBETL and Power-BIyes, but no Power-BI dashboards yet.
Meta dataAll meta data is stored into an (Azure) SQL server database and seperated into static tables and customer dependent tables (dbo schema).(Azure) SQL Databaseyes
Object treeThe object tree consists of server->database->schema->table or view->columns. We can ingest object trees using sql meta data queries that almost every database vendor supports(Azure) SQL Databaseyes
MappingsMapping specify how the transformation between source and target is done. mappings are related to one specific object(i.e. table) in the object tree(Azure) SQL databaseyes
Business rulesA mapping can refer to a business rule in order to describe a generic transformation (e.g. transform an integer into a date using specific rules or calculate a derived value)(Azure) SQL databaseyes
PropertiesProperties are linked to objects in the object tree (e.g. a schema or specific table). Properties can be used for specifying you data warehouse architecture (e.g. which template to use for building tables in your raw dwh).(Azure) SQL databaseyes