By seperating what needs to be done and how it is done we obtain a separation of concerns. You can easily change the how without affecting the what or vice versa.
| Component | Description | Tooling/technique | Part of current BETL release? |
|---|---|---|---|
| ETL Orchestration | Define jobs that run ETL task sequentially or in parallel. Schedule and run jobs. | Custom build. Use the ADF triggering mechanism. | no |
| Orchestration monitor | Monitor running jobs and restart in case of an error. Also send notifications. Gather statistics. | Custom build. Power-BI | no |
| ETL Tool | Define and execute ETL tasks. Securely connect to source and target database and transfer data from source to target | Azure data factory (ADF) | some demo pipelines. |
| ETL Engine | Generate ETL using meta data and templates | BETL for Azure | yes |
| Templates | Describes for example how to update a historic table or create a ddl statement for creating a table. SQL code generation needs the engine meta data of course | Handlebars | yes |
| Logging | Log every step performed by the ETL engine, so that we can troubleshoot issues. Our logging requirements are extensive. loglevel can be set to verbose, debug, info, warning or error | BETL and Power-BI | yes, but no Power-BI dashboards yet. |
| Meta data | All meta data is stored into an (Azure) SQL server database and seperated into static tables and customer dependent tables (dbo schema). | (Azure) SQL Database | yes |
| Object tree | The object tree consists of server->database->schema->table or view->columns. We can ingest object trees using sql meta data queries that almost every database vendor supports | (Azure) SQL Database | yes |
| Mappings | Mapping specify how the transformation between source and target is done. mappings are related to one specific object(i.e. table) in the object tree | (Azure) SQL database | yes |
| Business rules | A mapping can refer to a business rule in order to describe a generic transformation (e.g. transform an integer into a date using specific rules or calculate a derived value) | (Azure) SQL database | yes |
| Properties | Properties are linked to objects in the object tree (e.g. a schema or specific table). Properties can be used for specifying you data warehouse architecture (e.g. which template to use for building tables in your raw dwh). | (Azure) SQL database | yes |