BETL can be found on github here: https://github.com/basvdberg/BETL
View our video tutorial on youtube
BETL is a data warehouse automation framework. It allows you to generate ETL and DDL. Orchestration needs an ETL tool, but you can use the BETL meta data to build a generic orchestration process.
What is BETL
- Free, open source, GNU GPL
- Meta data driven
- 100 % T-SQL
- Generic (usable in many different data warehousing scenarios)
- Real time code generator (in contrast to design time code generators like e.g. BIML). Changes to meta data and templates are effective immediately. There is no need to build or deploy anything in your development environment.
What is BETL not
- A modeling method (e.g. Datavault, Kimball or Anchor modeling)
- A DWH architecture (e.g. staging-> raw dwh -> integrated dwh -> datamart ).
- Dependent ( e.g. dependent on vendor or tooling other than T-SQL and handle bars. no external libaries.).
- Restrictive ( all or nothing. You can choose which functionality you require in your project.)
- Visual (currently it’s only T-SQL. My next focus would be to build a meta data gui )
- Finished. Although I spent many hours on BETL, there are still some loose ends. You are very welcome to help.
Main reasons for using BETL
- Improve your productivity. However you need to invest some time to get to know BETL.
- Improve the quality of your ETL by seperating what to do and how to do it.
- Makes your work more fun. You will need to think about generic business rules and templates instead of implementing local ETL changes.
- It forces developers to folow certain design guidelines and best practices. E.g. naming conventions, change detection, logging, TSQL Batch insert performance, etc.
Pre requisits for using BETL
- T-SQL intermediate level.
- A general understanding of the Handlebars template language.
- You need a SQL Server database (in Azure or on premise).
Pre requisits for contributing to BETL
- T-SQL advanced level. Generate dynamic T-SQL using stored procedures, functions, custom data types.
- Knowledge of data warehousing best practices.
BETL can be used standalone or combined with for example Azure data factory or any other ETL tool.