Infra as code in Azure

The following link will explain the reasons for using infra as code:

Why Infrastructure As Code?

My recent setup involved: Azure data factory, SSIS, Azure SQL server, Azure dev-ops, key vault and ARM templates.

Git
I’ve made two different repositories. One for ADF and one for SSIS and SQL Server. Why?
Reason 1 quicker building and deploying
Changes to ADF are build and deployed independently of SSIS and Sql server. This is quicker.
Reason 2 less pulling Changes in ADF don’t require a pull in visual studio.
Reason 3 cleaner repo ADF has it’s own feature, developer, main and release branches.

Dev-ops variable libary
We have chosen to store secrets in the Azure devops libary of variables. This way we have 1 central location to manage secrets. From here we deploy the secrets in a seperate key vault for every DTAP environment.

Build pipeline for ADF

Variable groups

# https://aka.ms/yaml

trigger:
- Release/*
- develop-adf
#- main

pool:
  vmImage: 'ubuntu-latest'

steps:
# first we will recreate the required resources (adf and keyvault) so that everything is created from azure dev ops. 
#- script: echo pipeline.workspace! 
#  displayName: 'Run a one-line script'
#- script: 'dir $(pipeline.workspace)'
#- script: 'dir $(Agent.BuildDirectory)'
#- script: 'dir $(Agent.BuildDirectory)\s'
#- script: 'dir $(Agent.BuildDirectory)\s'
#- script: 'dir $(Agent.BuildDirectory)\s\*.*'

# Installs Node and the npm packages saved in your package.json file in the build
- task: NodeTool@0
  inputs:
    versionSpec: '10.x'
  displayName: 'Install Node.js'

- task: Npm@1
  inputs:
    command: 'install'
    workingDir: '$(Build.SourcesDirectory)'
    verbose: true
  displayName: 'Install npm package'

# Validates all of the Data Factory resources in the repository. You'll get the same validation errors as when "Validate All" is selected.
# Enter the appropriate subscription and name for the source factory.

#- task: Npm@1
#  inputs:
#    command: 'custom'
#    workingDir: '$(Build.SourcesDirectory)' #replace with the package.json folder
#    customCommand: 'run build validate $(Build.SourcesDirectory) /subscriptions/<subscription-id>/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/<adf-name>'
#  displayName: 'Validate'

# Validate and then generate the ARM template into the destination folder, which is the same as selecting "Publish" from the UX.
# The ARM template generated isn't published to the live version of the factory. Deployment should be done by using a CI/CD pipeline. 

- task: Npm@1
  inputs:
    command: 'custom'
    workingDir: '$(Build.SourcesDirectory)' #replace with the package.json folder
    customCommand: 'run build export $(Build.SourcesDirectory) /subscriptions/<subscription-id>/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/<adf-name>"ArmTemplate"'
  displayName: 'Validate and Generate ARM template'


- task: CopyFiles@2
  inputs:
    sourceFolder: '$(Build.SourcesDirectory)/dev-ops'
    contents: '**' 
    targetFolder: '$(Build.SourcesDirectory)/ArmTemplate/dev-ops'
    #cleanTargetFolder: false # Optional
    #overWrite: false # Optional
    #flattenFolders: false # Optional
    #preserveTimestamp: false # Optional

# Publish the artifact to be used as a source for a release pipeline.
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.SourcesDirectory)/ArmTemplate' 
    artifact: 'ArmTemplates'
    publishLocation: 'pipeline'
   
  

• Make sure that you specify a secret only once. Hence I made a group for DTA and a group for DTAP. For example because DTA contains the reference to the service connection which is different for production.
• you can use variables inside other variable expressions. I have a variable called env which is used for building the generic names for resources. For example sql server: ms-sqls-dwh-$(env).database.windows.net ( <cloud environment><resource type><application name><environment name>)

Deploy ADF pipeline

Deploy resources ADF and Keyvault. This uses an ARM template that is build and tested in Visual studio using the ARM Extension. ( You can validate the template and deploy it from Visual studio). In Dev-ops the parameters are replaced by using the Override template parameters option. This ARM template is placed in the same branch as the ADF pipelines (folder dev-ops) and there is copy step in the build pipeline to copy the ARM template into the build artifact.

-adfName "$(adfName)" -keyVaultName "$(keyVaultName)" -connectionStringSqldbConf "$(connectionStringSqldbConf)" -connectionStringSqldbDwh "$(connectionStringSqldbDwh)" -connectionStringSqldbDM "$(connectionStringSqldbDM)" -connectionStringSqldbSTA "$(connectionStringSqldbSTA)"

This is what the ARM template file looks like ( azuredeploy.json )

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "adfName": {
      "type": "string"
    },

    "env": {
      "type": "string"
    },

    "keyVaultName": {
      "type": "string"
    },

    "keyVaultAdminObjectId": {
      "type": "string"
    },

    "connectionStringSqldbBetl": {
      "type": "string"
    },

    "connectionStringSqldbAW": {
      "type": "string"
    },

    "connectionStringSqldbRDW": {
      "type": "string"
    },

    "gitAccountName": {
      "type": "String"
    },
    "gitRepositoryName": {
      "type": "String"
    },
    "gitCollaborationBranch": {
      "type": "String"
    },
    "gitRootFolder": {
      //      "defaultValue": "/",
      "type": "String"
    },
    "gitProjectName": {
      "type": "String"
    }

  },
  "variables": {
    "location": "westeurope",
    "fullAdfName": "[concat('Microsoft.DataFactory/factories/', parameters('adfName'))]",
    "fullKeyVaultName": "[concat('Microsoft.KeyVault/vaults/', parameters('keyVaultName'))]",
    "tenantId": "[subscription().tenantId]",
    "repoConfigurationGit": {
      "type": "FactoryVSTSConfiguration",
      "accountName": "[parameters('gitAccountName')]",
      "repositoryName": "[parameters('gitRepositoryName')]",
      "collaborationBranch": "[parameters('gitCollaborationBranch')]",
      "rootFolder": "[parameters('gitRootFolder')]",
      "projectName": "[parameters('gitProjectName')]"
    },
    // only setup git for DEV!
    "repoConf": "[if(equals(toUpper(parameters('env')),'DEV'), variables('repoConfigurationGit'), '')]"

  },
  "resources": [
    { //Data Factory 
      "name": "[parameters('adfName')]",
      "apiVersion": "2018-06-01",
      "type": "Microsoft.DataFactory/factories",
      "properties": {
        "repoConfiguration": "[variables('repoConf')]",
        "globalParameters": {
          "Environment": {
            "type": "string",
            "value": "[parameters('env')]"
          }
        }

      },
      "location": "[variables('location')]",
      "identity": {
        "type": "SystemAssigned"
      },
      "tags": {}
    },

    { // keyVault
      "apiVersion": "2016-10-01",
      "type": "Microsoft.KeyVault/vaults",
      "name": "[parameters('keyVaultName')]",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[resourceId('Microsoft.DataFactory/factories', parameters('adfName'))]"
      ],
      "properties": {
        "sku": {
          "name": "standard",
          "family": "A"
        },
        "tenantId": "[variables('tenantId')]",
        "accessPolicies": [
          {
            "tenantId": "[variables('tenantId')]",
            "objectId": "[reference(concat(variables('fullAdfName'), '/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-PREVIEW').principalId]",

            "permissions": {
              "keys": [
                "Get",
                "List",
                "Update",
                "Create",
                "Import",
                "Delete",
                "Recover",
                "Backup",
                "Restore"
              ],
              "secrets": [
                "Get",
                "List",
                "Set",
                "Delete",
                "Recover",
                "Backup",
                "Restore"
              ],
              "certificates": [
                "Get",
                "List",
                "Update",
                "Create",
                "Import",
                "Delete",
                "Recover",
                "Backup",
                "Restore",
                "ManageContacts",
                "ManageIssuers",
                "GetIssuers",
                "ListIssuers",
                "SetIssuers",
                "DeleteIssuers"
              ]
            }
          },
          {
            "tenantId": "[variables('tenantId')]",
            "objectId": "[parameters('keyVaultAdminObjectId')]",

            "permissions": {
              "keys": [
                "Get",
                "List",
                "Update",
                "Create",
                "Import",
                "Delete",
                "Recover",
                "Backup",
                "Restore"
              ],
              "secrets": [
                "Get",
                "List",
                "Set",
                "Delete",
                "Recover",
                "Backup",
                "Restore"
              ],
              "certificates": [
                "Get",
                "List",
                "Update",
                "Create",
                "Import",
                "Delete",
                "Recover",
                "Backup",
                "Restore",
                "ManageContacts",
                "ManageIssuers",
                "GetIssuers",
                "ListIssuers",
                "SetIssuers",
                "DeleteIssuers"
              ]
            }

          }
        ]
      },
      "resources": [
        {
          "type": "secrets",
          "name": "connectionStringSqldbBetl",
          "apiVersion": "2016-10-01",
          "properties": {
            "value": "[parameters('connectionStringSqldbBetl')]"
          },
          "dependsOn": [
            "[variables('fullKeyVaultName')]"
          ]
        },
        {
          "type": "secrets",
          "name": "connectionStringSqldbAw",
          "apiVersion": "2016-10-01",
          "properties": {
            "value": "[parameters('connectionStringSqldbAw')]"
          },
          "dependsOn": [
            "[variables('fullKeyVaultName')]"
          ]
        },
        {
          "type": "secrets",
          "name": "connectionStringSqldbRdw",
          "apiVersion": "2016-10-01",
          "properties": {
            "value": "[parameters('connectionStringSqldbRdw')]"
          },
          "dependsOn": [
            "[variables('fullKeyVaultName')]"
          ]
        }
      ]
    } // key vault

  ],
  "outputs": {}
}

the pre and post deployment steps are standard microsoft scipts:

https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment

I place this script in my dev-ops folder and call it using the following path:

$(System.DefaultWorkingDirectory)/Build ADF/ArmTemplates/dev-ops/adf_util.ps1

and these arguments:

-armTemplate "$(System.DefaultWorkingDirectory)/Build ADF/ArmTemplates/ARMTemplateForFactory.json" -ResourceGroupName "$(resourceGroupName)" -DataFactoryName "$(adfName)" -predeployment $true -deleteDeployment $false