← Back to Projects
Data Integration Platform CI/CD (Azure Data Factory)

Data Integration Platform CI/CD (Azure Data Factory)

Reworked CI/CD for an internal Azure Data Factory platform, replacing manual promotions with a Dev → Preprod → Prod pipeline, clear environment-specific configuration, and approval gates for safer releases.

DevOps Engineer2025 · ~3 monthsInternalAzure Data FactoryCI/CDDevOpsData Engineering

Data Factory deployments between Development, Preproduction, and Production were mostly manual and inconsistent, which slowed releases and caused avoidable issues.

  • Data Factory wasn't following a clean CI/CD approach.
  • Manual promotion caused drift and repeat deployment errors.
  • Environment-specific values were effectively hard-coded, making changes risky.
  • Database and data engineers were getting blocked by deployment problems.
  • Introduce a repeatable CI/CD process aligned with Microsoft's recommended approach for ADF.
  • Promote changes safely across Dev → Preprod → Prod.
  • Parameterise environment-specific values without duplicating code.
  • Reduce deployment issues for engineers working on pipelines and data pipelines.
  • Set up a reliable promotion path for ADF across environments.
  • Removed repeated deployment blockers for data and database engineers.
Data Integration Platform CI/CD (Azure Data Factory) architecture diagram
  • Azure Data Factory instance per environment.
  • Managed Identity used to connect ADF to dependent Azure resources.
  • Linked services parameterised to support different environments safely.
  • ARM templates generated from ADF for controlled deployments.
Data Integration Platform CI/CD (Azure Data Factory) pipeline flow
  • CI pipeline generates ARM templates from the Dev Data Factory (adf_publish flow).
  • Environment-specific parameters applied during deployment (Preprod/Prod).
  • Automated deployment to Preprod to keep environments in sync.
  • Approval gates for Preprod (test team) and Production (DevOps team).
ARM-based CI/CD (supported ADF approach)
ARM templates require additional configuration, but the deployment model is predictable and well-supported.
Auto deploy to Preprod, gate Production
Production changes take an extra step, but releases are safer and easier to track.
  • Managed identities used for Data Factory access to Azure resources.
  • No secrets stored in source control or pipeline definitions.
  • Secrets pulled from Azure Key Vault via linked services instead of being stored in pipelines or config.
  • Production deployments restricted via approvals and permissions.
  • Deployments followed the same process across environments, reducing environment-specific failures.
  • Standard promotion flow reduced deployment-related incidents.
  • Rollback is straightforward by redeploying the last known-good template.
  • Reliable CI/CD pipeline for Azure Data Factory deployments.
  • Fewer deployment errors and less configuration drift across environments.
  • Improved productivity for database and data engineers by removing deployment blockers.
  • Add automated validation checks for ADF changes before deployment.
  • Improve pipeline output with clearer summaries (which pipelines/linked services changed).
  • Expand monitoring and alerting around ADF execution failures.