Back to projects

Data Integration Platform CI/CD (Azure Data Factory)

Reworked CI/CD for an internal Azure Data Factory platform, replacing manual promotions with a Dev → Preprod → Prod pipeline, clear environment-specific configuration, and approval gates for safer releases.

Impact

Set up a reliable promotion path for ADF across environments.
Removed repeated deployment blockers for data and database engineers.

Data Factory deployments between Development, Preproduction, and Production were mostly manual and inconsistent, which slowed releases and caused avoidable issues.

  • Data Factory wasn't following a clean CI/CD approach.
  • Manual promotion caused drift and repeat deployment errors.
  • Environment-specific values were effectively hard-coded, making changes risky.
  • Database and data engineers were getting blocked by deployment problems.
  • Introduce a repeatable CI/CD process aligned with Microsoft's recommended approach for ADF.
  • Promote changes safely across Dev → Preprod → Prod.
  • Parameterise environment-specific values without duplicating code.
  • Reduce deployment issues for engineers working on pipelines and data pipelines.
  • Azure Data Factory instance per environment.
  • Managed Identity used to connect ADF to dependent Azure resources.
  • Linked services parameterised to support different environments safely.
  • ARM templates generated from ADF for controlled deployments.
  • CI pipeline generates ARM templates from the Dev Data Factory (adf_publish flow).
  • Environment-specific parameters applied during deployment (Preprod/Prod).
  • Automated deployment to Preprod to keep environments in sync.
  • Approval gates for Preprod (test team) and Production (DevOps team).
ARM-based CI/CD (supported ADF approach)
ARM templates require additional configuration, but the deployment model is predictable and well-supported.
Auto deploy to Preprod, gate Production
Production changes take an extra step, but releases are safer and easier to track.
  • Managed identities used for Data Factory access to Azure resources.
  • No secrets stored in source control or pipeline definitions.
  • Secrets pulled from Azure Key Vault via linked services instead of being stored in pipelines or config.
  • Production deployments restricted via approvals and permissions.
  • Deployments followed the same process across environments, reducing environment-specific failures.
  • Standard promotion flow reduced deployment-related incidents.
  • Rollback is straightforward by redeploying the last known-good template.
  • Reliable CI/CD pipeline for Azure Data Factory deployments.
  • Fewer deployment errors and less configuration drift across environments.
  • Improved productivity for database and data engineers by removing deployment blockers.
  • Add automated validation checks for ADF changes before deployment.
  • Improve pipeline output with clearer summaries (which pipelines/linked services changed).
  • Expand monitoring and alerting around ADF execution failures.