Functieomschrijving

Data Engineer - Azure Databricks & Application Development
 
Position Overview
 
DigiTribe is looking for a Data Engineer with deep expertise in Azure and Databricks to design, develop, and maintain scalable data engineering solutions. You will build production-grade ETL/ELT pipelines, develop data processing applications, and contribute to a modern lakehouse architecture. Working cross-functionally with Architecture, Data, AI, Cloud, and CI/CD teams, you will translate business and analytical requirements into robust, reusable, and cost-efficient data solutions.
 
Key Responsibilities

• Design, develop, test, and maintain scalable data engineering solutions on Microsoft Azure and Databricks, using Python, PySpark, Scala, and SQL.
• Implement reliable batch and streaming data workflows using Azure Databricks, Delta Lake, and modern data lakehouse patterns.
• Develop data processing applications for ingestion, transformation, enrichment, validation, and serving of analytical datasets.
• Build CI/CD pipelines for data applications using Azure DevOps YAML pipelines and apply software engineering best practices: modular design, clean code, version control, code reviews, and documentation.
• Develop reusable libraries, utilities, and frameworks to accelerate data product delivery and support less experienced engineers through coaching and knowledge sharing.
• Collaborate with Cloud, DevOps, and Platform teams on deployment, environment configuration, and operational readiness.

• 5+ years of hands-on experience in data engineering, application development, or cloud-based data platform development.
• Strong hands-on development experience with Python, PySpark, Scala, SQL, and ETL/ELT development.
• Solid experience with the Azure Databricks ecosystem: notebooks, jobs, Spark clusters, Delta Lake, lakehouse architecture, and performance tuning.
• Good knowledge of Azure data services: Azure Data Lake Storage, Azure Data Factory, Azure Key Vault, Azure DevOps, and Azure monitoring/logging capabilities.
• Experience with CI/CD pipelines (Azure DevOps YAML), version control, automated testing, modular development, and code reviews.
• Understanding of cloud security, access management, observability, and cost-aware development practices.
• Working knowledge of Terraform or infrastructure-as-code concepts is a plus.

• Experience with large-scale distributed data processing using Spark and streaming/real-time ingestion patterns.
• Experience building reusable data engineering frameworks, libraries, or shared components.
• Knowledge of data quality frameworks, metadata management, lineage, and governance practices.
• Experience integrating data pipelines with AI, machine learning, or analytics use cases; familiarity with MLOps is an advantage.
• Preferred certifications: Microsoft Azure Data Engineer Associate, Azure Developer Associate, Azure Solutions Architect Expert, Databricks Certified Data Engineer Associate/Professional, or Databricks Certified Developer for Apache Spark.

• Strong communication and documentation skills, with the ability to explain technical data engineering concepts to diverse audiences.
• Strong problem-solving mindset with a focus on automation, reliability, and maintainability.
• Team-oriented, proactive, and comfortable working in a cross-functional environment.
• Continuous improvement mindset and willingness...