IN_Manager_Data Engineer_D&A_Advisory_Mumbai

Responsibilities: 

Job Description: Data Engineer (Azure Databricks) 

Role Overview: 

Data Engineer with strong hands-on expertise in Azure and Databricks to build, optimize, and operate scalable data pipelines and lakehouse solutions. You will partner with architects, analysts, and business stakeholders to deliver trusted, analytics-ready datasets for insurance use cases. 

Key Responsibilities 

· Design, build, and support batch and near real-time data pipelines on Azure using standard engineering practices. 

· Develop and optimize Databricks notebooks/jobs using PySpark/Spark, SQL, and Python; follow code review, testing, and documentation standards. 

· Implement and maintain lakehouse tables (Delta) and curated layers; apply partitioning and file-layout best practices for performance. 

· Create orchestration and scheduling using Azure Data Factory and/or Databricks Workflows; manage dependencies, retries, and SLAs. 

· Implement data quality checks, observability (logging/metrics/alerts), and incident triage to ensure reliable pipelines in production. 

· Apply security and governance controls for regulated/PII data (RBAC, encryption, secrets management) and support audits as needed. 

· Tune Spark workloads and manage Azure/Databricks costs (cluster policies, job sizing, caching, indexing/z-order) to meet performance targets. 

Required Skills & Qualifications 

· Bachelor’s/master’s degree in computer science, Engineering, or related. 

· 3–6 years of data engineering experience delivering production pipelines on Azure. 

· Strong Databricks expertise (Spark/PySpark), SQL, and Python. 

· Hands-on with Azure data services (ADLS Gen2, ADF) and CI/CD using Git-based workflows; familiarity with DevOps practices is a plus. 

· Good understanding of data modeling and lakehouse patterns (Delta). 

· Strong communication and stakeholder management; ability to translate requirements into scalable data solutions (insurance domain preferred). 

Preferred Attributes 

· Certifications: Azure Data Engineer Associate and/or Databricks (preferred). 

· Exposure to Databricks Unity Catalog/Delta Live Tables and insurance data domains (policy, claims, billing) is a plus; experience with performance/cost optimization is valued. 

Mandatory skill sets: 

Data Engineer/Azure  

Preferred skill sets: Data Engineer/Azure  

Years of experience required: 

10--12 Years  

Education qualification: 

B.E.(B.Tech)/M.E/M.Tech