Lead II - AWS Databricks Engineer Company: UST Job ID: 64536 Experience: 7–9 Years Locations: Bangalore, Hyderabad, Pune, Trivandrum Open Positions: 1 Position Overview UST is seeking an experienced Lead Data Engineer to architect, build, and optimize scalable, cloud-native data systems. The role involves leading the development of enterprise data platforms using Databricks and AWS. Candidates must possess deep domain expertise in US Healthcare, with a strict requirement for hands-on experience handling Medicare Advantage data. Key Responsibilities * Lead the design, implementation, and maintenance of robust data pipelines on AWS using Databricks. * Architect scalable lakehouse structures leveraging Apache Iceberg to ingest, process, and store complex healthcare data. * Establish comprehensive data governance and data lineage frameworks using Databricks Unity Catalog. * Implement secure access controls, data masking, and row-level security through Role-Based Access Control (RBAC). * Architect and scale ingestion frameworks capable of processing HL7 and FHIR data feeds natively into the lakehouse ecosystem. * Oversee end-to-end processing of Medicare Advantage data assets, ensuring accurate risk adjustment and reporting workflows. * Ensure strict adherence to healthcare regulations, including HIPAA compliance and robust data governance frameworks. * Mentor junior engineers, conduct code reviews, and drive technical best practices across the data team. Mandatory Skills & Qualifications * 7+ years of data engineering experience, with at least 2 years in a technical lead role. * Production experience building and optimizing large-scale pipelines within Databricks. * Production experience deploying data infrastructure on AWS, specifically S3, EMR, Lambda, IAM, and Glue. * Comprehensive understanding of US Healthcare data, with explicit hands-on experience processing Medicare Advantage data, including: * CMS Enrollment * RAPS/EDPS Submissions * MMR/MOR Files * HCC Risk Adjustment Models * Proven experience working with healthcare interoperability standards, specifically FHIR and HL7 models. * Deep understanding and practical experience with open table formats, specifically Apache Iceberg. * Expert-level knowledge of Databricks Unity Catalog for centralized data governance. * Proven experience implementing enterprise-level Role-Based Access Control (RBAC) and row-level security for protected health information (PHI). * Mastery of Python, PySpark, and advanced SQL for complex distributed computing tasks. Good-to-Have Skills * Databricks Certified Data Engineer Professional certification. * AWS Certified Data Analytics certification. * Familiarity with orchestration tools such as Apache Airflow or AWS Step Functions. Ideal Candidate Candidates with strong hands-on expertise in AWS, Databricks, Python, PySpark, and Apache Iceberg, along with proven experience in US Healthcare and Medicare Advantage data engineering, are encouraged to apply. Important: Hands-on Medicare Advantage healthcare data experience is a strict requirement for this position.