Kforce is working with a client in search of a Lead Data Engineer- Azure Databricks, to join their team in NYC. This position is full time, direct hire and on-site 1-day a week. Summary: As a Senior Data Engineer within this department, you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources. This role requires expertise in creating efficient data pipelines, handling both streaming and batch data processing, and ensuring data integrity throughout the ETL lifecycle. The Lead Data Engineer will implement monitoring solutions, optimize ETL jobs for performance, and provide comprehensive support from data ingestion to final output. With in-depth knowledge of the Databricks platform and strong analytical skills, this position will significantly enhance the department's ability to deliver high quality, data-driven insights and solutions. Responsibilities * Strategize, design, develop and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions * Design and develop ETL code to integrate various data sources * Create and maintain efficient data pipelines on the Databricks platform * Develop, maintain, and optimize ETL pipelines for both streaming and batch data processing * Ensure data integrity and consistency throughout the ETL lifecycle * Provide comprehensive support for ETL processes, from data ingestion to final output * Write and execute unit tests and integration test cases for ETL code to ensure high-quality outcomes * Implement monitoring solutions for ETL jobs to ensure timely and successful data processing * Proactively identify, troubleshoot, and resolve issues in ETL workflows * Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting * Mentor other data engineers in the team, cross-train and provides guidance