Key Responsibilities:

Design, develop, and implement data processing pipelines and workflows using Apache Spark, Scala, and Python to handle large-scale data. Work with cloud technologies, primarily AWS, to build and maintain data infrastructures including EC2, S3, Lambda, EMR, and Redshift. Manage and integrate MongoDB to store, retrieve, and process large datasets efficiently. Collaborate with data scientists, analysts, and other teams to understand data needs and develop solutions that meet business requirements. Optimize data processes for scalability and performance by utilizing the best practices for distributed computing. Ensure data quality, consistency, and security across all systems. Develop automation scripts and monitor system performance to prevent issues and improve the data pipeline process. Participate in architecture discussions to design scalable, high-performance data systems. Troubleshoot and resolve data-related issues in a timely manner.