Specialist Data Engineer Job ID: 6711 Business Unit: MTA Headquarters Location: New York, NY, United States Regular/Temporary: Regular Department: Information & Warehouse Mgt Date Posted: Jul 22, 2024 Description JOB TITLE : Specialist Data Engineer SALARY RANGE : $ 110,748 - $130,719 HAY POINTS : P4 / 451 DEPT/DIV : Information Technology / Products SUPERVISOR : Deputy Chief Information and Warehouse Management LOCATION : Various/ 2 Broadway New York, NY 10004 HOURS OF WORK : 9:00 am - 5:30 pm/7.5 hrs. or as required. DEADLINE : Open Until filled This position is eligible for telework which is currently two days per week. New hires are eligible to apply 30 days after their effective date of hire. SUMMARY : This role is responsible for the design, deployment, development, and maintenance of data solutions. The Data Engineer will participate in a variety of data-related projects and work closely with Data analysts, data scientists, Business users, and other stakeholders to gather requirements and build data pipelines that meet the organizations’ needs. The Data Engineer will deploy and support Data platforms that process and store data. They will contribute to software development methods, tools, and techniques and apply agreed standards and tools to achieve well-engineered outcomes. RESPONSIBILITIES + Design, develop, and implement data solutions to meet business requirements and data ingestion needs facilitating accurate and timely data availability for analysis and decision-making. + Extract, load, and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files. + Write well-designed, efficient code that adheres to security standards. + Monitor and troubleshoot data pipelines to identify and resolve issues promptly to minimize disruptions in data processing for on-premises and cloud environments. + Implement data quality checks and validation processes to ensure accuracy and completeness of data. collaborate with the data team to resolve them. + Writes complex queries and scripts to efficiently manipulate, transform, and process raw data. + Creates and executes data validation processes to ensure the reliability and consistency of incoming data. Build processes to monitor data quality. + Continuously optimize data pipelines for performance, scalability, and reliability. + Create and maintain technical documentation. Contribute to building and maintaining data catalog and lineage. + Design and develop CI/CD processes that ensure high availability and agility. + Develop cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability. + Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform. + Stay updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization's data intelligence capabilities. + Monitors and reports on supplier performance, customer satisfaction, adherence to security requirements and market intelligence. + May mentor less experienced staff + Performs other duties and tasks. + May need to work outside of normal work hours (i.e., evenings and weekends) + Travel may be required to other MTA locations or external sites. KNOWLEDGE, SKILLS, AND ABILITIES + Strong knowledge of Big Data architectures, large data warehouses, and Data Lake solutions. + Experience designing and implementing Modern Data and Analytics solutions including Lakehouse architecture and medallion architecture. + Proficient in cloud services including Azure Databricks. Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview, and Power BI. + Experience deploying and running cloud-based data solutions and infrastructure-as-code frameworks like Terraform. + Proficient in cloud deployments (MS Azure preferred) in an agile SDLC environment, leveraging modern programming languages, and DevOps. + Experience with major database platforms including Oracle, SQL Server as well as Cloud databases and NOSQL Databases. + Strong understanding of data engineering concepts, ELT/ETL principles, and data modeling. + Experience with data integration techniques for both structured and unstructured data. + Solid programming skills in languages such as Python, Pyspark, and SQL. + Experience with Airflow. + Experience in DevOps, Git Repos, and CI/CD pipelines for code deployment. + Experience deploying and administering cloud-based data solutions using infrastructure-as-code and infra-automation tools like Terraform, Ansible, etc. + Experience with Microsoft Purview is a plus. + Strong knowledge of Microsoft Azure Cloud. AWS and GCP are desirable. + Functional knowledge of Microsoft Power BI + Experience with Jira, Confluence. + Demonstrated ability to work independently and navigate organizational ambiguity. + Effective written and verbal communication skills + Hands-on programming experience in a business setting. + Proficiency in at least one software engineering methodology, including but not limited to Agile, Scrum, DevOps, Extreme Programming (XP), Kanban, Lean, and Rapid Application Development (RAD). + Experience applying structured validation and testing methods, including but not limited to Unit Testing, Integration Testing, System Testing, Acceptance Testing, and Regression Testing. + Demonstrated ability to work independently and navigate organizational ambiguity. + Effective written and verbal communication skills EDUCATION AND EXPERIENCE Qualifications: + Education: Bachelor’s Degree + Experience: At least 3 years of relevant experience. An equivalent combination of education and experience may be considered instead of a degree. Preferred Qualifications: + Microsoft Certified: Azure Enterprise Data Engineer Associate + Microsoft Certified: Azure Data Fundamentals Preferred Technical Skills: + Data Structures and Algorithms (Thorough Knowledge/Fully Proficient) + Cloud Computing (Thorough Knowledge/Fully Proficient) Soft Skills: + Active Listening, Attention to Detail, Customer Service, + Prioritization, Problem-Solving, Effective Verbal and Written Communication Competencies: Collaborates Building partnerships and working collaboratively with others to meet shared objectives. Cultivates Innovation Creating new and better ways for the organization to be successful. Customer Focus Building strong customer relationships and delivering customer-centric solutions. Values Diversity Recognizing the value that different perspectives and cultures bring to an organization. Communicates Effectively Developing and delivering multi-mode communications that convey a clear understanding of the unique needs of different audiences. Tech Savvy Anticipating and adopting innovations in business-building digital and technology applications. OTHER INFORMATION: Under the New York State Public Officers Law & the MTA Code of Ethics, all employees who hold a policymaking position must file an Annual Statement of Financial Disclosure (FDS) with the NYS Commission on Ethics and Lobbying in Government (the “Commission”). Equal Employment Opportunity MTA and its subsidiary and affiliated agencies are Equal Opportunity Employers, including those concerning veteran status and individuals with disabilities. The MTA encourages qualified applicants from diverse backgrounds, experiences, and abilities, including military service members, to apply.