A production engineer is a developer with who thinks deeply about systems and how they behave in the wild. Whether it be networking, or the Linux kernel, or even a specific interest in scaling, algorithms, or distributed systems. You are a systems engineer who aims to code themselves out of a job by automating all the things, learning to leverage great development practices like Test-Driven-Development or continuous integration (to start with).
Responsibilities
Infrastructure Responsibilities
Radiate knowledge about the service's infrastructure and reliability to the rest of the development team.
Identify parts of the system that do not scale, provide immediate palliative measures and drive long term resolution of these incidents.
Plan the growth of Benzinga's infrastructure.
Development/Deployment Responsibilities
Document every action so your learnings turn into repeatable actions and then into automation.
Improve the deployment process to make it as boring as possible.
Manage our infrastructure with Kubernetes and Terraform.
Security Responsibilities
Proactively identify and reduce security risks
Develop security training and guidance to internal development teams
Ability to discover and patch SQLi, XSS, CSRF, SSRF, authentication and authorization flaws, and other web-based security vulnerabilities (OWASP Top 10 and beyond)
Knowledge of common authentication technologies including JWT, OAuth, SAML, CAs, OTP/TOTP
Production Responsibilities
Design, build and maintain core infrastructure pieces that allow Benzinga to scale to support hundred of thousands of concurrent users.
Be on an on-call rotation to respond to benzinga.com availability incidents and provide support for service engineers with customer incidents.
Debug production issues across services and levels of the stack.
Monitoring Responsibilities
Make monitoring and alerting alert on symptoms and not on outages.
Manage day-to-day maintenance and evolution of Benzinga's Prometheus monitoring and alerting infrastructure
Bundle Prometheus monitoring as an out-of-the-box monitoring solution for Benzinga products
Configure dashboards and alerts using Grafana
Build and maintain the benzinga.com public monitoring gateway
Improve coverage of Benzinga performance monitoring
Create automated alerts to notify team members of regressions
Requirements
You can reason about software, algorithms, and performance from a high level.
You have experience thinking about systems - edge cases, failure modes, behaviors, and specific implementations.
You have worked with distributed systems and have a solid understanding of how modern web stacks are built, and why.
You know your way around Linux and the Unix Shell.
Strong communication skills
Experience with Docker, Nginx, Go, Kubernetes, Kong, Prometheus
Experience with managing large amounts of telemetry
Oddly enjoys updating services at 1am on a Sunday, hanging out with other devs while everyone else is at the bar sleeping.
FIrst to jump in when fighting fires
Prepares us to fight fires as little as possible
Self-motivated with strong organizational skills