Midlevel Backend Engineer - Storage Team

What We’re Building

Honeycomb is built to help engineering teams deeply explore and understand their own production systems — in real time. It’s a service for the near and present future, where distributed systems are the new default, every service is a platform, and empowered generalist software engineers are the new ops. We are passionate about consumer-quality developer tools and excited to build technology that raises our industry’s expectations of what our tools can do for us. We’re working with well known companies like HelloFresh, Slack, LaunchDarkly, and Vanguard and more across a range of industries. This is an exciting time in our trajectory, as we’ve closed Series D funding, scaled past the 200-person mark, and were named to Forbes’ America’s Best Startups of 2022 and 2023!

The Honeycomb product handles enormous amounts of data and the Storage team owns the systems (called “Retriever” internally) responsible for storing and querying those data. Curious how we do it? Interested in finding ways to improve systems at scale? Come help us build resilient architectures, obsess over query performance and optimization, collaborate with product teams to ensure our customers have access to their data in a highly performant way, and have fun on a highly skilled and engaging team. We’re looking for a backend systems software developer to join the team. Our services are built in Go, and while experience with Go is not a requirement, an interest in becoming a Go expert is a must. It would be amazing (but not required) if you have some experience building databases, have worked with performance-limited environments or have Kubernetes Operator chops.

If you are interesting in learning more about the Storage team and our work, check out this talk on how we used serverless to speed up our server, this talk on why we chose to create a distributed columnar store and this blog post on our recent work to virtualize our storage engine.

Who We Are

How We Work

We are a remote-first company, which means we believe it is not where you sit, but how you deliver that matters most. We invest in our people and care about how you orient to our culture and processes. At the same time we imbue a lot of trust, autonomy, and accountability from Day 1. #LI-Remote

What You’ll Do:

• Scale high-throughput services. Honeycomb processes millions of events per second for our customers. This role will let you explore this bespoke architecture and contribute to improving its efficiency, reliability and performance. You’ll help us ensure that the product remains performant for all of our customers, irrespective of their size and complexity.
• Maintain and mature our datastore. You’ll be a critical part in the operational management, performance and reliability of our distributed columnar datastore. This includes optimization as well as experimentation with new approaches to meeting the needs of our largest customers who have loads of data.
• Support software in production. We believe in owning the software that we build. Ownership is built through collaborating to define our team’s KPIs, product SLOs, and alerts. Every engineer on the team participates in an on-call rotation to support issues that our customers encounter (reactive), to improve sticking points that our customers may face (proactive), and to eliminate sources of engineering toil.
• Partner with your team. The Storage team is a highly collaborative, diverse, dedicated and enthusiastic group of individuals working towards a common goal. You’ll have plenty of opportunity to partner with and drive high impact initiatives that enable our customers to do their best work.
• Experiment. There aren’t many easy answers on the Storage team. We lean into a culture of experimentation and data-backed development to test out our hypotheses. If you love a culture of experimentation, this is a great place to be.

About You:

• You work well with other teams. As a product-focused team in the platform engineering org, Storage enables other engineering teams to deliver new and exciting features to our customers. You’ll have the opportunity to explore ways we can support these efforts as well as consider ways to unlock additional functionality for the company.
• You care deeply about software quality and testing. A strong candidate for this role would be interested in testing and have experience finding the best ways to keep a high level of quality without spending an ever-growing amount of time on maintaining tests. They would also be a practiced code reviewer: able to give kind, thoughtful feedback to other engineers on pull requests to the codebase.
• You value clear, open communication. We see clear, thoughtful, and regular written & face-to-face communication (including a willingness to give & receive feedback) as a fundamental engineering skill and one of the strongest predictors of future engineering impact, not just a nice to have, and we hope you do too.

What You Get When You Join The Hive!

• Base pay (range) of $165,000 - $180,000 USD
• A stake in our success - generous equity with employee-friendly stock program
• It’s not about how strong of a negotiator you are - our pay is based on transparent levels relative to experience
• Compensation benchmarked to San Francisco market - no matter where you live (or move)!
• 401k plan to help you plan for your future
• A remote-first mindset and culture (really!)
• 100% employee coverage for Health, Dental, Vision, Life and Disability insurance and 75% for dependents
• Time to Recharge - In addition to our Unlimited PTO policy, we provide ~23 days off through out the year. This includes a company wide break at the end of the year, and we honor having at least one three day weekend a month (if there is not already a locally observed holiday that month, we add one!)
• Pick Your Perk - $600 a year to spend on the perks that you care about most
• Work Life Balance and Flexible Schedule options
• The tech you need AND a $500 Home Setup Stipend
• $200 Reimbursement for Cell/Wifi/CoWorking
• $1500+ Annual Professional Development Allowance
• Up to 16 weeks of paid parental leave, regardless of path to parenthood
• Maven Inclusive Family-Building benefit including unlimited virtual appointments, coaches & counselors, and $10K ‘wallet’ to support adoption, surrogacy, IVF, and egg/sperm freezing
• Modern Health well-being benefit including self guided resources and access to 12 mental health and 8 coaching sessions - at no additional cost to you.
• Semi-annual performance conversations (we call them Review & Rewards conversations) - so you know where you stand, and how you’ll be rewarded for your impact
• Annual compensation review, benchmarking to industry and inflation changes

Our Interview Process:

We know that interviewing can be super stressful! So we want to be as transparent and kind as possible around what our interview process looks like. Please feel free to read more on this blog post.

But in general, here is the flow:

1. Recruiter Screen

2. Hiring Manager Screen

3. Technical Take Home exercise, as well as a 30 minute call with the team to talk through your work and decision-making process. And it's not about how far you get - it's more about how you think about and approach problems.

4. Virtual Onsite, where you'll meet more of the team, including members of the product triad.

Diversity & Accommodations:

We're building a diverse and inclusive workplace where we learn from each other, and welcome nontraditional candidates, and people of all backgrounds, experiences, abilities and perspectives. You don't need to be a millennial to join us, all gens are welcome! Further, we (of course) follow federal and state disability laws and are happy to provide reasonable accommodations during the application phase, interview process, and employment. Please email [email protected] to discuss accessible formats or accommodations. As an equal opportunity employer our hiring process is designed to put you at ease and help you show your best work; if we can do better - we want to know!