Lighthouse · Posted 3w ago
Backend Engineer (Data Acquisition)
Job relayed by Stapply Data, its source: you apply there.
- Location
- 🇪🇸 Barcelona, Spain
- Contract
- Full-time
- Role
- Data engineering
- Source
- Stapply Data
Stack
- Rust
- C++
The posting
Join our Crawler Engineering team as a Backend Engineer focused on data acquisition. In this building role, you will design and develop high-performance systems in Rust that collect billions of data points from the web. You will work on our in-house proxy layer and browser tooling, investigate hard crawler problems, and build clear interfaces for other teams and AI agents to use. You will also own the quality of your code and work closely with crawler engineers, DevOps, and Security to ship secure and scalable solutions.
- Design and develop high-performance systems in Rust for data acquisition, handling close to 200 million requests a day. - Work on in-house proxy layer and browser tooling, investigating hard crawler problems and testing alternative approaches. - Collaborate with crawler engineers, DevOps, and Security to ship secure and scalable solutions, ensuring the quality of your code.
- You have a solid grasp of how the web works under the hood: HTTP, TCP/IP, TLS and proxies - You have strong analytical and debugging skills. You're comfortable with packet captures, profilers and reading other people's source code - You have an R&D mindset. You're curious about how browsers and network stacks work internally, you're comfortable with ambiguity, and you'd rather prove a hypothesis than assume it - You're fluent in English - You're experienced with a systems language such as Rust or C/C++, or you have a strong track record in another backend language and a real drive to go deep in Rust - You have hands on experience building, deploying and running production backend services in a cloud environment - You communicate clearly, share your opinions, welcome feedback and work well in a team - Not ticking every box? No problem! We value diverse backgrounds and unique skill sets, and we encourage individuals from all walks of life to apply. If your experience looks a little different from what we've described, but you're passionate about what we do and are a quick learner, we'd love to hear from you!