Crawler engineer
job summary:
this role focuses on enhancing and mastering lex machina's crawling infrastructure to support our customer-facing products.
you will be responsible for designing and maintaining robust automated data collection systems in a challenging and large-scale environment.
your work will play a critical role in driving the success of our products, making this an impactful and rewarding opportunity.
responsibilities:
build, maintain, and optimize automated data collection and storage systems, ensuring reliability and scalability.
master and enhance lex machina's crawling infrastructure to support customer-facing applications.
extract and process large datasets from industries such as retail pricing, neobanks, and airlines or otas.
debug, troubleshoot, and optimize crawlers for maximum performance and accuracy.
collaborate with cross-functional teams to align data collection processes with business goals.
create and maintain comprehensive documentation of crawling infrastructure and processes.
communicate effectively with stakeholders by asking and answering questions, addressing issues and providing detailed updates.
qualifications:
bachelor's degree in computer science, software engineering, or a related field (or equivalent experience).
3-5+ years of experience in backend software engineering, with a focus on python in challenging environments.
proven expertise in large-scale data collection, processing, and storage systems.
strong problem-solving skills with an enthusiasm for detail and mastery of technical systems.
excellent communication skills, including thorough documentation and the ability to ask/answer questions diligently.
self-motivated, with a passion for delivering high-quality work that supports customerfacing products.
preferred qualifications:
experience with distributed systems and scalable architecture.
knowledge of modern software development practices, such as ci/cd and agile methodologies.
familiarity with cloud environments such as aws or azure.