Lead Data Engineer
Title: Lead Data Engineer
Department: Data Science
Reports To: Director of Informatics
FLSA Status: Exempt
Location: Hybrid - New York, NY
Job Summary
As a Lead Data Engineer on Prove’s Data Science team, you will build the technical backbone that translates complex identity data into verified digital trust. Prove is fundamentally a data company: our identity solutions and analytical products are built on extracts spanning all tiers of data sophistication. In this role, every pipeline and architecture you design connects directly to revenue-generating products that secure transactions and prevent fraud for more than 1,000 global enterprise customers.
You will spearhead the development and maintenance of high-throughput data pipelines that power our core inference systems, taking raw intelligence and refining it into reliable scientific statements. Acting as a bridge across disciplines, you will translate multi-domain logic into resilient, highly performant directed acyclic graphs (DAGs), while proactively establishing data governance and ontology standards. This is an environment built for experimentation, offering you the autonomy to deploy modern paradigms and tools to solve intricate problems in identity resolution.
Key Responsibilities:
Architect, build, and maintain resilient data pipelines that power real-time and batch inference, moving data from raw intelligence to verifiable scientific statements.
Translate complex, multidisciplinary domain logic into efficient, maintainable, and modular directed acyclic graphs (DAGs) and execution workflows.
Proactively implement and advocate for robust data governance frameworks, metadata management, and semantic data ontologies.
Partner closely with data science, informatics, and product engineering teams to serve high-integrity data extracts that directly power revenue-generating products.
Continuously evaluate, benchmark, and adopt cutting-edge data engineering tools, frameworks, and storage paradigms.
Optimize distributed data processing pipelines for high availability, computational efficiency, and cost governance.
Champion engineering excellence across the team through code reviews, architectural discussions, automated testing, and CI/CD practices.
Promote, maintain and enhance our cultural values of humility, passion, inclusion, and leadership.
Qualifications and Experience:
Strong passion for learning about our products and markets through in-house and external training.
7+ years of professional experience in data engineering or developing complex, data-driven software products.
Deep, first-principles understanding of data systems, distributed compute engines, data modeling, and modern pipeline architecture.
Demonstrated track record of collaborating across technical and scientific functions, with the ability to translate cross-disciplinary logic into concrete data systems.
High intellectual curiosity and a proactive mindset toward experimentation and technological innovation.
Hands-on experience with modern data orchestration (e.g., Airflow, Dagster, Prefect) and large-scale distributed frameworks.
Preferred Qualifications:
Advanced degree (Master’s or Ph.D.) in Computer Science, Mathematics, Statistics, Physics, or a related quantitative field.
Experience architecting creative solutions for identity resolution, graph data structures, or fraud prevention systems.
Fluent and terminal-native in Linux/Unix environments, command-line utilities, and shell scripting.
Experience in high-growth/pre-IPO Technology companies.
The anticipated salary pay range for Metro 3 areas for this role is $175,000 - $200,000 plus company bonus.