Engineer III - Data Engineer
What you'll do
Data Pipeline & Orchestration: Design, build, and run batch and scheduled data pipelines (Airflow, Dagster, Prefect) and lead orchestrator upgrades.
Data Ingestion & Scraping: Own and extend data collectors and engines, including web/API ingestion, large-scale web scraping (Playwright, Selenium, Scrapy), parsing, normalization, and file merging.
Distributed Processing: Build scalable distributed processing architectures (Spark on EMR, Databricks, Dataproc) and analytical query layers (Athena, Snowflake, BigQuery), including cross-cloud ingestion pipelines.
Caching & Entity Matching: Improve entity matching and caching services (Redis, MemoryDB, DynamoDB) to optimize accuracy, latency, and operational cost.
Data Access & APIs: Develop and maintain high-performance Python dataaccess layers and APIs for downstream ML and product teams.
Data Quality & Observability: Own data quality initiatives, including validation frameworks, retention policies, backfills, monitoring, and OpenTelemetry-based observability.
Cross-Functional Collaboration: Partner with data science, forecasting, and platform teams to ship features safely behind feature flags (LaunchDarkly, Unleash).
Engineering Leadership: Raise engineering standards across the team through rigorous code reviews, technical design documents, and mentoring.
What you bring
Experience: 6+ years of software or data engineering experience, including 3+ years of experience building and maintaining production data pipelines.
Python Mastery: Strong Python development skills with a commitment to writing clean, tested code (using pytest, nox, or tox).
Orchestration Expertise: Deep experience with workflow orchestration (Airflow DAG design, operators, scheduling, and failure handling) or modern alternatives like Dagster and Prefect.
Data Ecosystems: Hands-on experience with distributed compute (Spark) and cloud data ecosystems (AWS EMR/Athena/S3, Databricks, Snowflake, GCP).
Storage & Modeling: Solid SQL and data-modeling skills, alongside practical experience with NoSQL databases, caching layers, and message queues (MongoDB, Redis, Kafka, RabbitMQ).
Infrastructure Familiarity: Comfort deploying and managing containerized workloads on Kubernetes using Docker and Helm.
Production Ownership: A proven track record of owning production pipelines, including participating in on-call rotations, incident response, and root-cause analysis.
Communication: Clear written technical communication and a history of documenting architectural decisions.
Nice to have
Scraping Infrastructure: Experience managing large-scale web datacollection and distributed crawler infrastructure.
Data Science / ML Concepts: Familiarity with record linkage, entity resolution, fuzzy matching, and NLP preprocessing (spaCy, NLTK).
Multi-Cloud Pipelines: Hands-on experience building multi-cloud datapipelines (e.g., bridging GCP/BigQuery and AWS).
ML Ops: Experience designing data pipelines that directly feed machine learning or forecasting systems.