Lead Data Engineer
Lead Data Engineer (dbt / Databricks Migration)
Location: [Fully remote] - Fixed term Contract
.Monks Technology Services, part of Media.Monks and S4 Capital, is a global consulting firm mastering AI-powered transformations for the Fortune 100. We combine long-term strategic thinking, deep enterprise experience, and a human-centered approach to help clients transform business processes and dominate their industries.
As part of our commitment to inclusion and in compliance with Law 2466 of 2025 from Colombia, we have available spaces for people with disabilities.
About the Role
As a Lead Data Engineer, you’ll lead a collaborative engineering pod delivering a large-scale data platform modernization initiative. You will guide the migration of a legacy Python/Spark ETL framework into dbt Core models on Databricks, modernizing production reporting tables that support analytics, product, and business intelligence functions.
You will provide hands-on technical leadership while partnering closely with internal client engineers. This includes reverse-engineering existing ETL logic, designing scalable dbt models and staging layers, establishing testing and validation practices, and shipping changes through a CI-gated pull request workflow. You will lead the team in a structured, documentation-first, and highly collaborative environment.
Responsibilities
Lead the end-to-end migration of legacy Python/Spark ETL configurations into modular, maintainable dbt Core models
Provide hands-on technical leadership, mentorship, code review, and delivery guidance to data engineers
Partner with internal client engineers to define migration priorities, technical standards, and implementation plans
Reverse-engineer existing Python ETL logic and translate it into scalable dbt SQL models, staging layers, and reusable Jinja macros
Design foundational shared macros, dimension tables, and medium-to-high-complexity domain models
Implement dbt sources, incremental model patterns, and schema.yml tests, including not_null, unique, and relationships
Validate output parity between newly developed dbt models and legacy production pipelines before go-live
Establish and uphold engineering standards for model design, documentation, testing, and maintainability
Manage delivery through a Git-based, PR-reviewed workflow with automated dbt compile and dbt test CI/CD gates
Identify migration risks, resolve technical blockers, and communicate progress and tradeoffs to engineering stakeholders
Support scheduled dbt orchestration and production deployment within the Databricks ecosystem
Collaborate effectively across distributed teams and time zones
Other duties as assigned
About You
Qualifications & Skills
5+ years of production experience with dbt Core; dbt Certified Developer certification preferred
Hands-on experience leading data engineering teams, mentoring engineers, reviewing code, and owning technical delivery
Advanced knowledge of dbt, including Jinja macros, sources.yml, schema.yml tests, reusable model patterns, and incremental models
Hands-on experience with Databricks, Delta Lake, Unity Catalog, and Databricks SQL Warehouses
Advanced SQL skills, including CTEs, window functions, complex transformations, and Jinja templating
Experience migrating custom or legacy Python-based ETL pipelines into modular, source-agnostic dbt models
Experience validating output parity between modernized models and legacy production pipelines
Proficiency with Git-based pull request workflows and CI/CD gates, such as GitHub Actions running dbt compile and dbt test
Strong technical leadership, stakeholder communication, problem-solving, and documentation skills
Experience with third-party connectors such as Fivetran or data sourced from externally managed lakehouses is a plus
Familiarity with media, e-commerce, or consumer analytics domains is a plus
Experience working asynchronously across distributed time zones is a plus
Exposure to Databricks Workflows for orchestrating scheduled dbt runs is a plus
.Monks Technology Services does not discriminate on the basis of race, sex, color, religion, age, national origin, marital status, disability, veteran status
#LI-Remote #LI-BC1