Principal Data Engineer
The Enterprise Data Platform team is looking for a Principal Data Engineer to help revolutionize how we think about and deliver customer insights, playing an important role in the future success of the company.
The ideal candidate will be a platform evangelist, having a deep appreciation for the business as well as strong technical skills. You should be self-directed, a creative problem solver, and able to adjust to a fast-paced environment. This position plays the important role of bridging the gap between our business stakeholders and our technical data teams.
What you'll do:
Architecting large-scale data intelligence solutions, performing architectural assessments, crafting architectural options and analysis around Snowflake Data Warehouse and/or Google BigQuery
Design and evolve an open lakehouse on AWS: S3 with Apache Iceberg/Delta table formats, a medallion architecture (bronze → silver → gold), and Trino as the federated query engine.
Finalize preferred solution alternative working with IT and Business stakeholders to build data ingestion pipeline using tools like AWS EMR, Snowpipe, AWS DMS, etc.etc
Need to have professional knowledge of cloud platforms like AWS, GCP, etc.
Developing ETL pipelines in and out of data the warehouse using but not limited to Python, Scala,SnowSQL, etc.
Own orchestration patterns in Dagster: asset-based pipelines, code-location design, cross-asset lineage, sensors and schedules, partitioning and backfills.
Set dbt standards for the enterprise data model: layering (staging/intermediate/marts), testing, documentation, and CI for model changes.
Hands on experience with building productionized data ingestion and processing pipelines using Java, Spark, Scala, Python
In-depth understanding of Data Warehouse/ODS, ETL concept and modeling structure principles
Tune Trino for performance and cost: query planning, catalog configuration (Glue/Hive/REST), resource groups, and scaling on Kubernetes.
Set technical direction and write architecture decision records for review by peers and leadership.
Mentor engineers across distributed teams in the US and India, and raise the bar through code and design reviews.
Who you are:
12+ years of experience working with data in the following ways: SQL development, Shell Scripting, Python, Snowflake, AWS, software engineering or database development
Ideal candidate has led platform architecture for a multi-team data organization
Experience developing end-to-end data pipelines to transform and process data for reporting layer
Experience with large-scale data sets using both structured and unstructured data
Production experience with at least two of: Trino/Presto, Iceberg/Delta/Hudi, Dagster/Airflow, dbt, Spark
Full grasp of continuous integration and delivery, including branching strategies, automated deployment and functional automation within CI and CD Pipelines
Strong communication skills across both business and technical communities
Strong data modeling: dimensional/Kimball, conformed dimensions, slowly changing dimensions (SCDs), and modeling for multiple brands or tenants
Experience with Kubernetes (EKS), infrastructure as code (Terraform/CDK), and containerized data services
Bachelor or Master’s degree in Computer Science, Information Technology or Equivalent
Performing in an agile, self-directed team environment
Nice to have: experience with semantic layers (Cube, dbt Semantic Layer), data catalogs, ML/identity resolution, marketing or ad-tech data, and big data
Experience with billing data and systems is a big plus
#LI-HK1 #LI-Remote
The specific salary offered to a candidate may be influenced by a variety of factors including the candidate’s experience, their education and work location. In addition, some roles may be eligible for additional on target commission pay or bonus. The compensation package includes health and welfare benefits including paid leave.
Pay Transparency - All Full Time Employees
$137,800—$172,250 USD