SDE III

SDE III/IV -  Data Platform Engineering

Mission

Architect a self-serve Data "Platform-as-a-Product" powering InMobi's global-scale data ecosystem. Integrate OSS tools, proprietary services, and  Cloud/SaaS into unified infrastructure. Requires deep data engineering

expertise (batch/streaming pipelines, data modeling, query optimization, governance) combined with platform engineering to build production-grade  solutions for data engineers and analysts.

Core Responsibilities

Design & Development - Bridge OSS tools (Spark, Flink, Airflow, Iceberg), internal services, and cloud offerings into cohesive data platform infrastructure. Build intuitive platform integrations enabling push-button  data workflows.

Scale Engineering - Operate distributed systems processing petabytes of data daily. Own multi-region Kubernetes infrastructure with elastic scalability  and fault tolerance.

Performance Optimization - Optimize compute utilization (Spark/Flink clusters, Velox/Gluten acceleration) for large-scale batch and real-time  streaming with sub-second latency.

Observability & Data Quality - Build comprehensive telemetry (metrics, logs,traces) and data quality frameworks for 24/7 uptime. Enforce SLAs/SLOs with automated incident response and data validation.

Required Skills & Experience (Must-Have)

• 7–10 years building, optimizing, and operating production data platforms

• Deep data engineering fundamentals: data modeling, partitioning strategies, query optimization

• Distributed compute: Spark (PySpark/Scala), Flink streaming, performance  tuning at petabyte scale

• Data lake architecture: Iceberg table format, Polaris catalog, schema   evolution, time travel

• Orchestration: Airflow DAG development, dependency management, SLA  monitoring

• Data transformation: DBT modeling, testing, documentation, incremental  builds

• Data quality: Great Expectations, dqueue validation frameworks, drift  detection

• Query acceleration: Velox, Gluten integration, columnar formats (Parquet, ORC)

• Data governance: OpenMetadata catalog, lineage tracking, access control

• Kubernetes platform development: operators (Spark/Flink), Yunikorn    scheduler, multi-tenancy, autoscaling

• Cloud infrastructure: GKE multi-region clusters, GCS object storage,    hybrid cloud/on-prem architecture

• Programming: Python, PySpark, Scala for data pipelines and platform  tooling

• IaC: Terraform, Helm, GitOps for reproducible deployments

• CI/CD: Automated testing, deployment pipelines for data platform  components

Good-to-Have

• Experience building cloud data platform / control plane development

• Advanced observability: Prometheus/Grafana, Loki, Firehydrant integration

• Real-time streaming: Kafka integration, exactly-once semantics,  backpressure handling

• Cost optimization: Resource allocation, query optimization, storage   tiering strategies