SDE 2

SDE II - Data Platform - Lucknow

Mission

Contribute to building a self-serve Data Platform-as-a-Product enabling scalable, reliable, and easy-to-use data infrastructure across batch and streaming workloads.\

Core Responsibilities

Build and maintain scalable data pipelines (batch & streaming) using Spark/Flink

Develop integrations across Airflow, Iceberg, and cloud platforms

Apply best practices in data modeling, partitioning, and query optimization

Support Kubernetes-based data platform workloads and troubleshoot issues

Optimize pipelines for performance, cost, and reliability

Implement monitoring, logging, and data quality checks

Participate in on-call and incident resolution

Required Skills

3–6 years in data/platform engineering

Strong fundamentals: SQL, data modeling, partitioning, optimization

Hands-on with Spark (PySpark/Scala); exposure to Flink is a plus

Experience with Airflow (or similar orchestration tools)

Familiarity with data lake formats (Iceberg/Delta/Hudi)

Working knowledge of Kubernetes & cloud (GCP/AWS/Azure)

Programming: Python (must-have), Scala (good-to-have)

Exposure to CI/CD, Git, and IaC (Terraform/Helm basics)

Good to Have

Kafka / streaming systems experience

DBT, Great Expectations, or similar tools

Observability tools (Prometheus, Grafana)

Metadata/catalog tools (OpenMetadata/DataHub)

Success Metrics

Delivers reliable, production-ready pipelines

Improves performance and platform usability

Strong ownership, debugging, and collaboration skills