Staff Engineer - Databricks

You Will:

Data Architecture and Design: Designing and overseeing the architecture of scalable and reliable data platforms, including data pipelines, storage solutions, and processing systems

Data Modelling and Management:Developing and implementing data models, ensuring data quality, and establishing data governance policies

Data Pipeline Development: Building and optimising data pipelines for ingesting, processing, and transforming large datasets from various sources

Performance Optimisation: Identifying and resolving performance bottlenecks in data pipelines and systems, ensuring efficient data retrieval and processing

Technology Evaluation and Innovation: Staying abreast of emerging data technologies and exploring opportunities for innovation to improve the organisation’s data infrastructure

Troubleshooting and Problem Solving: Diagnosing and resolving complex data-related issues, ensuring the stability and reliability of the data platform

Data Security and Compliance: Implementing data security measures, ensuring compliance with data governance policies, and protecting sensitive data

Perform other duties as assigned

You Have:

Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

10+ years of experience in data engineering or a similar role.

Enterprise SaaS software solutions with high availability and scalability

Solution handling large scale structured and unstructured data from varied data sources

Experience in building and maintaining data platform systems such as distributed compute, data orchestration, distributed storage, streaming infrastructure ensuring scalability, reliability, efficiency and security

Working with Product engineering team to influence designs with data, AI and analytics use cases in mind

In depth experience in System design involving large Petabytes of data with Databricks Lakehouse

Experience in modern AI/Data infrastructure patterns, Semantics layer Organizing data for AI agents (metadata, context)

AWS, GCP,  Snowflake and Data pipeline frameworks like Airbyte/Airflow

Programming languages like Python, SQL, and potentially Java or Scala

Modern software engineering practices like Kubernetes, CI/CD, IAC tools (Preferably Terraform), Observability, monitoring and alerting

Solution Cost Optimisations and design to cost

Driving engineering excellence initiative

Legally eligible to work in India on an ongoing basis

Fluency in English is required