Role Description
Job Role: Senior Data Engineer
Job Location: Any UST Location
Notice Period: Immediate Joiners Only
Job Summary
We are seeking a highly skilled
Senior Data Engineer with 6+ years of experience in designing, developing, and optimizing large-scale data platforms and pipelines. The ideal candidate will have strong expertise in
Databricks, Apache Spark, ETL/ELT development, cloud data engineering, and data modelling. This role involves building scalable data solutions, implementing modern Lakehouse architectures, and collaborating with cross-functional teams to support data-driven business initiatives.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and distributed data processing frameworks.
- Build and optimize high-performance big data solutions using Apache Spark with Python, Scala, or SQL.
- Develop and manage Data Lake and Lakehouse architectures on cloud platforms such as AWS, Azure, or GCP.
- Implement batch and real-time data ingestion pipelines from diverse structured, semi-structured, and unstructured data sources.
- Optimize Databricks workloads through performance tuning, cluster configuration, workload management, and cost optimization.
- Create and maintain reusable Databricks notebooks, workflows, jobs, and automation frameworks.
- Leverage Delta Lake features including ACID transactions, schema evolution, partitioning, and data versioning.
- Design efficient data models to support analytics, reporting, and machine learning workloads.
- Collaborate with Data Scientists, Data Analysts, Product Teams, and Business Stakeholders to deliver scalable data solutions.
- Implement data quality checks, monitoring frameworks, and automated validation processes.
- Ensure compliance with enterprise data governance, security, privacy, and regulatory standards.
- Troubleshoot production issues, perform root cause analysis, and drive continuous platform improvements.
- Contribute to CI/CD implementation and DevOps best practices for data engineering solutions.
Required Skills & Qualifications
- 6+ years of experience in Data Engineering and Big Data technologies.
- Strong hands-on experience with Databricks and Apache Spark.
- Proficiency in Python, Scala, and SQL for data processing and transformation.
- Experience designing and implementing ETL/ELT pipelines in enterprise environments.
- Strong knowledge of Data Modeling, Data Warehousing, and Lakehouse architectures.
- Experience with Delta Lake, data partitioning, schema management, and performance optimization.
- Hands-on experience with cloud platforms:
- Amazon Web Services (AWS)
- Microsoft Azure
- Google Cloud Platform (GCP)
- Knowledge of distributed computing concepts and large-scale data processing.
- Experience implementing CI/CD pipelines and deployment automation.
- Strong troubleshooting, debugging, and performance tuning skills.
- Excellent communication and stakeholder management abilities.
Preferred Qualifications
- Experience with streaming technologies such as Spark Structured Streaming, Kafka, or Event Hubs.
- Knowledge of Data Governance, Data Quality, and Data Security frameworks.
- Experience with Infrastructure as Code (Terraform, CloudFormation, ARM Templates).
- Databricks, AWS, Azure, or GCP certifications are preferred.
- Exposure to MLOps and advanced analytics platforms is a plus.
Skills
Data Engineering, Databricks, Data Processing, Data Modeling
