Role Summary
We are looking for a Data Engineer responsible for operating, monitoring, and enhancing data ingestion and processing pipelines within a Google Cloud–based data platform. The role combines day-to-day platform reliability, data validation, and development activities, supporting continuous platform evolution, new data onboarding, and automation initiatives. The ideal candidate is hands-on, detail-oriented, and comfortable working across production and pre-production environments.
Key Responsibilities
A) Data Ingestion Operations & Monitoring
- Monitor and ensure successful daily and monthly data ingestion across multiple upstream systems.
- Oversee scheduled jobs and queries, validating successful execution and investigating failures or delays.
- Maintain and enhance data ingestion monitoring checks, including:
- Updating checks when new datasets or tables are introduced.
- Adapting monitoring logic based on frequency, country, or business dimensions.
- Track and investigate unprocessed or delayed files in cloud storage, performing root cause analysis and coordinating resolution with relevant stakeholders.
- Manage and follow up on incident or service requests related to missing or delayed data until fully resolved.
B) Automation & Platform Reliability
- Support and troubleshoot automation workflows used to capture ingestion exceptions and feed monitoring logic.
- Ensure automation processes run reliably and intervene quickly in case of failures.
- Handle ad-hoc automation requests to reduce manual effort and improve operational efficiency.
- Proactively identify recurring issues and implement improvements to strengthen pipeline stability.
C) Data Validation & Reporting
- Perform data validation and reconciliation activities by comparing datasets across environments or systems when required (typically on a recurring or monthly basis).
- Develop, maintain, and execute Python-based validation scripts to identify discrepancies and data quality issues.
- Prepare recurring operational and reconciliation reports in line with agreed schedules.
D) Development & Enhancements
- Support environment setup and migration activities within Google Cloud, including configuration changes between production and pre-production environments.
- Create, test, and deploy new tables and datasets following platform standards.
- Implement schema changes and enhancements to existing tables while ensuring backward compatibility and data integrity.
- Enable onboarding of new countries, business units, or data domains into existing ingestion and monitoring frameworks.
- Continuously refine ingestion and validation logic to align with evolving business requirements.
Technologies & Tools
- Google Cloud Platform (GCP)
- SQL
- Python
- ODBC
- Postman
- Workflow and automation tools
Issue tracking tools (e.g., ticketing systems)
