We are seeking a dynamic professional to fill our Data Engineering role who will be responsible for designing, developing, and maintaining the data architecture, infrastructure, and pipelines that enable Northern Arc to collect, store, process, and analyze large volumes of data. This role will play a crucial part in ensuring data availability, quality, and accessibility for data-driven decision-making.
Responsibilities
- Design, develop, and maintain data pipelines to ingest, process, and transform data from various sources into usable formats.
- Implement data integration solutions that connect disparate data systems, including databases, APIs, and third-party data sources.
- Create and manage data storage solutions, such as data lakes, data warehouses, and NoSQL databases.
- Optimize data storage for performance, scalability, and cost-efficiency.
- Establish data quality standards and implement data validation and cleansing processes.
- Collaborate with data analysts and data scientists to ensure data consistency and accuracy.
- Develop ETL processes to transform raw data into a structured and usable format.
- Monitor and troubleshoot ETL jobs to ensure data flows smoothly.
- Implement data security measures and access controls to protect sensitive data.
- Ensure compliance with data privacy regulations and industry standards (e. g., GDPR, HIPAA).
- Optimize data pipelines and queries for improved performance and efficiency.
- Identify and resolve bottlenecks in data processing.
- Maintain comprehensive documentation for data pipelines, schemas, and data dictionaries.
- Create and update data lineage and metadata documentation.
- Design data infrastructure to scale with growing data volumes and business requirements.
- Implement data recovery and backup strategies to ensure data availability and resilience.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver data solutions.
This job was posted by Rasika More from Northern Arc Capital.
