About The Company
D Square Consulting Services Pvt Ltd is a leading technology solutions provider renowned for delivering innovative and reliable IT services to clients across various industries. With a commitment to excellence, D Square specializes in infrastructure management, cloud computing, data analytics, and enterprise solutions. The company prides itself on fostering a collaborative environment that promotes continuous learning and professional growth. Dedicated to leveraging cutting-edge technologies, D Square aims to empower organizations to achieve operational efficiency and competitive advantage through tailored IT strategies and solutions.
About The Role
We are seeking a highly skilled Big Data Engineer with extensive experience in supporting and maintaining large-scale data processing systems. This remote position involves working during the night shift from 11:00 PM to 7:00 AM EST, with occasional weekend support, to ensure the seamless operation of our market regulation data infrastructure. The successful candidate will be responsible for monitoring, troubleshooting, and automating data pipelines and infrastructure components in a hybrid cloud environment primarily utilizing AWS and Azure. This role requires a proactive approach to system reliability, performance optimization, and incident management, ensuring the integrity and availability of critical regulatory data systems.
Qualifications
The ideal candidate will possess a minimum of 5 years of experience as a Big Data Engineer or in a similar support role. A strong background in systems administration across Linux and Windows environments is essential, complemented by hands-on experience with cloud services such as AWS and Azure. Proficiency in SQL, scripting languages like Python and Bash, and familiarity with workload automation tools are required. Certifications in AWS, Linux administration, or workload automation are advantageous. A bachelor’s degree in Computer Science, Information Systems, or a related field is mandatory, with additional certifications considered a plus.
Responsibilities
- Monitor and support large-scale data operations and batch processing workflows within AWS cloud environments for market regulation purposes, promptly identifying and resolving processing failures.
- Run and troubleshoot complex SQL queries in Postgres, Oracle, or equivalent relational databases to validate data integrity and processing status.
- Automate monitoring, alerting, and remediation processes using scripting languages such as Python, Bash, and PowerShell to enhance operational efficiency.
- Implement and manage job scheduling using tools like JAMS or Broadcom AutoSys, resolving conflicts and ensuring timely execution of scheduled tasks.
- Maintain and support hybrid infrastructure comprising enterprise Linux and Windows systems across on-premises and cloud environments, ensuring system stability and security.
- Operate and manage cloud resources in AWS and Azure, including container orchestration and deployment, to support scalable and reliable data processing.
- Support CI/CD pipelines using Jenkins, GitLab, and related tools, troubleshooting deployment issues and adhering to best practices in version control and automation.
- Implement comprehensive monitoring and observability solutions, analyzing logs and performance metrics to identify bottlenecks and optimize system performance.
- Document operational procedures, automation scripts, and runbooks; participate in disaster recovery (DR), business continuity planning (BCP), and SOX compliance testing and reporting.
- Lead or collaborate on process improvement initiatives focused on automation, containerization, and AI-driven observability, working closely with DevOps, Security, and Application teams to enhance system resilience.
Benefits
At D Square Consulting Services Pvt Ltd, we believe in supporting our employees’ growth and well-being. We offer competitive compensation packages, including health insurance, paid time off, and opportunities for professional development. Our remote work environment provides flexibility to balance personal and professional commitments. Additionally, employees have access to training programs, certifications, and participation in innovative projects that foster skill enhancement and career advancement. We prioritize a collaborative and inclusive workplace culture that encourages innovation and recognizes individual contributions.
Equal Opportunity
D Square Consulting Services Pvt Ltd is an equal opportunity employer. We are committed to creating a diverse and inclusive work environment where all employees are valued and treated with respect. We do not discriminate based on race, ethnicity, gender, age, religion, disability, sexual orientation, or any other protected characteristic. We encourage applicants from all backgrounds to apply and join our team dedicated to technological excellence and innovation.
