Data Engineer

We are seeking a motivated and experienced Data Engineer to join our dynamic team and play a pivotal role in shaping our data infrastructure. The ideal candidate will have a proven track record in designing, building, and maintaining robust data pipelines and warehouses. You will leverage your expertise in cloud technologies (AWS, Snowflake), Python, SQL, ETL/ ELT tools, API integrations, and modern data engineering practices like DBT and CI/CD to deliver high-quality, secure, and scalable data solutions that adhere to stringent data quality and governance standards.

Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines using AWS, Snowflake, Python, SQL, and DBT to ingest, process, and transform large volumes of data from diverse sources, including databases, APIs, and third-party systems, ensuring optimal performance, scalability, and data quality.
• Build and optimize data warehouses and data lakes to support business intelligence, reporting, and analytics initiatives, with a focus on data quality, security, and governance.
• Implement and maintain ETL/ELT processes using DBT to ensure data accuracy, consistency, and timeliness, adhering to strict data security protocols and data quality standards.
• Develop and maintain API integrations to extract and load data from various sources, ensuring seamless data flow, integration with internal and external systems, and adherence to data quality guidelines.
• Collaborate closely with Data analysts, BI developers, and Analytics engineers to understand their data requirements and provide them with reliable, secure, accessible, and high-quality data.
• Monitor and troubleshoot data pipelines and API integrations, proactively identifying and resolving issues to ensure data availability, integrity, and quality, while maintaining data security best practices.
• Schedule and orchestrate the jobs using tools like Airflow or proficiency working in event-driven architecture and lambdas to trigger jobs.
• Implement data governance policies and procedures to ensure compliance with regulatory requirements and industry standards.
• Implement and maintain CI/CD pipelines to automate the testing, deployment, and monitoring of data pipelines and DBT models.