Data Engineer

The Data Engineer is responsible for building, maintaining, and ensuring data production pipelines run smoothly. This role will monitor data pipelines for our data analytics platform, proactively diagnose and resolve issues, enhance pipeline components as needed, and develop support tools to enable non-technical support staff to troubleshoot common issues over time. In addition, this role will implement methods to improve data reliability and quality as well as develop and test architectures that enable data extraction and transformation for predictive or prescriptive modeling. This role may perform these duties in Cloud, non-Cloud, and hybrid environments.

Essential Duties & Responsibilities:

• Analyze and organize raw data sets to meet both functional and non-functional requirements.
• Develop and maintain data sets.
• Improve data quality and efficiency.
• Create scalable, production-ready pipelines for data movement and transformation.
• Work with various stakeholders, including data, design, product, and executive teams, to assist with data-related technical issues.
• Implement and enhance support tools for monitoring and acting on data pipeline issues.
• Interpret trends and patterns.
• Conduct data analysis and report on results.
• Prepare data for prescriptive and predictive modeling.
• Build algorithms and prototypes.
• Combine raw information from different sources.
• Explore ways to enhance data quality and reliability.
• Identify opportunities for data acquisition.
• Develop analytical tools and programs.

Education and Experience:

• Bachelor’s degree in Computer Science, Information Technology, Mathematics, or another related field.
• Two (2) to four (4) years of software development experience with a focus on data engineering.
• Experience with Cloud-based deployment and data processing technologies (AWS Lambda, Step Functions, S3 storage, Simple Service (SQS), EMR clusters, CloudWatch monitoring, and CloudWatch insights).
• Experience with supporting data transformation and analytics solutions preferred.
• Experience in Healthcare Information Technology (HIT) is highly preferred.
• Working experience with the following languages: Spark, Python (2 years preferred), SQL (2 years), Kafka (preferred).
• Working experience with relational databases, queries, and optimization with a high emphasis on Postgres.
• Data engineering certification is a plus.

Skills, Knowledge, and Abilities:

• Technical expertise in data models, data mining, segmentation techniques, and software architecture.
• High-level understanding of data schemas for structured and unstructured data.
• Big Data experience (HDFS, Hive, HBase, M/R, Impala, Spark, HUE) preferred.
• Understanding of software architecture.
• Simple Notification Service (AWS SNS).
• Change Data Capture.
• Working knowledge of data lake architecture (Lambda architecture, Delta Lake, Apache Hudi).
• Pipeline orchestration (Apache Airflow, Apache NiFi).
• Strong verbal and written communication skills.
• Excellent attention to detail.
• Customer service skills with a high level of professionalism.
• Strong desire to learn.
• Basic computer skills, including Microsoft Outlook, Word, and Excel.
• Able to manage a variety of tasks concurrently.