Junior Data Engineer

As a Data Engineer, you will play a crucial role in designing, developing, and maintaining the data infrastructure and systems within an organization. You will work closely with cross-functional teams, including data scientists, analysts, and software engineers, to ensure the efficient flow, storage, and accessibility of data for analytical and reporting purposes.

Responsibilities:

• Data Pipeline Development: Designing and implementing scalable and reliable data pipelines to extract, transform, and load (ETL) data from various sources into data warehouses or data lakes. This involves understanding data requirements, identifying relevant data sources, and building automated workflows to ensure data quality and integrity.
• Data Modeling and Integration: Developing and maintaining data models, schemas, and integrations to facilitate data ingestion, transformation, and aggregation. This includes designing database schemas, defining data structures, and establishing efficient data integration processes between different systems.
• Data Quality Assurance: Implementing and maintaining data quality checks and validation procedures to ensure accuracy, completeness, and consistency of data. This involves developing data profiling techniques, anomaly detection mechanisms, and data cleansing routines to identify and resolve data quality issues.
• Performance Optimization: Monitoring and optimizing the performance of data pipelines, databases, and data processing systems. This includes identifying and resolving bottlenecks, improving query performance, and implementing caching strategies to enhance overall data processing efficiency.
• Data Security and Governance: Collaborating with the data governance and security teams to establish data access controls, data encryption mechanisms, and data privacy measures. Ensuring compliance with data protection regulations and best practices in data management and security.
• Collaboration and Documentation: Working closely with cross-functional teams to understand their data requirements and provide technical guidance on data-related matters. Documenting data pipelines, data flows, and data infrastructure architectures to facilitate knowledge sharing and maintain system documentation.
• Emerging Technologies: Keeping up-to-date with the latest trends and advancements in data engineering, big data technologies, cloud platforms, and distributed computing frameworks. Evaluating and recommending new tools, frameworks, and technologies to improve data engineering processes and infrastructure.

Qualifications:

• Bachelor’s degree in computer science, Information Systems, or 3-5 years of relevant job experience.
• Proven experience as a Data Engineer or a similar role, preferably in a large-scale data-driven environment.
• Strong programming skills in languages, specifically Databricks and Pyspark, and others such as Python, Java, or Scala.
• Experience with data integration tools, ETL frameworks, and workflow management systems (e.g., Apache Airflow).
• Proficiency in working with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• Familiarity with distributed computing frameworks (e.g., Hadoop, Spark) and cloud platforms (e.g., AWS, Azure, GCP).
• Knowledge of data modeling concepts, data warehousing, and dimensional modeling.
• Understanding of data governance, data security, and data privacy principles.
• Strong analytical and problem-solving skills, with attention to detail.
• Excellent communication and collaboration abilities, with the capacity to work effectively in cross-functional teams.

Clearance Statement: Applicants selected for this position will be subject to a government security investigation and must meet eligibility requirements for access to classified information. Only US citizens are eligible for a security clearance. For this position, Intelligent Waves will consider only applicants with security clearances or applicants who are eligible for security clearances.