Data Engineer

Role Overview:

We are seeking a skilled Data Engineer with extensive experience in building scalable data
pipelines, ETL processes, and data infrastructure. The ideal candidate will have strong
expertise in Python, modern ETL tools, and AWS services, combined with the ability to work
with frontend technologies for data visualisation and reporting interfaces. Should
have Healthcare/Clinical trial experience

Required Skills:

• Data Engineering & ETL
• Python Development:
  • Strong proficiency in Python for data engineering.
  • Strong knowledge of PySpark for large-scale data processing.
• Modern ETL Tools & Frameworks:
  • dbt (data build tool) for data transformation.
  • Apache Airflow for workflow orchestration.
  • Great Expectations for data validation.
  • Experience with stream processing using Apache Kafka or AWS Kinesis.
• Data Warehousing & Lakes:
  • Experience building and maintaining data lakes.
  • Data modelling and schema design.
• AWS Expertise:
  • AWS Glue for ETL processes.
  • Amazon S3 for data storage.
  • Redshift for data warehousing.
  • Lambda for serverless applications.
  • Step Functions for workflow orchestration.
  • Route 53 and VPC for networking.
  • EMR for big data processing.
• Database Management:
  • Strong knowledge of MySQL.
  • Experience with complex queries and optimisation.
  • Data pipeline performance tuning.
  • Data quality monitoring and testing.
• Frontend Development:
  • Basic proficiency in React, Angular, or jQuery.
  • Experience creating data visualisation interfaces.
  • Understanding of HTML, CSS, and JavaScript.
• General Technical Skills:
  • Experience with CI/CD pipelines.
  • Proficiency with Git and version control.
  • Data governance and security best practices.
  • Documentation and technical writing.

Nice to Have:

• Advanced Data Tools:
  • Apache Beam.
  • Apache Flink.
  • Kubernetes for data infrastructure.
• DevOps Experience:
  • Infrastructure as Code (Terraform, CloudFormation).
  • Docker and container orchestration.
  • Monitoring tools (Prometheus, Grafana).

What We're Looking For:

• The ideal candidate will be a data engineering specialist who can:
  • Design and implement scalable data architectures.
  • Build efficient and maintainable data pipelines.
  • Ensure data quality and reliability.
  • Balance technical excellence with practical solutions.
  • Collaborate effectively with data scientists and analysts.