Jobs in London - Data Engineer (Questionnaire) • Central London

Company: Our Future Health

Central London

We are looking for a Data Engineer to join our growing data engineering team at Our Future Health. Our data engineers work with our participant data (including various health records from the NHS, genetic data, Our Future Health’s own questionnaires) and build pipelines and systems to process it, control quality and create data releases for researchers. You’ll support the development of specific pipelines related to questionnaire and physical measurements data. Our Future Health will be the UK’s largest ever health research programme, bringing people together to develop new ways to detect, prevent, and treat diseases. We are a charity, supported by the UK Government, in partnership with charities and industry. We work closely with the NHS and with public authorities across all nations and regions of the UK. What you’ll be doing: We ask Our Future Health volunteers to complete a health questionnaire which is currently delivered through our website and has been completed by hundreds of thousands of people. The questionnaire data is processed using our data platform and made available to researchers working on approved research projects in our trusted research environment. We have some exciting plans to improve the way that the questionnaires are collected as well as the range of questionnaires and will also be looking to improve how the data are presented to researchers. Your key responsibilities will include but not be limited to: Support the technical product owner for questionnaire and physical measurements data releases, ensuring that requirements around schemas and data processing logic are well specified from a technical perspective. Play a supporting role in developing data pipelines and contributing to the code base. Ensure best practices are used in the team (CI/CD, designing modular and reusable code, etc) Anticipate problems that could occur with data pipelines and take action to prevent them. Identify and describe problems when they occur and be able to develop solutions that address them for the short and long term. Support effective multidisciplinary working with other teams, helping teams understand the role of data engineering in Our Future Health and how to work effectively together. Keep abreast of best practice in data engineering across industry, research and Government, and identify opportunities to bring these into Our Future Health. RequirementsYou know how to communicate to and between technical and non-technical stakeholders as well as facilitate discussions within a multidisciplinary team including scientists, software engineers, product managers and other data engineers. To succeed in this role, you will also have some of the following skills: Experience working in an agile development team. Understanding and working knowledge of information governance and data security approaches appropriate for sensitive health data. Familiarity with a range of data structures, data standards and data exchange formats, with experience with questionnaire data being desirable (in addition, OMOP and FHIR experience would be advantageous). You have experience with the design and deployment of complex data schemas. You have experience in validating and cleaning complex datasets. Experience building and maintaining robust, scalable and efficient pipelines capable of processing very large amounts of data from one or multiple systems. Highly proficient in Python. Highly proficient in SQL. Experience working with data lakes; experience with Spark, Databricks. Understanding of common data transformation and storage formats, e.g. Apache Parquet. Good understanding of cloud environments (ideally Azure), and workflow management systems (e.g. Dagster, dbt, Airflow, Prefect). Understanding of containerisation (e.g. Docker) and deployment (e.g. Kubernetes). Follow best practices like code review, clean code and unit tests. Experience with version control and Git/GitHub. Benefits Up to £60,000 per annum basic salary. Generous company pension package with employer contributions of up to 12%. 30 days annual leave (plus bank holidays.) Individual development budget Flexible and remote working arrangements and a lovely new office in Holborn, Central London. Join us - let’s prevent disease together.