Job description

Azure Data Engineer

Insurance

Hybrid Working : 2 days per week in Tunbridge Wells, Kent and the remainder from home

We currently have an opportunity to join one of our growing engineering teams as a Data Engineer for a client within the insurance sector.

Working with Big Data technologies, you will be responsible for collecting, storing, processing, and analysing huge sets of data in a cloud based data lake. Working alongside other Data Engineers and Data Architects your primary focus will be choosing optimal solutions to design, develop, implement, monitor & maintain key deliverables to support the businesses growing need for data consumption. You will also be responsible for integrating them with the architecture used across the company.

Enabling BI and Insight Analysts the ability to easily interrogate and navigate data in a meaningful way and helping them to create actionable insight for the business.
If you have 'Big Data' experience and able to conduct Enterprise Data modelling for Analytics, and are passionate about data and how impactful it can be then this may be the role for you.

The Data Engineer will be responsible for:

• Build ingestion pipelines in the cloud based Data Lake to accommodate multiple data sources in line with the Groups Data Architects principles. This will need to be able to support real time streaming and batch ingestion.
• Partner with the Data Scientists, Architects, software developers, and business experts to understand how data needs to be translated, modelled and presented for consumption. Striving for One version of truth.
• Interrogate and model data from multiple sources, translating into an Enterprise useable logical data model fit for Analyst consumption.
• Through data and understanding of business processes, advise on Analytical uses in Insight
• Provide guidance and translation to Analysts on how to interpret and navigate the data for specific use case.
• Help the Data Architect to create an overview of the Data Lineage (from data flows, data transformations inside applications to Analytical output)
• Providing estimates of work received and ratifying these estimates based on experience/facts.
• Provide clear documentation of the business rules embedded in the modelled data ensuring full transparency of data transformations
• Identify and champion process improvements including data content, quality, standardization, automating manual processes and optimizing data delivery. Apply Machine Learning and Automation techniques to processes within Lake helping to detect Quality issues with data.

Key competencies, skills and knowledge

• Experience in the following would be of benefit: Python, Spark, R , SQL, .net, Java, C++
• Big Data experience (Microsoft Azure, Hadoop, Google BigQuery etc)
• Working knowledge of Azure Data Factory, Data Lake, DataBricks and Synapse Analytics
• Working Knowledge of Power BI and providing optimised data sets that can be fully utilised by the product.
• Able to translate business requirements into detailed technical design
• Experience with integration of data from multiple data sources
• Machine Learning, AI and Automation techniques for Data Management & Analytics
• Awareness of Data Management best practice, including Data Lifecycle Management across Core IT & Big Data ecosystems as well as Data Privacy & Security constraints