Lead Data Engineer

DSIT • Full-time • Darlington, London, Manchester, Wolverhampton • £59.59k - £70.01k / year • 3d ago

Job summary

Here at the Ministry of Housing, Communities & Local Government (MHCLG), we work on things that make a real difference to people’s lives. 

Whether it's through the homes we live in, the work of our local councils, or the communities we’re all part of, our work is at the top of the political agenda. We have ambitious and far-reaching outcomes to achieve this year and, if you’re thinking of joining us, there’s never been a more exciting time. 

We have 3,500+ staff who are based in 20 offices across the UK. 

AI technology is rapidly evolving and there is immense opportunity to harness this capability to transform delivery at MHCLG. We are looking for a passionate and enthusiastic lead data engineer to join our team and be at the leading edge of AI delivery in a central government department, to deliver high-impact projects at pace to improve outcomes and productivity.

The new AI directorate at MHCLG will drive forward AI development and adoption across the department and support partners in local government. As a lead data engineer, you will unlock new approaches through the adoption of AI tools and techniques to help enable faster and more informed decision making. You will be motivated, flexible, enthusiastic and passionate about utilising data and AI responsibly for the public good.

We particularly welcome candidates from an ethnic minority background and other underrepresented groups to apply, as we work to continually improve our ability to represent the places and communities we support through our work.

You will join a multi-disciplinary team of technical experts (including data engineers, data scientists, developers, delivery managers, service owners) with an ambitious and exciting remit to harness the potential of AI, working on high impact projects that deliver operational efficiencies and support policy development through the rapid prototyping and scaling of products and services. You will work collaboratively across the organisation, interacting with a range of stakeholders to explore, build and maintain foundational AI capability and champion the role and impact of data engineering in the process.

Find out more about what it's like to work in a digital, data and technology role at MHCLG including our culture, ways of working, career progression and staff benefits.

Job description

As the Lead Data Engineer, you’ll:

work closely with data scientists, developers, and the wider team of DDaT expertise to develop robust, repeatable processes for managing and automating routine and repetitive tasks for data flows between teams
lead workstreams to identify and deliver data pipelines for AI prototyping and scaling, applying knowledge of systems integration
identify areas of innovation in data tools and techniques and recognise appropriate timing for adoption
support and enable effective data management, ensuring documentation, data dictionaries, and metadata are readily available across a range of stakeholders
confidently define and build robust data models and ETL processes to support AI delivery in line with best practice and standards
establish enterprise-scale data integration procedures across the data development life cycle to ensure decisions and actions are based upon reliable and accurate data
continuously improve delivery by identifying potential efficiencies and savings in existing processes, ensuring resources are utilised appropriately. Lead on collaboration and building capability while supporting the continuous development and upskilling of the wider technical team
communicate complex technical issues clearly and concisely to the team and to senior leaders to aide decision-making. Champion data engineering across the organisation and more widely across government

Person specification

As the Lead Data Engineer, you’ll:

understand the concepts and principles of data modelling and lead on the delivery of relevant data models
have extensive working knowledge and experience with ETL processes and data integration, including the use of APIs, data exchange with numerous systems, SQL experience and technical database expertise
have the ability to build data pipelines using cloud technologies such as Azure Data Factory/Databricks and be experienced writing parameterised code such as SQL/PL-SQL and/or Python/Spark
be able to develop integration procedures across all stages of the data development life cycle while continually improving processes and efficiency and be able to identify issues in data pipelines and manage their resolution
utilise key stakeholder management skills to effectively manage expectations, actively listen and interpret business needs, communicate progress, and navigate challenging conversations, all while fostering a service orientated culture around data
have strong experience working with cloud data storage (e.g. Azure blob storage / data lake), ideally involving the development and update of new and existing data sources across multiple environments (sandbox / test / production)
apply knowledge of standard development practices including code versioning and reviews, and ensuring coding standards are applied and familiarity using CI/CD / DevOps/ Github repository for versioning, code promotion and change management
have experience of defining and documenting data workflows across systems and operations, including capturing and documenting complex data definitions and business rules, applying data governance and metadata principles where appropriate

Behaviours

We'll assess you against these behaviours during the selection process:

Seeing the Big Picture
Making Effective Decisions
Communicating and Influencing
Working Together
Managing a Quality Service

Benefits

Alongside your salary of £59,586, Ministry of Housing, Communities and Local Government contributes £17,262 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.

Learning and development tailored to your role
An environment with flexible working options
A culture encouraging inclusion and diversity
A Civil Service pension with an employer contribution of 28.97%

Things you need to know

Selection process details

This vacancy is using Success Profiles (opens in a new window), and will assess your Behaviours and Experience.

MHCLG want to bring in a diverse workforce at all levels.

Our application system is designed to remove as much bias as possible from the recruitment system – this means that a hiring manager does not know your name, your details, see your whole application in one go (or have your CV at review stage unless stated otherwise).

Artificial Intelligence

Artificial Intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance for more information on appropriate and inappropriate use.

CV Declaration

We recruit based on your knowledge and skills, and not background, gender or ethnicity - this is called anonymous recruitment.

Please remove references to your:

name/title
educational institutions
age
gender
email address
postal address
phone number
nationality/immigration status

Most of our campaigns utilise multiple assessors and so it is possible that your application would be viewed by different assessors.

At sift, through your CV we will be assessing:

Experience

Technical

The interview will be of a blended nature consisting of the following success profiles elements:

Behaviour (Seeing the Big Picture, Making Effective Decisions, Communicating and Influencing, Working Together, Managing a Quality Service)

Technical

Prepared Presentation (10 minutes or less)

In full the campaign will test the below Success Profile Elements:

Experience: Yes - Experience questions will be based around the essential skills and criteria as listed in the person specification

Technical: Yes – Technical questions will be based around the essential skills and criteria as listed in the person specification

Behaviours: (Seeing the Big Picture, Making Effective Decisions, Communicating and Influencing, Working Together, Managing a Quality Service)

We do not consider direct CV applications – you must apply for this role via the application link on Civil Service Jobs:

Please note that near miss offers may be made at the lower grade to candidates who do not meet the grade criteria for this campaign.

Group 1 Digital and Data roles

MHCLG has implemented the Digital and Data capability framework for Group 1 roles. Applicants that are successful at the sift stage will be required to complete a capability assessment at interview.

MHCLG will honour completed capability assessments for this role from other Government Departments for existing Civil Servants. Please provide a copy of your capability assessment to the Hiring Manager when applying. If you have any queries on pay, please contact the Hiring Manager.

Salary is determined by performance at interview, within the range advertised. Each experience or technical skill is assessed between 1-3, representing working towards, at or above the job level requirements. You are awarded a proficiency level accordingly, and you will be given opportunities at least once a year to re assess your capability and progress through the pay scale within your grade. An additional digital allowance may be payable depending on level of assessed capability.

Candidates moving from another government department have the option to retain their current salary where the principle for implementation is there is no detriment where existing pay exceeds the indicative level. Where individuals are at a lower salary than their assessed level, they will receive an upward adjustment.

Grade 7 Group 1 Digital and Data salary

The salary for this role will between £64,979 - £70,011 (London), £59,586 - £64,641 (National) depending on the proficiency level assessment. 
An additional digital allowance may be payable depending on the level of assessed capability up to £9,000
For applicants in receipt of existing allowances, we will assess each case individually and aim to match the digital allowance implemented with no detriment.
For existing civil servants, the usual policy on level transfer and promotion will apply and is non-negotiable.

Please note that the average employer pension contribution is based upon the National minimum salary for this role. Should your agreed starting salary for this role be different, the average employer pension contribution will be calculated accordingly. If you are a Secondee, this will not apply as you will remain on your home organisation’s terms and conditions.

Benefits

Transfers across the Civil Service on or after 4 October 2018:

Any move to MHCLG from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax-Free Childcare. Determine your eligibility at https://www.childcarechoices.gov.uk.

For further information about the benefits available to MHCLG employees, please see the attached Candidate Pack.

Geographical Location:

Manchester
London
Wolverhampton
Darlington*

There may be opportunities for candidates to work flexibly depending on the business needs. This will be discussed with the vacancy manager on a case-by-case basis if you are successful for the role.

*Please note: The Darlington Economic Campus is a pioneering new cross-government hub which will bring together people across departments and public organisations to play an active role in the most important economic issues of the day. The work of the Campus will make a real difference to people both across the UK and internationally. There will be substantial career opportunities and exciting prospects - a career at the Campus means you will be working at the heart of Government, with access to the benefits and fantastic opportunities offered by the civil service. This role is based at MHCLG and we will be joined on the campus by:

HM Treasury
Department for International Trade
Department for Business, Energy and Industrial Strategy
Office for National Statistics
Department for Education

For further information on the DEC, please take a look at the attached DEC candidate pack.

Sift and Interview dates

Sifting is envisaged to take place from 5th August 2025 with interview dates to be confirmed. All interviews are currently being held remotely via videocall.

Reserve List

In the event that we identify more appointable candidates than we currently have posts available, we will hold applicant details on a reserve list for a period of 6 months from which further appointments can be made. This may include roles at a lower grade. Candidates placed on a reserve list will be informed of this. Those candidates who do not wish to remain on the reserve list should contact recruitment@communities.gov.uk to be removed from the reserve list.

SC (Security Check):

Important note

Please note that successful candidates will need to pass the Security Check – this requires you to have been resident in the UK for the past 5 years. Please refer to the MHCLG Notes on Security Clearance section of our Candidate Pack for further information on Security Check (SC).

Candidates should also note that with effect from 1st August 2018 the department will also check all applicants who are successful at interview, against the Internal Fraud Database (IFD) held by the Cabinet Office. In accordance with the Civil Service Internal Fraud Policy, any applicant who is included on the IFD will be refused employment by MHCLG Please see the Candidate Pack for further information on the Internal Fraud Database.

Candidate Pack Information

Please see attached Candidate pack for further information.

Before starting your application it’s very important to make sure that you are eligible to apply and meet the Civil Service nationality requirements. All candidates are expected to read the information provided in the MHCLG candidate pack regarding nationality requirements and rules

Internal Fraud Database

The Internal Fraud function of the Fraud, Error, Debt and Grants Function at the Cabinet Office processes details of civil servants who have been dismissed for committing internal fraud, or who would have been dismissed had they not resigned. The Cabinet Office receives the details from participating government organisations of civil servants who have been dismissed, or who would have been dismissed had they not resigned, for internal fraud. In instances such as this, civil servants are then banned for 5 years from further employment in the civil service. The Cabinet Office then processes this data and discloses a limited dataset back to MHCLG as a participating government organisations. MHCLG then carry out the pre employment checks so as to detect instances where known fraudsters are attempting to reapply for roles in the civil service. In this way, the policy is ensured and the repetition of internal fraud is prevented.

For more information please see- Internal Fraud Register

Feedback will only be provided if you attend an interview or assessment.

Security

Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).

See our vetting charter (opens in a new window).

People working with government assets must complete baseline personnel security standard (opens in new window) checks.

Nationality requirements

This job is broadly open to the following groups:

UK nationals
nationals of the Republic of Ireland
nationals of Commonwealth countries who have the right to work in the UK
nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in a new window)

Working for the Civil Service

The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants.

We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window).

The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria.

The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy.