Lead Data Engineer (London)
AND Introduction
At AND, we accelerate the development of digital capabilities. In practice, that means helping ambitious leaders and organisations build the teams, products, processes and even operational structures they need to close the digital skills gap within their organisation today, so that they thrive tomorrow.
Clients rely on our experience, agility and craft skills across tech and business strategy, software development and product management to address some of the toughest challenges facing their businesses.
We bring aboard thinkers, tinkerers, passionate software craftspeople and inspiring technologists to help us solve these challenges. Together, we’re united by a sense of pragmatism, purpose and a deeply-held belief that digital products and technology alone won’t transform a business or save the world: it’s the people that count.
About you
AND Digital are investing significantly in Data and Insight for 2021, we see these new roles as being an exciting opportunity to help shape a new engineering chapter within AND. It is a unique opportunity to work on some of the biggest data challenges in the UK alongside building and scaling a new team within AND.
As a Lead Data Engineer, you will spend your time helping clients achieve their data engineering needs. Primarily the development, deployment and maintenance of data pipelines in a fast moving, Agile environment, utilising a combination of both cloud and on premises tooling.
You’ll also make a real impact by taking an active role in the team’s Agile Development practices, technical decision making and development, generating value and continuously striving to improve the quality and reliability of our data and processes.
You will spend the majority of your time helping deliver client data engineering opportunities and supporting the growth of our junior-mid level technologists; both within AND and for our clients.
This will include being able to:
Influence the client on how they approach data engineering issues Collaborate with tech leads on system and data integration Write code to extract ingest or transform new and existing data from multiple sources Transform data in various ways to support data scientists and product analysts Build and design a scalable and extensible data architecture Conduct large scale batch and real-time processing from varying sources
To do that it’s essential you bring the following
At least 4+ year experience writing Python, R, SQL or Scala Experience with managing data at rest (traditional databases such as PostgreSQL and MongoDB as well as bulk data storage solutions such as Parquet, Avro, HDFS and similar) Experience with large scale data streaming solutions (using technologies such as Kafka, Kinesis and Pulsar) At least 1 year experience working with ETL technologies (such as Apache Spark or orchestration with Apache Airflow) Understanding of management of data from logging and monitoring tools (such as the ELK stack, Splunk and Open Telemetry) Experience with BI and Data warehouse tools (Tableau, Power BI, Snowflake and DataBricks) Understanding of the use cases for Data Lakes vs Data Lakehouses Good understanding and track record of delivering complex data solutions using Agile methods including Scrum, SAFe etc. Hands-on experience of modern software delivery, including CI/CD and DevOps practices A keen understanding of industry best-practice around standards, quality and continuous improvement in the field of Data Engineering Experience in the field of Data Engineering, Science or Analytics
It’s helpful if you also have:
Consultancy or professional services experience across a number of sectors Javascript/Typescript/NodeJS experience Experience with Hadoop, Spark, Redshift or Parquet Experience in Dimensional modelling and Data Warehouse patterns
Joining AND
From the work we deliver, to the way we serve and support our people, we work hard to ensure that there’s nowhere quite like AND. But joining a company is a two-way street: the fit has to work on both sides. So before you apply, here’s three key things to understand about us:
We’re built for people – like, real humans. Not ‘resources’ or ‘staff’. That means happiness and wellbeing really do matter to us, and we hate unnecessary hierarchy and bureaucracy. There’s no well-trodden path ahead: AND is growing fast and forging a new trail. That’s exciting, and gives us all the autonomy and opportunity we love – but bear in mind it also demands focus, patience and resilience. Diversity is a priority. After all, to build great products that a wide variety of different people love to use, we need a wide variety of people to help us build them. So diversity is more than a policy or a word: it’s business critical for us.