articledjh

thoughts

Daniel James Harper

Software engineer with 17 years of experience building distributed systems, data infrastructure, and web applications. Skilled in systems design, stream processing, and working across edge and core systems at scale.


Experience

Senior Systems Engineer, Cloudflare

March 2020 -
London/Remote, UK

Main responsibilities

Developing and maintaining systems related to Cloudflare’s API Shield product suite.

Responsibilities include:

  • Systems Design
  • Working across edge systems (latency sensitive, per-request security features and data enrichment) and core systems (batch processing, high throughput data pipelines, API services)

Key technologies

Golang, Rust, Clickhouse, Kafka


Senior Software Engineer, BBC

June 2018 - February 2020
London, UK

Main responsibilities

Developing and maintaining “big data” analytics systems that help the BBC manage costs and deliver the best experiences to the iPlayer consuming public.

Key technologies

Java, Presto, Beam/Flink, Amazon Web Services (Kinesis, Lambda, EMR, S3, DynamoDB, EC2, Aurora, SQS, Athena), Hadoop, Python, Node.js, Parquet file format

Recent projects

Developing a stream processing system to collect anonymised A/V event data from iPlayer clients and help visualise this to interested parties through a web based visualisation tool. Written in Java 8, using technologies such as Kinesis, Apache Beam and Athena to process, aggregate and query the data.

Developing a batch based log processing system to help the BBC monitor A/V distribution over the internet. This consumes logs from a variety of Content Delivery Networks (CDNs), enriches them with additional information, then aggregates the data for use by a web based visualisation tool. This system ingests 20 billion rows a day, and has results available for analysis within minutes of the source arriving. Written in Java 8 and makes use of cloud technologies such as AWS Lambda, S3, Aurora, EC2 and DynamoDB. This replaced a batch processing system that used Hadoop on AWS.

Developing a stream processing system that the BBC use to monitor live streams from the iPlayer, and alert operations teams when something has gone wrong. This system aggregates 4000-7000 events per second and can alert teams within one minute of detected aberrations occurring. Written in Node.js and makes use of cloud technologies such as AWS Kinesis and Lambda.

Other achievements

Published a blog post on the BBC Internet blog about a project I worked on within the team.


Software Engineer, BBC

July 2014 - May 2018
London, UK

Developing and maintaining “big data” analytics systems that help the BBC manage costs and deliver the best experiences to the iPlayer consuming public.


Applications Developer, Sony Computer Entertainment Europe

August 2010 - July 2014
London, UK

Developed and maintained internal enterprise applications for European territories using Java/J2EE, Spring, and Oracle. Projects included retail management systems and logistics applications.


Applications Developer, Atos

August 2008 - July 2010
Nottingham, UK

Developed and maintained Java/J2EE applications for public and private sector clients.


Education

University of Lincoln

September 2005 - July 2008
BSc (Hons) Games Computing
First Class Honours (1:1)

Dissertation (Final year project)

Is a steering orientated driving agent more effective than those that are pre-trained?

This involved developing an AI system that allows a vehicle to drive around a given track unaided by any prior training. Written C++ and uses the OpenGL API.


Lakeview College Sixth Form Centre

September 2003 - July 2005
A-Levels
Computing (A), Media Studies (C), Geography (D)


Courses

  • Architecting on AWS
  • Big Data on AWS
  • Distributed Systems (Caches)
  • SSL/TLS and Internet PKI


Technical Attributes

Languages: Go, Rust, Python, Java
Operating Systems: macOS, Linux (Ubuntu, Debian, CentOS)
Datastores: Clickhouse, Postgres, Kafka, DynamoDB, S3, Presto/Athena
Cloud/Platforms: Amazon Web Services (Kinesis, Lambda, EMR, S3, EC2, Aurora), Cloudflare
Specialties: Distributed systems, stream processing, data pipelines, edge computing


About Me

Outside of programming and technical activities, I’d like to emphasise my other interests and pursuits, including:

  • A lifelong passion for video games, born from the early days of the Amiga 1200.
  • Writing technical blog posts, some of which have reached the front page of Hacker News (see djharper.dev)
  • Being an avid reader of a wide variety of books, both nonfiction and fiction. My favourite book of all time is 1984 by George Orwell.
  • Listening to podcasts, mainly themed around technology, documentaries, and video games.


Personal Projects

I have a number of projects that I enjoy working on in my spare time. These include: