djhworld

thoughts

Daniel James Harper

Enthusiastic, passionate developer with 16 years of software development experience in enterprise and personal projects, including data engineering and web applications. I strive to seek roles in creative, forward thinking companies that offer personal development and challenging work.


Experience

Senior Software Engineer, Cloudflare

March 2020 -
London/Remote, UK

Key technologies

Golang, Kubernetes, Clickhouse


Senior Software Engineer, BBC

June 2018 - February 2020
London, UK

Main responsibilities

Developing and maintaining “big data” analytics systems that help the BBC manage costs and deliver the best experiences to the iPlayer consuming public.

Key technologies

Java, Presto, Beam/Flink, Amazon Web Services (Kinesis, Lambda, EMR, S3, DynamoDB, EC2, Aurora, SQS, Athena), Hadoop, Python, Node.js, Parquet file format

Recent projects

Developing a stream processing system to collect anonymised A/V event data from iPlayer clients and help visualise this to interested parties through a web based visualisation tool. Written in Java 8, using technologies such as Kinesis, Apache Beam and Athena to process, aggregate and query the data.

Developing a batch based log processing system to help the BBC monitor A/V distribution over the internet. This consumes logs from a variety of Content Delivery Networks (CDNs), enriches them with additional information, then aggregates the data for use by a web based visualisation tool. This system ingests 20 billion rows a day, and has results available for analysis within minutes of the source arriving. Written in Java 8 and makes use of cloud technologies such as AWS Lambda, S3, Aurora, EC2 and DynamoDB. This replaced a batch processing system that used Hadoop on AWS.

Developing a stream processing system that the BBC use to monitor live streams from the iPlayer, and alert operations teams when something has gone wrong. This system aggregates 4000-7000 events per second and can alert teams within one minute of detected aberrations occurring. Written in Node.js and makes use of cloud technologies such as AWS Kinesis and Lambda.

Other achievements

Published a blog post on the BBC Internet blog about a project I worked on within the team.


Software Engineer, BBC

July 2014 - May 2018
London, UK

Developing and maintaining “big data” analytics systems that help the BBC manage costs and deliver the best experiences to the iPlayer consuming public.


Applications Developer, Sony Computer Entertainment Europe

August 2010 - July 2014
London, UK

Main responsibilities

Developing and maintaining enterprise software for the European territories of the SCE group, using a Java/JEE based development stack.

Key technologies

Java, J2EE, Spring, Spring MVC, Oracle 11g, Perforce, Weblogic

Summary

My role as a developer was to develop and maintain a suite of internal facing applications based on the needs of the business.

Notable projects included: -

  • Taking on Lead Developer role on a project to help the business to increase revenue on voucher sales to retailers. This involved designing and building an application using Spring MVC to help retailers engage earlier on upcoming products and manage orders. Since going live the business put over €12m worth of vouchers through the system
  • Designing and implementing a new logistics application using Spring MVC and MyBatis frameworks to manage shipments and truck movements. This was used extensively through the run up to the Playstation 4 launch.


Applications Developer, Atos

August 2008 - July 2010
Nottingham, UK

Main Responsibilities

Developing and maintaining applications for company clients within the public and private sector using Java/J2EE, including: -

  • Natural England/DEFRA
  • Travelex
  • Network Rail


Education

University of Lincoln

September 2005 - July 2008
BSc (Hons) Games Computing
First Class Honours (1:1)

Dissertation (Final year project)

Is a steering orientated driving agent more effective than those that are pre-trained?

This involved developing an AI system that allows a vehicle to drive around a given track unaided by any prior training. Written C++ and uses the OpenGL API.


Lakeview College Sixth Form Centre

September 2003 - July 2005
A-Levels
Computing (A), Media Studies (C), Geography (D)


Courses

BBC

  • AWS Essentials
  • Architecting on AWS
  • Big Data on AWS
  • Introduction to Google Cloud
  • Distributed Systems (Caches)
  • SSL/TLS and Internet PKI
  • Effective Use of Kanban

Sony Computer Entertainment Europe

  • Java Enterprise Edition

Atos

  • Introduction to PL/SQL


Technical Attributes

Languages I enjoy using: Java, Go
Operating Systems I’m comfortable with: OSX, Ubuntu Linux, CentOS
Datastores I’ve had experience with: DynamoDB, S3 (with Hadoop/Hive, Athena, Presto), AWS Aurora, Oracle 11g, Postgres 9.3
Cloud/Web services: Amazon Web Services (Kinesis, Lambda, EMR, S3), Heroku


About Me

Outside of programming and technical activities, I’d like to emphasise my other interests and pursuits, including:

  • A life long passion for videogames, born from the early days of the Amiga 1200.
  • Writing, some of my blog posts have reached the front page of Hacker News (see djharper.dev)
  • Being an avid reader of a wide variety of books, both nonfiction and fiction. My favourite book of all time is 1984 by George Orwell.
  • Listening to live radio and podcasts, mainly themed around comedy, documentaries, technology and video games.
  • Winner of “Best application” award at the Guardian “SXSW Hackday” 2011.


Personal Projects

I have a number of projects that I enjoy working on in my spare time. These include:


Contact Me

See my Stack Overflow page for more information https://stackoverflow.com/users/story/199278