Wednesday, 1 April 2015

Forecasting demand using Big Data

As we walk, cycle or drive around Oxford, make telephone calls, send texts or emails and do our shopping, many of us are unaware of exactly how much data is being generated by our activities. "Big data" is a catch-phrase for describing the overwhelming volume, velocity and variety of this stream of information. Big data has the potential to provide many opportunities for the public and private sectors, offering a means of fusing different sources of information and supporting decision-making in real-time.

Perhaps the most interesting aspect of big data is how it deepens our understanding of human behaviour seen through the collective actions of many individuals. We tend to consume services following the temporal cycles in our everyday lives. There are three evident cyclical patterns based around the hour of day, the day of the week and the season of the year. All of these patterns can be seen in electricity consumption, call centre activity, internet usage, financial transactions, traffic flow and the use of healthcare services.

Fortunately the repetition of these patterns offers potential for accurate demand forecasting. Services can be delivered with greater efficiency if staff and limited resources are scheduled in order to meet forecasted demand. The National Grid has been balancing supply and demand for years and knows the value of accurate forecasts. If they get it wrong, the lights go out and everybody notices. While power outages still happen in many countries, we take it for granted in the UK that we have reliable access to electricity at all times. Amazingly, we are relatively tolerant of imbalances in supply and demand in other sectors and this may explain why sophisticated demand forecasting is not widely utilised.

Take healthcare, for example. The NHS has a target of seeing 95% of patients arriving in A&E within four hours. Until recently there was little information about the performance of our local hospitals or indeed how they compare with the rest of the country. Now weekly A&E data about the percentage of patients seen in four hours is available. This week the John Radcliffe Hospital A&E scored 87.1%, slightly less than the national average of 91.5%. Here is a chance for Oxford City to become smarter.

There are many opportunities to use big data and quantitative models to forecast demand, develop early warning systems and improve staff scheduling. The graph below shows the average hourly A&E arrivals at the John Radcliffe for different days of the week. We immediately see the hour of the day effect with low demand during the night and two peaks at 12:00 and 18:00. Most striking is the near doubling of arrivals in the early hours of Saturday and Sunday, which can be attributed to the effect of weekend partying and pubs closing at 11:00 on Friday and Saturday night. While A&E staff are well aware of the additional burden caused by weekend festivities, the data analysis paints a clear picture of its impact on arrivals.
Big data can facilitate a better understanding of social behaviour and the effect of the environment. Arrivals increase on bank holidays. Temperature is another important factor with arrivals increasing in warm weather. But this is just the start. Forecasts of extreme weather events and information about social events could be used to construct an accurate model.

New book: