Sharing real-time relevant data with utility consumers - Saving costs while helping the planet
Changing consumer behavior with timely relevant information and the right incentives has the potential to significantly reduce electricity consumption, leading to a win-win situation for everyone- cost savings for consumers, helping utility companies offset the intensifying energy demands on their aging infrastructures and reducing environmental impact.
Opower (acquired by Oracle in 2016) is a leading provider of customer engagement and energy efficiency cloud services to utilities. Its clients numbering over 85, include eight of the US’ 10 largest utilities. The company’s consumer engagement platform which pairs behavioral science with big data analytics reaches more than 18 million homes across North America, Europe, and Asia.
The platform presents insights into household energy use through informative dashboards, alerts, incentives, and household comparisons. The information delivered through mail, email, text messages, and web generates heightened awareness of the energy consumption and impact on consumers’ wallets and thereby incentivises consumers to reduce energy consumption.
For instance, customers alter their behavior to use energy during off-peak times, swap out older and inefficient appliances, and anticipate factors such as weather in order to save energy and money.
However, achieving these benefits is not easy.
Dealing with escalating data volumes
The primary focus area for utility companies is their physical infrastructure. Managing the data that is collected from that infrastructure is secondary.
Questions such as “How do I manage my flow of electricity? How do I make sure the lights are on?” are what utilities are concerned about, noted Eric Chang, Opower’s technical lead for data services at the time (as of 2013).
Advanced metering infrastructure (AMI) technologies such as smart meters, thermostats, and other in-home devices lead to an escalation in data volume, further complicating the situation. These devices generate and send providers frequent voltage and temperature readings. For example, a smart meter may generate 100 readings per day, per household. When that is multiplied by thousands of customer households for just one utility provider, the volumes of data are enormous. Data size is further compounded by related data streams including demographics data, smart appliances and sensors, weather data, consumer behavior information, and social media data. Utilities are usually not equipped to manage these kinds of volumes of data.
Data is only valuable if is processed and analysed. Opower recognised that processing and analysing this information would provide the utilities with much-needed insights into customer’s energy consumption that could be used to drive energy savings, reduce demand at peak times, adopt new rate structures, support smart meter roll outs, and lower cost to serve.
Shifting to a scalable data platform
As utilities are unable to cope with the vast volumes of smart meter data, Opower’s service model seeks to deliver that scalability, with the cost scaling based on use.
Opower’s first data management platform was a MySQL open source relational database management system (RDBMS). It was adequate with a small number of utilities clients.
As Opower’s client base and consumption data grew and its products offered more functionality, the company needed to adopt a new architecture –one that could scale to process and store large amounts of data without the cost and operational overhead of its relational databases.
Mr Chang said, “For us that new architecture included Apache Hadoop and [Apache] HBase.” In 2011, the company began moving to a distributed services-oriented architecture.
Opower deployed Cloudera Enterprise as the enterprise data hub to expand its big data capabilities and to store, transform, and query time series and other data. Key components of Opower’s architecture include CDH and HBase. Apache Hive serves as Opower’s data warehouse.
The enterprise data hub improved processing speed and efficiency, and the ability to easily pull data in parallel from multiple sources. In addition, HBase enabled Opower to deliver insights at scale, in real time, to any user.
HBase serves as a highly scalable data store that is built on top of Hadoop and offers fast, random read/write access to users and applications. Raw energy consumption data is batched regularly and imported into HBase, analytics are performed, and the results are stored there.
Immediate consumers of the data stored in HBase include web front ends that directly forward queries from end users and other internal processes, for example, processes that generate customer emails or paper mailings.
Opower manages its Hadoop cluster through Cloudera Manager.
“Cloudera Manager provides a lot of tools that enable us to know exactly what the cluster is doing at a given time,” said Tim Luo, a former software engineer (as of 2013) on the Opower data services team. The tool allows Opower’s team to respond to alerts quickly, manage and tweak configurations, restart servers, and ensure that overall cluster health is good.
Chang explains the closed loop system that makes it easy for Opower to correlate the impact of those efforts.
“The utilities provide us with meter reads, we perform analysis, we target a certain number of customers to deliver that analysis to, the customers act on that information and change their behavior, the utilities provide new meter reads, we compare those to the original reads, and we measure the actual impact in energy savings in that targeted population.”
Minute changes in the behavior of individual consumers, leading to say two to three percent energy savings, can add up and make a big difference when they are driven across several millions of households.
The company has helped its utility partners save energy consumers more than US$320 million in utility bills – and generated nearly three terawatt hours in energy savings. That is enough energy to power every household in Salt Lake City and St. Louis for an entire year.
Today, Opower is using big data to engage customers and motivate them to save energy and money. Increasingly, utilities will use big data to fuel insights that will drive new opportunities for innovation and efficiency in their energy delivery.
Opower does not only influence consumer energy savings; it imposes a mindfulness on itself.
The Hadoop platform infrastructure allows Opower to be much smarter about how it utilise its resources across the platform. Opower also worked closely with Dell on acquisition of hardware to ensure lower power consumption drives.
Opower continuously evolves its products so that as devices, communication channels and preferences, and customer behaviors change, its clients can respond effectively. Opower is constantly innovating, measuring the outcomes, and using that information to improve the platform.
Cloudera plays an important role in enabling this continuous innovation.
A key advantage of Cloudera is its open source aspect and the new functionalities which come out of that. Cloudera focuses not just on understanding the software, but understanding how to use it and how to interact with the open source community to get the most value from it.
Mr Drew Hylbert, Opower’s former vice president of technology and infrastructure (as of 2013), said about Cloudera, “They are an excellent partner both in keeping the systems going and delivering new technology on top of HBase. As we continue to scale, we know we can rely on Cloudera to help us understand how to grow – the impact on our cluster, how we need to provision, and what configurations need to be tweaked to accommodate that additional scale.”