A smart meter may generate 100 readings per day, per household. When that is multiplied by thousands of customer households for just one utility provider, the volumes of data are enormous. Data size is further compounded by related data streams including demographics data, smart appliances and sensors, weather data, consumer behavior information, and social media data
Changing consumer behavior with timely relevant information
and the right incentives has the potential to significantly reduce electricity
consumption, leading to a win-win situation for everyone- cost savings for
consumers, helping utility companies offset the intensifying energy demands on
their aging infrastructures and reducing environmental impact.
by Oracle in 2016) is a leading provider of customer engagement and energy
efficiency cloud services to utilities. Its clients numbering over 85, include eight
of the US’ 10 largest utilities. The company’s consumer engagement platform
which pairs behavioral science with big data analytics reaches more than 18
million homes across North America, Europe, and Asia.
The platform presents insights into household energy use
through informative dashboards, alerts, incentives, and household comparisons.
The information delivered through mail, email, text messages, and web generates
heightened awareness of the energy consumption and impact on consumers’ wallets
and thereby incentivises consumers to reduce energy consumption.
For instance, customers alter their behavior to use energy
during off-peak times, swap out older and inefficient appliances, and anticipate
factors such as weather in order to save energy and money.
However, achieving these benefits is not easy.
Dealing with escalating
The primary focus area for utility companies is their
physical infrastructure. Managing the data that is collected from that
infrastructure is secondary.
Questions such as “How do I manage my flow of electricity? How
do I make sure the lights are on?” are what utilities are concerned about,
noted Eric Chang, Opower’s technical lead for data services at the time (as of 2013).
Advanced metering infrastructure (AMI) technologies such as
smart meters, thermostats, and other in-home devices lead to an
escalation in data volume, further complicating the situation. These devices generate
and send providers frequent voltage and temperature readings. For example, a
smart meter may generate 100 readings per day, per household. When that is multiplied
by thousands of customer households for just one utility provider, the volumes
of data are enormous. Data size is further compounded by related data streams
including demographics data, smart appliances and sensors, weather data,
consumer behavior information, and social media data. Utilities are usually not
equipped to manage these kinds of volumes of data.
Data is only valuable if is processed and analysed. Opower
recognised that processing and analysing this information would provide the
utilities with much-needed insights into customer’s energy consumption that
could be used to drive energy savings, reduce demand at peak times, adopt new
rate structures, support smart meter roll outs, and lower cost to serve.
Shifting to a
scalable data platform
As utilities are unable to cope with the vast volumes of
smart meter data, Opower’s service model seeks to deliver that scalability,
with the cost scaling based on use.
Opower’s first data management platform was a MySQL open
source relational database management system (RDBMS). It was adequate with a
small number of utilities clients.
As Opower’s client base and consumption data grew and its
products offered more functionality, the company needed to adopt a new
architecture –one that could scale to process and store large amounts of data
without the cost and operational overhead of its relational databases.
Mr Chang said, “For us that new architecture included Apache
Hadoop and [Apache] HBase.” In 2011, the company began moving to a distributed
Opower deployed Cloudera
Enterprise as the enterprise data hub to expand its big data capabilities
and to store, transform, and query time series and other data. Key components
of Opower’s architecture include CDH
Apache Hive serves as Opower’s data
The enterprise data hub improved processing speed and
efficiency, and the ability to easily pull data in parallel from multiple
sources. In addition, HBase enabled Opower to deliver insights at scale, in
real time, to any user.
HBase serves as a highly scalable data store that is built
on top of Hadoop and offers fast, random read/write access to users and
applications. Raw energy consumption data is batched regularly and imported
into HBase, analytics are performed, and the results are stored there.
Immediate consumers of the data stored in HBase include web
front ends that directly forward queries from end users and other internal
processes, for example, processes that generate customer emails or paper mailings.
Opower manages its Hadoop cluster through Cloudera Manager.
“Cloudera Manager provides a lot of tools that enable us to
know exactly what the cluster is doing at a given time,” said Tim Luo, a former software engineer (as of 2013) on the Opower data services team. The tool allows Opower’s
team to respond to alerts quickly, manage and tweak configurations, restart
servers, and ensure that overall cluster health is good.
Chang explains the closed loop system that makes it easy for
Opower to correlate the impact of those efforts.
“The utilities provide us with meter reads, we perform
analysis, we target a certain number of customers to deliver that analysis to,
the customers act on that information and change their behavior, the utilities provide
new meter reads, we compare those to the original reads, and we measure the actual
impact in energy savings in that targeted population.”
Minute changes in the behavior of individual consumers,
leading to say two to three percent energy savings, can add up and make a big
difference when they are driven across several millions of households.
The company has helped its utility partners save energy
consumers more than US$320 million in utility bills – and generated nearly
three terawatt hours in energy savings. That is enough energy to power every
household in Salt Lake City and St. Louis for an entire year.
Today, Opower is using big data to engage customers and
motivate them to save energy and money. Increasingly, utilities will use big
data to fuel insights that will drive new opportunities for innovation and
efficiency in their energy delivery.
Opower does not only influence consumer energy savings; it
imposes a mindfulness on itself.
The Hadoop platform infrastructure allows Opower to be much
smarter about how it utilise its resources across the platform. Opower also
worked closely with Dell on acquisition of hardware to ensure lower power
Opower continuously evolves its products so that as devices,
communication channels and preferences, and customer behaviors change, its clients
can respond effectively. Opower is constantly innovating, measuring the
outcomes, and using that information to improve the platform.
Cloudera plays an important role in enabling this continuous
A key advantage of Cloudera is its open source aspect and
the new functionalities which come out of that. Cloudera focuses not just on understanding
the software, but understanding how to use it and how to interact with the open
source community to get the most value from it.
Mr Drew Hylbert, Opower’s former vice president of
technology and infrastructure (as of 2013), said about Cloudera, “They are an excellent
partner both in keeping the systems going and delivering new technology on top
of HBase. As we continue to scale, we know we can rely on Cloudera to help us understand
how to grow – the impact on our cluster, how we need to provision, and what configurations
need to be tweaked to accommodate that additional scale.”
Content from Cloudera customer success story, dated November 2013, on www.cloudera.com.
We release new articles daily on trending topics within technology and the public sector. Subscribe to have weekly digests of our articles conveniently sent to your email address.
Mövenpick Hotel and Convention Centre KLIA
One Farrer Hotel
Sheraton Towers Singapore
Putrajaya Marriott Hotel
Marina Bay Sands, Singapore
JW Marriott Jakarta