Peter Harrison presented at a recent breakfast event in Singapore. The Breakfast Dialogue was around the topic and journey of virtualisation. Peter spoke about how the department has made it journey from physical infrastructure, through to a more agile and flexible environment of virtualisation. I caught up with Peter on a brisk July day in Perth, Western Australia.
I wanted to discuss with Peter what the department had done around virtualisation. Understanding the reasons and benefits of how the department went about using virtualisation. Virtualisation in many ways was the in-house version of scalable resource management. Maximising the use of computer infrastructure as well as provide a flexible environment that could scale as needed.
Peter said, “My job is to make sure that our computers and applications are available for the business.” It was about “providing the tools that enabled the business of protecting children and support to vulnerable families”. His responsibilities were to provide the expertise in providing digital services that enabled the business to work more efficiently. Peter said, “The business does not care about the technology, they are focused solely on delivery of services to their clients, the vulnerable children and families of Western Australia”. It was his job to make sure that “the technical plumbing was working effectively”. The other was to preserve the infrastructure and software, that in times of failure, he could resolve the problem and return the department back to working order.
Implementation of virtualisation
In discussing Peter’s role at the department, I asked him how they went about implementing virtualising their systems. Peter pointed out “what we have been doing for a time, is the virtualisation our physical assets bit by bit”. The department had adopted a strategy which included a six-month review of all assets as they came to end of life”. Peter would review these assets, doing one of three things. One, if the asset was not needed the department would decommission and move on. Two, if the systems was needed it would be reviewed to see if it could be virtualized. If the answer was yes, then it would be added to the farm, decommissioning the old equipment. Doing this has lessened our overall costs, providing a more standard practices in managing and supporting this infrastructure. The department has reached close to 80-90% of the department’s infrastructure has been virtualized. Third, rarely where we do need to keep the infrastructure, we would then buy a replacement. Overtime this has been lessening. Peter said, “In total we have some 700 servers, a vast majority of them are nonproduction”. They consist of development, test and preproduction. “We have about 200 production servers, delivering all the services to the department”.
Biggest Advantage to the Department
One of the biggest advantages the department has benefited from, was an offer from one of their suppliers. It was upgrading all their enterprise servers to a data centre Operating System. This enabled Peter to roll out unlimited virtual servers for no added cost to the department. The adoption of virtualizing of servers was carried out in a more aggressive manner. Western Australian Government has an arrangement with their suppliers with a whole of Government agreement.
Where the cost and use of software is paid for once a year, referred to as a true up. Peter said, “This has enabled us to manager our cost of software with a one-off investment each year”. This has benefitted the department, but we were unable to true down. The department still paid the cost of licenses, even though they had reduced their count. “It was only at the end of the three year contract the department could true down on their licenses”. This was because of the renewal of the contract with their supplier. This agreement enabled the department to on sell the licenses to other departments. Thus enabling smaller departments to pick up those licenses, for the cost of renewal. Better spreading the software across government. Peter said, “It gave us the incentive to reduce our licenses and costs”.
As part of his journey, we discussed his work in the virtualisation of services. Peter said, “We have two companies providing services to the Department. One, manages our infrastructure, the other manages our key core applications. Our core applications, because of legislation gets many changes, other changes are from improvements”. The practice of the past, was to create systems and environments on request from the support company. On receiving that request, their infrastructure contractor would build a compilation of servers to meet that need. This could take a week or more. Next the department started to virtualize server and systems using a golden image, further reducing their time to build, sometimes down to an hour.
Next Stage was to Automate
Peter said, “The next stage was to automate the process, our applications contractor could then build their own environments. This could be done by using a pick and buy list on a web page, providing that service”. Peter has been looking to automate the entire process. One that could build the environments, but on a defined period reclaim those same resources. This would enable better management of my resources, licenses would be uninstalled, virtual space returned to the pool of available resources". Peter signalled, “We would be able to give them enough warning that their time was coming up. The systems would automatically reclaim the resources unless indicated by the developer”. One of the biggest advantages of this arrangement was the department could create as many Virtual Machines (VM’s) as they wanted, full licenses were not needed, as they could use developer licenses further lessening their costs.
The whole automation exercise has had a major impact in the delivery of their services. Freeing much of Peter's support resources from mundane day-to-day work. They were now free to work on other projects, introducing new technology. Peter said, “We introduced Lync last year and they are looking to expand it to include telephony for their unified communications platform”.
We next started to discuss Peter’s latest project that he was engaged in. Peter said, "The department is working on a major project, to removing a thin client technology from many of their regional sites." This technology was introduce many years ago when bandwidth was limited. It was designed to give access to their major systems with limited bandwidth and resources. This project was to replace that technology on over 800 personal computers. These computers were dotted all over Western Australia in 160 sites. To put some context to the size of this project. Western Australia is over 2.53 million square kilometres in size, no mean feat especially when some of these remote sites could be up to 200 kilometres from the nearest town. Peter said, “We are three quarters of the way through that, to be completed at the end of August this year”. The biggest challenge the department had was around training. Peter pointed out, “Because of the nature of the work we do our staff need to be fully trained. To do that effectively we would have to fly them around the state. So to reduce that cost we explored the area of eLearning and streaming video. Because of the arrangement we had with our thin client technology, this started to become a barrier”. With rolling out the full systems in the regions they have been able to give this training virtually, travel has been reduced at a notable savings to the department.
One of the biggest lessons learned was by taking it slowly and seeking expert advice. Their supplier of infrastructure support took on the exercise of looking at their current solution, then seeking alternatives. The department, first started with a pilot, using champions around the districts. They started a pilot in two districts, one down in the southeast of Western Australia, in a town called Esperance, some 700 kilometres from Perth was the first pilot site. The other, North of Perth some 1400 kilometres away in a town called Roebourne. These locations were chosen as they were near the extremes, having the pilots being pushed to reflect a truer testing. The other reason was they had champions in those locations with technical experience to support the pilots. The testing was carried out using the new configuration next to their current configuration. The pilots having proven successful, the department then went to roll out the new systems to other district, testing all the way. Peter said, “In one of the districts we involved the Director. That director joined the project board as a senior user”, this enabled a better uptake buy all. Peter pointed out, “The District Director approved the pilot and the new system environment”. The district directors comments were, “It's wonderful, it's fantastic, the way IT had approached the whole project was exemplary”. Peter said, “We got much positive feedback from him, these were his words, his recommendation was to continue with the full role out”.
My final question to Peter was if he were to talk with a peer in another country, what country would it be, and what he would want to know. Peter said’ “That’s a hard one, we have two unique challenges. One is the tyranny of distance. Our network is one of the largest in the world, in one single country with over a 160 sites". The size of Western Australia is over 2.5 million square kilometres. The departments head office is found in Perth with 350 people. Many of the sites around the country can range from 1 or 2 staff members, others range from 20-50 staff of a total complement of 2,500 staff in the department. These sites are usually in remote locations, some being over 200 kilometres from the nearest town. Many our children in care are indigenous decent and that brings it many challenges as well. Peter suggested, "I would be looking to talk with countries like India and China. They would have to deal with major distances and challenging conditions with communications”. I thanked Peter for his time and we went our separate ways.