This piece was originally published in the April 2018 issue of electroindustry.
Caption: Digital operations may reduce the number of control wires by more than 80 percent, reducing wiring complexity and resulting risks for operations and maintenance personnel. Photo courtesy of ABB
Dave Sterlace, Global Head of Data Center Technology, ABB Inc., and Mr. Sterlace advises a diverse data center client base on digitalization strategy.
In the past 20 years, the data center industry has grown from single server rooms to one-million-square-foot buildings with tens of thousands of servers connected to the grid at transmission-level voltages that use 100-plus megawatts of power—and this new scenario is only a beginning.
This growth is due in great part to the transition from analog to digital techniques and devices. The internet, with its ubiquitous high-speed connections, has become an indispensable facet of everyday life. Some analysts anticipate more than 20 billion connected devices by 2020. The proliferation of web-enabled and web-connected devices allows relationships and insights that were previously impossible. This Internet of Things (IoT) will require additional data centers and new ways of connecting them.
On a very basic level, the mere connection of a device to the internet is useful only if you’re able to benefit from that link. For example, if I need only certain real-time data, without archiving or acting upon it, then why bother with an internet connection? If a circuit breaker is not connected, we know only if it’s on or off. Additional data may tell us why it’s on or off. Thus, a connected device may troubleshoot problems or optimize processes.
New techniques and devices that are behind this connectivity explain why this is such an exciting time to be a part of the electrical industry.
Concerns with Digitization
Digitization is not without legitimate concerns. Taking industrial controls designed for closed loop communications and exposing them to the internet presents a security risk. In order to be connected to the internet, a device needs a media access control (MAC) address, which is typically assigned by a gateway. It may work as a bridge between an in-house industrial network like ModBus and a transmission control protocol/internet protocol (TCP/IP) network.
One concern is that existing control networks either have no security or only minimal security built into them. Ars Technica recently reported the cybervulnerability of a major manufacturer’s programmable logic controllers (PLCs). Direct web-connected devices (especially on the consumer side) raise serious questions, like how much cybersecurity do you think you’ll get with a $79 webcam?
Another consideration is how to communicate. The various industrial protocols all have their merits, but many are either proprietary or hierarchical. The ideal would be something that was open and able to communicate peer to peer in order to reduce latency issues. One solution that seems to be emerging as a frontrunner is the IEC 61850 standard, which was initially developed for smart grid applications. It offers open protocol, peer-to-peer communications, and growing acceptance by industry.
One benefit of digitization is the repeatability, transportability, and consistency of signals. For a control system, circuit protection, or process automation, this is very beneficial. For repeatability in overcurrent settings, most data centers use microprocessor-based trip units on their power circuit breakers. Analog signals also decay over longer distances, making them hard to transmit across a large industrial plant or data center. With digital signals, transmission isn’t limited by distance. Finally, the widespread use of TCP/IP networks allows plug-and-play compatibility for ease of troubleshooting and upgrading systems.
Safer, Smaller Infrastructure
In the case of data centers, operators are also concerned about exposing personnel to hazardous energy levels. An industry estimate puts the electrical density of a data center between 30 and 100 times the energy per square foot (or meter) of a commercial building, resulting in more switchboards and switchgear as well as higher fault levels.
Many clients use medium voltage (MV) deep within their data centers rather than just at service entrances. This creates situations where operations personnel are exposed to more MV and higher energy equipment than ever before. Digital intelligent electronic devices (IEDs) make it possible to change settings easily, even remotely. Furthermore, within the low-voltage control compartment of MV switchgear, digitization allows the standard copper control wiring to be replaced with low-energy and fiber-optic circuits, nearly eliminating hazards to maintenance personnel. A further benefit of using IEDs in control systems is that they signal when there is a breakage, so no “false positives” when it comes to safety.
Because of IEDs’ flexibility and use of process signals, they are significantly smaller than traditional wound potential transformers (PTs) and current transformers (CTs) for instruments/relay controls. An MV breaker is just a switch and needs relays (e.g., overcurrent) in order to function, unlike low voltage, where the sensing is built into the breaker. IEDs are also more configurable. Each PT or CT must be manufactured for the specific voltage and current involved. When applied to traditional current and voltage sensing applications, ABB found that variations range from 5,700 (i.e., the result of calculating all the possibilities) to seven. Imagine what that can do for lead times, troubleshooting, and critical spare parts. For example, in weight alone, transitioning from traditional PTs and CTs to process-level digital sensors can save more than 300 pounds per use and reduce the entire switchgear lineup by almost 30 percent versus a traditional drawout-style construction of 15kV class switchgear.
Flexible, Predictive Operations
One of the lofty goals of the electrical system in a data center is to be scalable like the IT equipment. Of course, with hard-wired equipment, this presents a considerable challenge. However, the communications capability, as well as some new approaches in power protection and control, can begin that journey.
A clear benefit of the IEC 61850 digital communications standard is the ability for communications between peers. Many control systems are hierarchical and don’t scale well enough to the demands of the hyperscale data centers. Because of the peer-to-peer nature, timely decision-making can be kept at a local level and more critical systemwide decisions can be done at a super- or hypervisory level. This also frees up network traffic for the more critical/impactful decisions. Data from individual devices can also be shared across the network for analysis and archival purposes. If cloud connected, this can be a very powerful tool across a fleet of devices, systems, and sites.
This affects power distribution. If the digital devices can be configured remotely, in near real time, that can allow a dynamic load shed/add capability to the switchgear. While this may have been done in the past, with digital control it can be done within the trip unit of the circuit breaker, without the complexity of additional PLCs and hard-wiring, and it can be reconfigured as necessary or desired. Some relay and protection operations in MV switchgear, using the benefits of fiber-optic connections, have been able to reduce the number of control wires by more than 80 percent!
If data gathered from the IEDs is archived in a central location, you can begin building a data “lake” in order to gain insight into your operations. A current example is using sound to understand when a motor bearing is about to fail. The scenario plays out like this: a motor emits X sound at Y frequency; by analyzing data over hundreds of installations, the motor has a 90 percent chance of failure within Z days. This can change maintenance from a break/fix or calendar-based model to a more proactive one, while not replacing items that aren’t in danger of failing. Similar data can be collected from circuit breakers and other electrical devices. A recent example from Facebook and Schneider Electric reports using IoT temperature sensors to ensure the bolts of a busway system remain within tightness specifications.
VR, AR, AI, and the Future
Although technology is advancing at a tremendous pace, one sure way to be wrong is to predict the future. However, some newer technologies that are not yet in widespread use are certainly worth considering, and their impact will be tremendous.
Virtual reality (VR), for one, is coming to maturity in the industry. It allows a technician to assess, diagnose, configure, and repair a system in a setting where real practice (e.g., a nuclear plant) may be cost- or safety-prohibitive. A technician can become a subject matter expert without ever leaving his or her desk.
Augmented reality (AR), where images are projected in a heads-up display over reality, can help complete the task. Interestingly, some of the DIY companies are looking at this as a major differentiator to help their customers install a faucet, for example.
Finally and famously, Google turned its considerable artificial intelligence (AI) engine on its HVAC system at a data center as part of a 20 percent project. The savings in energy was 40 percent without doing anything else to the system.
With data center infrastructure employing innovative techniques and devices, it has certainly moved into the digital age.
 “IoT Connected Devices to Reach 20.4 billion by 2020, Says Gartner,” Which-50, February 8, 2017, https://which-50.com/iot-connected-devices-reach-20-4-billion-2020-says-gartner
 Sean Gallagher, “Vulnerable industrial controls directly connected to Internet? Why not?” Ars Technica, January 25, 2018, https://arstechnica.com/information-technology/2018/01/the-internet-of-omg-vulnerable-factory-and-power-grid-controls-on-internet
 Rich Miller, “Facebook: Open Sharing Was Key to Addressing Arc Flash Incidents,” Data Center Frontier, October 19, 2017, https://datacenterfrontier.com/facebook-open-sharing-was-key-to-addressing-arc-flash-incidents
 “Google’s DeepMind trains AI to cut its energy bills by 40%,” Wired, July 20, 2016, http://www.wired.co.uk/article/google-deepmind-data-centres-efficiency