Preface xxi

Acknowledgments xxiii

1 An Overview of Reliability and Resiliency in Today’s Mission Critical Environment 1

1.1 Introduction 1

1.2 Risk Assessment 5

1.2.1 Levels of Risk 6

1.3 Capital Costs versus Operation Costs 7

1.4 Critical Environment Workflow and Change Management 9

1.4.1 Change Management 10

1.5 Testing and Commissioning 11

1.6 Documentation and Human Factor 16

1.7 Education and Training 20

1.8 Corporate Knowledge Transfer – the Means to Securing Tomorrow’s Critical Infrastructure 21

1.9 Operation and Maintenance 24

1.10 Employee Certification 25

1.11 Standards and Benchmarking 25

1.12 What is a Mission Critical Engineer 26

1.13 Conclusion 28

1.14 An Overview of Reliability and Resiliency in Today’s Mission Critical Environment - Questions to Consider 28

2 Energy and Cyber Security and its Effect on Business Resiliency 31

2.1 Introduction 31

2.2 Risks Related to Information Security 36

2.3 Electro Magnetic Pulse and Solar Flares 42

2.4 How Risks Are Addressed 47

2.5 Use of Distributed Energy Resources and Generation 52

2.6 Documentation and Its Relation to Information Security 55

2.7 Smart Grid 57

2.8 Conclusion 60

2.9 Energy Security and Its Effect on Business Resiliency – Questions to Consider 60

3 Mission Critical Engineering with an Overview of Green Technologies 63

3.1 Introduction 63

3.2 Companies’ Expectations: Risk Tolerance and Reliability 65

3.3 Identifying the Appropriate Redundancy in a Mission Critical Facility 67

3.4 Improving Reliability, Maintainability, and Proactive Preventative Maintenance 69

3.5 The Mission Critical Facilities Manager and the Importance of the Boardroom 71

3.6 Quantifying Reliability and Availability 71

3.6.1 Review of Reliability Terminology 72

3.7 Design Considerations for the Mission Critical Data Center 73

3.7.1 Data Center Certification 74

3.8 The Evolution of Mission Critical Facility Design 76

3.9 Human Factors and the Commissioning Process 77

3.10 Short Circuit & Coordination Studies 79

3.11 Introduction to Direct Current in the Data Center 84

3.11.1 Advantages of DC Distribution 85

3.11.2 Lighting Updates 87

3.11.3 DC Storage Options 87

3.11.4 Renewable Energy Integration 88

3.11.5 DC and Combined Cooling, Heat & Power 89

3.11.6 Safety Issues 91

3.11.7 Maintenance 91

3.11.8 Education & Training 92

3.11.9 Future Vision 93

3.12 Containerized Systems Overview 93

3.13 Mission Critical Engineering with an Overview of Green Technologies - Questions to Consider 95

4 Mission Critical Electrical System Maintenance & Safety 103

4.1 Introduction 103

4.2 The History of the Maintenance Supervisor and the Evolution of the Mission Critical Facilities Engineer 105

4.3 Internal Building Deficiencies and Analysis 107

4.4 Evaluating Your System 108

4.5 Choosing a Maintenance Approach 110

4.5.1 Annual Preventive Maintenance 111

4.6 Safe Electrical Maintenance 112

4.6.1 Standards and Regulations 112

4.6.2 Electrical Safety: NFPA 70E Arc Flash Mitigation 114

4.6.3 Personal Protective Equipment (PPE) 117

4.6.4 Lockout/Tagout 126

4.7 Maintenance of Typical Electrical Distribution Equipment 127

4.7.1 Thermal Scanning and Thermal Monitoring 127

4.7.2 15 KV Class Equipment 129

4.7.3 480 Volt Switchgear 130

4.7.4 Motor Control Centers and Panel Boards 131

4.7.5 Automatic Transfer Switches 131

4.7.6 Automatic Static Transfer Switches (ASTS) 132

4.7.7 Power Distribution Units 132

4.7.8 277/480 Volt Transformers 133

4.7.9 Uninterruptible Power Systems 133

4.8 Being Proactive in Evaluating the Test Reports 134

4.9 Designing for Safety and Reliability 135

4.10 Conclusion 136

5 Standby Generators 137

5.1 Introduction 137

5.2 The Necessity for Standby Power 138

5.3 Emergency, Legally Required, and Optional Systems 140

5.4 Standby Systems That Are Legally Required 141

5.5 Optional Standby Systems 142

5.6 Understanding Your Power Requirements 142

5.7 Management Commitment and Training 142

5.7.1 Lockout/ Tagout (LOTO) 143

5.7.2 Training 144

5.8 Standby Generator Systems Maintenance Procedures 144

5.8.1 Maintenance Record Keeping and Data Trending 145

5.8.2 Engine 145

5.8.3 Coolant System 145

5.8.4 Electrical / Control System 146

5.8.5 Generator 146

5.8.6 Automatic and Manual Switchgear 147

5.8.7 Load Bank Testing 147

5.9 Documentation Plan 148

5.9.1 Proper Documentation and Forms 148

5.9.2 Record keeping 148

5.10 Emergency Procedures 149

5.11 Cold Start 150

5.12 Non-Linear Load Concerns 151

5.12.1 Line Notches and Harmonic Current 151

5.12.2 Voltage / Frequency Drop 152

5.12.3 Voltage / Frequency Rise 152

5.12.4 Frequency Fluctuation 153

5.12.5 Synchronizing to Parallel 154

5.12.6 Automatic Transfer Switch 154

5.13 Conclusion 155

6 Fuel Systems Design and Maintenance 157

6.1 Introduction 157

6.2 Brief Discussion on Diesel Engines 158

6.3 Bulk Storage Tank Selection 159

6.3.1 Aboveground Tanks 159

6.3.2 Modern Underground Tanks and Piping Systems 160

6.3.3 Fuel Receiving Tanks 161

6.3.4 Generator Sub-Base Tanks 161

6.4 Codes and Standards 162

6.5 Recommended Practices for all Tanks 163

6.6 Fuel Distribution System Configuration 168

6.7 Day Tank Control System 170

6.8 Diesel Fuel and a Fuel Quality Assurance Program 174

6.9 Conclusion 186

7 Power Transfer Switch Technology, Applications, and Maintenance 187

7.1 Introduction 187

7.2 Transfer Switch Technology and Applications 189

7.3 Types of Power Transfer Switches 191

7.3.1 Manual Transfer Switches 191

7.3.2 Automatic Transfer Switches 191

7.4 Control Devices 204

7.4.1 Time Delays 204

7.4.2 In-Phase Monitor 205

7.4.3 Test Switches 206

7.4.4 Exercise Clock 207

7.4.5 Current, Voltage and Frequency Sensing 207

7.5 Design Features 207

7.5.1 Close Against High In-Rush Currents 208

7.5.2 Withstand and Closing Rating (WCR) 208

7.5.3 Carry Full Rated Current Continuously 208

7.5.4 Interrupt Current 209

7.6 Additional Characteristics and Ratings of ATS 209

7.6.1 NEMA Classification 209

7.6.2 System Voltage Ratings 209

7.6.3 ATS Sizing 209

7.6.4 Seismic Requirement 210

7.7 Installation & Commissioning, Maintenance, and Safety 210

7.7.1 Installation & Commissioning 210

7.7.2 Maintenance & Safety 212

7.7.3 Maintenance Tasks 214

7.7.4 Drawings and Manuals 215

7.7.5 Testing & Training 215

7.8 General Recommendations 218

7.9 Conclusion 219

8 The Static Transfer Switch 221

8.1 Introduction 221

8.2 Overview 222

8.2.1 Major Components 222

8.3 Typical Static Switch One Line 223

8.3.1 Normal Operation 223

8.3.2 Bypass Operation 224

8.3.3 STS and STS/transformer Configurations 225

8.4 STS Technology and Application 225

8.4.1 General Parameters 225

8.4.2 STS Location and Type 226

8.4.3 Advantages and Disadvantages of the Primary and Secondary STS/Transformer Systems 226

8.4.4 Monitoring, Data Logging, and Data Management 227

8.4.5 Downstream Device Monitoring 227

8.4.6 STS Remote Communication 228

8.4.7 Security 228

8.4.8 Human Engineering and Eliminating Human Errors 229

8.4.9 Reliability and Availability 230

8.4.10 Repairability and Maintainability 231

8.4.11 Fault Tolerance and Abnormal Operation 232

8.5 Testing 232

8.6 Conclusion 233

9 The Fundamentals of Power Quality 235

9.1 Introduction 235

9.2 Electricity

Basics 237

9.2.1 Basic Circuit 238

9.2.2 Power Factor 238

9.3 Transmission of Power 241

9.3.1 Life Cycle of Electricity 241

9.3.2 Single-Phase and Three-Phase Power Basics 243

9.3.3 Unreliable Power versus Reliable Power 245

9.4 Understanding Power Problems 245

9.4.1 Power Quality Standards 246

9.4.2 Power Quality Transients 249

9.4.3 RMS Variations 250

9.4.4 Causes of Power Line Disturbances 255

9.4.5 Power Line Disturbance Levels 261

9.5 Tolerances of Critical Loads 261

9.5.1 CBEMA Curve 263

9.5.2 ITIC Curve 263

9.5.3 Purpose of Curves 265

9.6 Power Monitoring 265

9.7 The Impact of Alternative Energy Generation 268

9.8 Conclusion 269

10 UPS Systems: Applications and Maintenance with an Overview of Green Technologies 273

10.1 Introduction 273

10.1.1 Green and Reliability Overview 273

10.2 Purpose of UPS Systems 275

10.3 General Description of UPS Systems 279

10.3.1 What is a UPS system? 279

10.3.2 How does a UPS system work? 279

10.3.3 Static UPS Systems 280

10.3.4 Online 281

10.3.5 Double Conversion 282

10.3.6 Double Conversion UPS Power Path 282

10.4 Components of a Static UPS System 284

10.4.1 Power Control Devices 284

10.5 Online - Line Interactive UPS Systems 291

10.6 Offline (Standby) 292

10.7 The Evolution of Static UPS Technology 293

10.7.1 Emergence of the IGBT 293

10.7.2 Two and Three-Level Rectifier/Inverter Topology 294

10.7.3 Silicon Carbide Replaces Silicon as UPS Semiconductor of Electricity 295

10.8 Rotary UPS Systems 299

10.8.1 UPSs Using Diesel 300

10.8.2 Hybrid UPS Systems 301

10.9 Redundancy, Configurations, and Topology 301

10.9.1 N 302

10.9.2 N+1 302

10.9.3 Isolated Redundant 303

10.9.4 N+2 303

10.9.5 2N 304

10.9.6 2(N+1) 305

10.9.7 Distributed Redundant / Catcher UPS 305

10.9.8 “Eco-Mode” for Static UPS 306

10.9.9 Availability Calculations 307

10.10 Energy Storage Devices 308

10.10.1 Battery 308

10.10.2 Flywheel Energy 314

10.11 UPS Maintenance & Testing 316

10.11.1 Physical Preventive Maintenance (PM) 318

10.11.2 Protection Settings, Calibration, and Guidelines 318

10.11.3 Functional Load Testing 319

10.11.4 Steady State Load Test 319

10.11.5 Steady State Load Test at 0%, 50% and 100% load: 320

10.11.6 Harmonic Analysis and Testing 320

10.11.7 Filter Integrity and Testing 321

10.11.8 Transient Response Load Test 322

10.11.9 Module Fault Test 322

10.11.10 Battery Run Down Test 322

10.12 Static UPS and Maintenance 323

10.12.1 Examples of Semi-Annual Checks and Services for UPS Systems 324

10.13 UPS Management 324

10.14 Conclusion 325

11 Data Center Cooling Systems 327

11.1 Introduction 327

11.2 Background Information 330

11.3 Cooling within Datacom Rooms 331

11.4 Cooling Process 332

11.4.1 Cooling Process in Datacom Space 332

11.4.2 Direct Expansion (DX) Systems 333

11.4.3 Chilled Water Systems 334

11.5 Cooling Final Dissipation 334

11.5.1 Air Cooled System 335

11.5.2 Water Side 335

11.6 The Refrigeration Process 337

11.6.1 Refrigeration Equipment – Compressors 337

11.6.2 Refrigeration Equipment – Chillers 338

11.6.3 Heat Rejection Equipment 342

11.6.4 Energy Recovery Equipment 353

11.6.5 Heat Exchangers 360

11.7 Components Inside Datacom Room 363

11.7.1 Computer Room Cooling Units 363

11.8 Conclusion 373

12 Data Center Cooling Efficiency, Concepts, & Technologies 375

12.1 Introduction 375

12.2 Heat Transfer Inside Data Centers 379

12.2.1 Heat Generation 379

12.2.2 Heat Return 380

12.2.3 Cooling Air 380

12.3 Cooling and Other Airflow Topics 381

12.3.1 Leakage 381

12.3.2 Mixing and its Relationship to Efficiency 382

12.3.3 Re-circulation 382

12.3.4 Venturi Effect 382

12.3.5 Vortex Effect 383

12.3.6 CRAC/CRAH Types 383

12.3.7 Potential CRAC Operation Issues 383

12.3.8 Sensible vs. Latent Cooling 384

12.3.9 Humidity Control 386

12.3.10 CRAC Fighting / Too Many CRACs 387

12.4 Design Approaches for Data Center Cooling 388

12.4.1 Hot Aisle/Cold Aisle 388

12.4.2 Cold Aisle Containment 388

12.4.3 In-Row Cooling with Hot Aisle Containment 388

12.4.4 Overhead Supplemental Cooling 389

12.4.5 Chimney or Ducted Returns 389

12.4.6 Advanced Active Airflow Management for Server Cabinets 390

12.5 Additional Considerations 390

12.5.1 Active Air Movement 390

12.5.2 Adaptive Capacity 390

12.5.3 Liquid Cooling 391

12.5.4 Cold Storage 392

12.6 Hardware & Associated Efficiencies 392

12.6.1 Server Efficiency 392

12.6.2 Server Virtualization 392

12.6.3 Multi-Core Processors 393

12.6.4 Blade Servers 393

12.6.5 Energy Efficient Servers 393

12.6.6 Power Managed Servers 393

12.6.7 Effect of Dynamic Server Loads on Cooling 393

12.7 Best Practices 394

12.8 Efficiency Problem Solving 394

12.9 Conclusion 396

12.10 Conversions, Formulas, Guidelines 396

13 Raised Access Floors 397

13.1 Introduction 397

13.1.1 What is an Access Floor? 397

13.1.2 What are the Typical Applications for Access Floors? 399

13.1.3 Why use an Access Floor? 399

13.2 Design Considerations 400

13.2.1 Determine the Structural Performance Required 400

13.2.2 Determine the Required Finished Floor Height 403

13.2.3 Determine the Understructure Support Design Type Required 404

13.2.4 Determine the Appropriate Floor Finish 405

13.2.5 Air Flow Requirements 406

13.3 Safety Concerns 409

13.3.1 Removal & Reinstallation of Panels 409

13.3.2 Removing Panels 409

13.3.3 Stringer Systems 411

13.3.4 Protection of the Floor from Heavy Loads 412

13.3.5 Grounding the Access Floor 417

13.3.6 Fire Protection 418

13.3.7 Zinc Whiskers 419

13.4 Panel Cutting (For all Steel Panels or Cement Filled Panels that do not Contain an Aggregate) 419

13.4.1 Safety Requirements for Cutting Panels 419

13.4.2 Guidelines for Cutting Panels 420

13.4.3 Cutout Locations in Panels; Supplemental Support for Cut Panels 420

13.4.4 Saws and Blades for Panel Cutting 420

13.4.5 Interior Cutout Procedure: 421

13.4.6 Round Cutout Procedure 421

13.4.7 Installing Protective Trim Around Cut Edges 421

13.4.8 Cutting and Installing the Trim 422

13.5 Access Floor Maintenance 423

13.5.1 Best Practices for Standard High Pressure Laminate Floor Tile (HPL) and for Vinyl Conductive & Static Dissipative Tile 423

13.5.2 Damp Mopping Procedure for HPL and Conductive & Static Dissipative Vinyl Tile 423

13.5.3 Cleaning the Floor Cavity 424

13.6 Troubleshooting 424

13.6.1 Making Pedestal Height Adjustments 425

13.6.2 Rocking Panel Condition 425

13.6.3 Panel Lipping Condition (Panel Sitting High) 425

13.6.4 Out-of-Square Stringer Grid (Twisted Grid) 426

13.6.5 Tipping at Perimeter Panels 427

13.6.6 Tight Floor or Loose Floor: Floor Systems Laminated with HPL Tile 427

13.7 Additional Design Considerations 428

13.7.1 LEED Certification 428

13.7.2 Energy Efficiency - Hot and Cold Air Containment 428

13.7.3 Airflow Distribution and CFD Analysis 429

13.8 Conclusion 437

14 Fire Protection in Mission Critical Infrastructures 439

14.1 Introduction 439

14.2 Hazard Analysis 441

14.3 Alarm and Notification 441

14.4 Early Warning Detection 444

14.4.1 Wireless Detection 445

14.5 Fire Suppression 445

14.5.1 Hybrid Fire Suppression Systems 448

14.5.2 Protecting Lithium Ion Batteries 449

14.6 Systems Design 450

14.6.1 Stages of a Fire 450

14.6.2 Fire and Building Codes 451

14.7 Fire Detection 452

14.8 Fire Suppression Systems 461

14.8.1 Water Mist Systems 467

14.8.2 Carbon Dioxide Systems 470

14.8.3 Clean Agent Systems 472

14.8.4 Inert Gas Agents 472

14.8.5 IG-541 473

14.8.6 IG-55 474

14.8.7 Chemical Clean Agents 474

14.8.8 Portable Fire Extinguishers 479

14.8.9 Clean Agents and the Environment 479

14.9 Conclusion 480

15 Managing Through Pandemics 481

15.1 Executive Summary: COVID-19’s Impact on Critical Infrastructure Globally 481

15.2 Architectural Solutions and Air Purification Systems 482

15.2.1 HVAC Systems 482

15.2.2 UV Technology 482

15.2.3 Bipolar Ionization 485

15.2.4 Copper Doorknobs 485

15.2.5 Architectural Improvements to be Considered 486

15.3 Building Equipment Solutions and Technology 487

15.3.1 Cleaning vs. Disinfecting vs. Sanitizing 487

15.3.2 Intensify Cleaning Frequency and Measures 487

15.3.3 IR Scans 488

15.3.4 Rethinking the flush, the sink, and the hand dryer 488

15.3.5 Technology 489

15.4 Operations, Maintenance and Training 491

15.4.1 Personal Protection 491

15.4.2 Change in Operation 491

15.4.3 Data Center Betterment Opportunities 492

15.5 Site Protection: Safeguarding the Staff and Visitors 493

15.6 The Workforce of Tomorrow 494

15.7 Assessment Tasks - HVAC and Air Handling Units Filter Upgrades 495

15.8 Managing Through Pandemics -Questions to Consider 496

15.9 Conclusion 497

Appendix A Policies and Regulations 499

A.1 Introduction 499

A.2 Industry Policies & Regulations 501

A.2.1 USA PATRIOT Act 503

A.2.2 Sarbanes-Oxley Act (SOX) 505

A.2.3 Comprehensive Environmental Response, Compensation, and Liability Act of 1980

(also known as the Superfund Act) 506

A.2.4 Executive Order 13423: Strengthening Federal Environmental, Energy and Transportation Management 507

A.2.5 ISO27000 Information Security Management System (ISMS) 508

A.2.6 The National Strategy for the Physical Protection of Critical Infrastructures and Key Assets 513

A.2.7 2009 National Infrastructure Protection Plan 514

A.2.8 North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Program 514

A.2.9 U.S. Security & Exchange Commission (SEC) 516

A.2.10 Sound Practices to Strengthen the Resilience of the U.S. Financial System 516

A.2.11 C4I Command, Control, Communications, Computers, and Intelligence 517

A.2.12 Basel II Accord 519

A.2.13 National Institute of Standards and Technology (NIST) 519

A.2.14 Business Continuity Management Agencies and Regulating Organizations 521

A.2.15 FFIEC - Federal Financial Institutions Examination Council 523

A.2.16 National Fire Prevention Association 1600 – Standards on Disaster/Emergency Management and Business Continuity Programs 524

A.2.17 Private Sector Preparedness Act 525

A.3 Data Protection 526

A.4 Encryption 528

A.4.1 Protecting Critical Data through Security and Vaulting 529

A.5 Business Continuity Plan (BCP) 529

A.6 Conclusion 531

Appendix B Consolidated List of Key Questions 535

Appendix C Airflow Management (A System Approach) 553

C.1 Introduction 553

C.2 Control is the Key 555

C.3 Obtaining Control 558

C.4 Air Management Technologies 565

C.5 Conclusion 570

Glossary 573

References 595

Index 609