from Hacker News

California power outage triggers chaos in science labs

by McKayDavis on 10/11/19, 12:11 AM with 301 comments

  • by TeMPOraL on 10/11/19, 6:19 AM

    Reminds me of a story a certain physics PhD told me about certain very known university in Poland.

    One day some years ago, the university administration thought, "why are we wasting money keeping power on, running lights and aircon and God knows what, when the buildings are vacant?". They soon turned that thought into action, and when a long holiday break came, they just shut off the power to the buildings as soon as everyone left for the day. About a week later, the PhD I mentioned and his co-workers came back to the lab, saw some droplets on the glass, and almost had a heart attack.

    Turns out, after a week of being without aircon to dry and warm it, the air was just about to cross the dew point threshold, and if it did, it would kill the optics on a million-dollar research laser they recently got installed.

    I hear that since then, they're a little bit less pound-foolish.

  • by whyenot on 10/11/19, 4:48 AM

    My university is not a top tier research university like UC Berkeley by any means, but our co-generation plant is able to power the whole campus in the case of a power cut. Even when the central plant gets knocked out (a squirrel did that last year), individual buildings have backup generators. The building where I work has three of them and all of our -80s are on back up power or could be quickly shifted over. That is several layers of redundancy. We have done a lot of planning for events like this. I'm a little surprised that a more research focused (and less resource starved) institution like Berkeley wasn't more prepared.
  • by twblalock on 10/11/19, 3:25 AM

    I'm glad I live in the city of Santa Clara, where I am served by Silicon Valley Power, one of few exceptions to the PG&E monopoly in northern California.

    Silicon Valley Power serves many homes in addition to large companies including Intel and Nvidia, and they do so with lower prices and better quality of service than PG&E. I will be very unhappy going back to PG&E if I move to another city.

    Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities.

  • by noodlesUK on 10/11/19, 2:41 AM

    PG&E has been doing blackouts and brownouts more than basically any other electric company I’ve ever experienced anywhere in the world since I was a little kid. These blackouts may be bigger, but if you’re an institution the size of a university, you should have significant disaster preparedness infrastructure. I mean US universities run police forces! I’m sure they could find a way of designing redundant power into their systems. They might even save money in the long run, and wouldn’t need to worry about campus hospitals or public safety going down in an outage (I’m sure some of that is already taken care of, but why not go all the way).
  • by quotemstr on 10/11/19, 2:56 AM

    It's fascinating how these PG&E threads are full of recriminations, but not solutions. The power went out for 2.6 million people (on 800k accounts). No natural disaster or unanticipated software bug caused this problem. The situation is purely a human coordination failure. The state is doing nothing except issuing sternly worded statements when it could be changing policy. This odd mood of angry, passive resignation is disturbing.

    This whole situation represents a loss of "social technology". Just as material technology is a bag of techniques for organizing matter into useful configurations, social technology is a bag of techniques for organizing people for a useful purpose. When a society loses a technology, be it material or social, it loses a capability. It becomes unable to do what used to come easily to it. For 120 years, California was able to keep the lights on. Now it can't. How is anyone supposed to not see this situation as a kind of foreboding regression?

    This new power grid unreliability is far from the only example of our struggling to do something that came easily to us 20, 30, or 40 years ago. We've definitely lost something, although it's hard to pin down exactly what.

  • by ChuckMcM on 10/11/19, 5:15 AM

    This is such a screw up. Personally I think this is PG&E trying to negotiate with the State to have it indemnify them against wildfires so that they neither have to pay to maintain their infrastructure, nor pay out large damage claims when that infrastructure burns down a town or two.

    I really wish the PUC wasn't captured, otherwise we might be able to fix this problem.

  • by ethanpil on 10/11/19, 6:17 AM

    California has the 5th highest rate per kWh[1]. (#2 excluding Alaske and Hawaii)

    Just like our sales tax rates, income tax rates, our gas prices (due to taxes), sanitation costs, ad nauseum. We pay excruciatingly high rates for everything, and get mindbogglingly low returns for the money.

    CA gov. and utilities are a textbook case in poor management and government out of control.

    Where does the money go?

    [1] https://www.chooseenergy.com/electricity-rates-by-state/

  • by mikorym on 10/11/19, 7:13 AM

    Not to be insensitive, but for a South African this is a kind of LOL moment. We have had power outages mess with experiments and universities since at least the early 2000's. That of course and the arson that burned down auditoriums.

    But in terms of experiments, if you want your MSc or PhD then you better make sure that you can go a couple of days without power to the grid.

    If you are in a rural area, especially a previously "white" area as the pre-1994 NP government called it, then you need to have your own > 5000 L water tank. The town where I went to primary school had 9 days without water or electricity a year or two ago. Probably half of the town (it's a small town though) never have water between 9PM and the next morning since they simply turn of the pumps (whether there is electricity or not). The government officials also steal water to go sell for $1 per 100 L to areas where there is no running water.

    You can think of it as a kind of osmosis between previously affluent areas and areas that had been mud huts. If you think of it that way it doesn't sound that bad, actually, but I think the net position is not the issue, the real issue to me is the corruption and apathy within the government and the rate of improvement. Africa should be ahead of China in economic growth if you look at the youth percentage and potentially massive workforce.

    If you want to completely redesign an entire country's power grid and make it 100% solar/wind, then South Africa is the perfect candidate. That is if you could forget about politics for a moment.

  • by egdod on 10/11/19, 4:20 AM

    Power outages happen sometimes in the normal course. They are not a new thing, and the fact that this one is planned doesn’t make it worse.

    If you have specimens that you cannot afford to have warming up, it’s inexcusable that you don’t already have a backup power source.

  • by claudeganon on 10/11/19, 3:21 AM

    I wonder at what point California will just nationalize their grid/energy infrastructure. These “markets” always end up as some private monopoly with all the corresponding terrible incentives around upgrades and maintenance. If it was actually controlled by the state, it could be mandated to build out renewable-centered, decentralized systems as well.
  • by atonse on 10/11/19, 2:00 AM

    Maybe these kinds of things will push us more towards decentralized power generation like solar and wind, and bigger lab-wide or campus-wide battery battery backups?
  • by legulere on 10/11/19, 4:54 AM

    Meanwhile in Germany the average duration of blackouts in 2017 was around 20 minutes per customer, in quest where it was pretty high because of storms. The frequency was 0.28 in 2017, which means you can expect every three years an outage.

    Source: https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualita...

  • by cwhittle on 10/11/19, 5:54 PM

    All of these big research universities in CA are on major and active fault lines. My old lab at Cal sat mere hundreds of feet from the Hayward fault that runs through campus. This is a proactive power shutdown when you know it's coming. If they can't an event with advanced warning, what will happen to hundreds of millions of dollars of frozen samples and research instruments when the unexpected earthquake hits any of these universities, not only disrupting power, but also disrupting structures? Incredibly foolish not to have at least reliable power backups and containments irrespective of what PGE is doing.
  • by StanislavPetrov on 10/11/19, 6:17 AM

    We're going to spend far in excess of 700 billion dollars to maintain our global military empire this year alone (not counting legacy costs for veterans and countless other huge, related expenses) while millions of Americans sit in the dark without power because we outsource our control of our power grid and other critical infrastructure to for-profit corporations. Hopefully some of those people sitting in the dark will give that some thought while the lights are out.
  • by secabeen on 10/11/19, 10:20 PM

    This isn't the best sourced article, the sources here are all random people complaining on twitter. It would be much more interesting to hear from an informed spokesperson from UC Berkeley, with actual data on the percentage of STEM research buildings without generators, etc. etc.

    Even then, one of the sources did describe that they have an e-power plug in their lab, for the -80 freezer. That's pretty much how it's supposed to be. If they have additional important fridges, those should also be plugged into e-power.

    I'm sure it's possible there are some labs at Berkeley that may be compromised in backup power, there are a lot of old buildings and infrastructure there. That still doesn't mean that that experience is common, just that the people impacted are complaining loudly. Most research labs may be totally fine, and just waiting for things to come back.

  • by pvaldes on 10/11/19, 2:19 PM

    Curiously, a similar problem happened last week in the Institute of Natural Products and Agrobiology from the Consejo Superior de Investigaciones Científicas in Tenerife.

    Temperature of ultrafrozen samples raised to -75 Celsius and years of hard work and a collection of DNA reference samples for agriculture were about to vanish

  • by dannykwells on 10/11/19, 1:37 PM

    Berkeley, my post doc alma mater, really has fallen on hard times. I get that PGE caused this, but UCB should have reliable back up power for their labs. Full stop. You can see why they're fighting so hard in the CRISPR case. They're desperate to raise dough.
  • by Mathnerd314 on 10/11/19, 7:27 AM

    Here on page 682 (10.3.1D.19) "High-value specimen refrigerators, freezers, etc." are listed as "optional" for emergency power: https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/...

    Clearly it's a big inconvenience but they're not in danger of losing any grants.

  • by fyfy18 on 10/11/19, 5:14 AM

    I think it is interesting to compare PG&E to how the electricity grid is owned and run in the UK. Many people here are saying they should be nationalised, well the UK did the opposite 30 years ago and it seems to be working fine.

    In the UK there is National Grid Electricity Transmission plc, which as the name suggests is responsible for the transmission infrastructure across the country. They are a publicly traded company, and the parent company owns electricity & gas distribution in the UK, and some in the US. There are then smaller companies who are responsible for maintaining more local infrastructure (as I understand it's still owned by National Grid, but they are responsible for fixing and upgrading it).

    As a consumer you can choose to buy electricity from a long list of companies, and easily switch between them. All that happens is a different company collects your meter reading and they send you a bill. A lot of these companies also own generation facilities (e.g. SSE, who are also publically traded) while some are just resellers (e.g. So Energy).

  • by gwbas1c on 10/11/19, 1:59 PM

    This is something I don't understand...

    Portable generators are so inexpensive that I'd think an institution like this could just keep some in a shed or garage, and shlep cords through windows for extended outages. I bought a top-of-the-line tiny Honda Inverter generator so I could keep my wife's breast milk frozen in case of an extended outage, but even something half the cost would be good enough to run a refrigerator.

    But, even then, if you plan for it, backup generators and Solar + Battery are very affordable!

    I recently installed a Powerwall purely for backup. Because I have solar, it's slightly more expensive than a standby generator, but with no maintenance or fuel I suspect it'll be a wash within a few years. (Backup generators need expensive maintenance every year.)

    Even Generac bought a Solar + Battery company because they know they can't compete, and their solution is nicer than my Powerwall! (The Generac solution includes a critical loads panel inside of it, unlike Powerwall where the critical loads panel is outside.)

  • by dmode on 10/11/19, 3:38 AM

    I have no idea why this shutdown was initiated. It had been much warmer and drier earlier in the summer. It is actually quite pleasant and cold these days, with morning dew. And no winds. I have no idea why these conditions warrant power shutoff
  • by lawrenceyan on 10/11/19, 3:31 AM

    I go to UC Berkeley. My midterms were postponed so I feel like even though I should be complaining, I really can't. PG&E basically gave me a week extra to study.
  • by dekhn on 10/11/19, 12:32 PM

    all the SREs on the thread are wondering why scientists don't automatically mail their complex samples to their friends across the country as a backup.

    and the answer is.... individual PIs optimize for the easy case (that power stays on), not the disaster recovery scenario. There is no SRE thinking at the individual PI level.

  • by bavcyc on 10/11/19, 10:07 AM

    Rather than answer several individual posts, I'll be lazy and do a larger post which involves a lot of simplification and hand waving. I apologize in advance for any errors, but this seemed like a good way to handle insomnia instead of my normal lurking. If something is unclear or you think wrong, let me know.

    Power Companies circa 1900 there were lots of power companies, I've seen pictures of urban areas where there were multiple circuits run on poles, how many companies tried to serve a certain area, I do not know. Through lobbying investor owned utilities (IOU), or in most cases a single IOU gained the rights to serve an area exclusive. IOUs concentrated, for the most part, on serving dense load concentrations. As such the US Govt implemented the Rural Electrification Act, so low density areas could be served. The 2 types of electric utilities are public power (municipal, REA, RUS, PPD, UD, etc) and for profit (IOU's typically). In most cases, the IOUs give something to the government in exchange for the IOU providing service exclusively, it might be a tax or it might be free street lights. Municipals tend to subsidize the local government in some way, either through returned dollars or free power (street lights, buildings, traffic lights). IOUs typically have a defined rate of return and are supervised by a governing entity of some sort.

    Vegetation Management The recent SERC compliance meeting had a good presentation on vegetation management from the utility perspective and another on enforcement trends. In my opinion a lot of the issues in the W US is the result of the policy 'no fire is good', the sand pile game I think illustrates the issue where the longer sand keeps from falling results in a larger collapses (see Yellowstone fire). Since the late 1980's the issues with not burning has been known, I had an ecology class where if I recall correctly that was discussed for a couple of days. Tree trimming and clearing out undergrowth is done on a regular basis when the utility has an easement, but especially in urban areas folks tend to plant trees too close to power lines or even worse encroach on the easement with buildings. Most utilities patrol transmission lines at least once a year if not twice or monthly, sometimes this is aerial and other times it is feet on the ground walking the line. As an aside, the NESC governs clearance of electric lines to stuff and how stuff should be built; the RUS has publications on line design if you want to read about it.

    Distribute Generation The electric grid in the US is divided into 3 areas, Eastern Interconnect, Texas and Western Interconnect; they all function essentially the same. If I have a generator connected to the grid, it has to synchronize to the grid before closing the breaker. If it is done correctly then there is very little mechanical stress on the generator, if done incorrectly then there is a large amount of mechanical stress on the generator. One mis-operation I know about involved the A and B phases being swapped during a re-wind, when the generator was closed in at commissioning it had a large bang/clunk and the breaker opened immediately. The generator then had to be examined, i.e. taken back apart, to figure out what went wrong and if it could be put back into service.

    If I have a generator partially supplying a facility (this can save a lot of money for an entity) and a fault happens on the grid then my goal is to protect the generator, so the generator will either shut off or island the facility while shedding load above the generator's capacity. This happens very quickly. One instance I know of, the urban area was supplied by transmission (aka remote generation), a 30 MW generator was the closest source, the utility had a fault because equipment misoperated and the generator was suddenly trying to supply all the power to that fault such that the generator protection operated and islanded the facility. It was no issue to close the grid interconnect back in (once it was ensured it was safe to do so) but the facility had to shed load to keep the generator running without causing electrical issues to load and damaging the generator.

    Once a facility is islanded and running on its own generation the phase angle is a don't care until it is time to synchronize back to the grid. As long as the facility can shed load to maintain frequency (there is a NERC standard on Under Frequency Load Shed if you want to read about it) and not ruin equipment by having a power quality issue. During dynamic studies for system stability, it can be observed that a generator will diverge from the system frequency phase angle but not trip off because it is isolated from the grid which requires verification that isolation is happening and the protection scheme will indeed work that way on the actual system.

    My observation is that most facilities, data centers and other processing facilities (refineries) tend to be the exception, concentrate on first costs when designing their electric infrastructure. It is possible to design a resilient system but it has a cost and it will not be utilized 100% until something goes wrong. And if you are doing research then that can be an issue as you may lose a large amount of data due to the power going out or possibly being sensitive to transients on the system, e.g. a switching operation on the transmission system affects the end user equipment. Even if you have redundant systems (and/or power supplies) it is possible to have single point of failures on your system. As well if you have enough local generation to supply your load, it may be more economical to not run 100% of your generation as the market price for electricity is cheaper than your cost of production (and there are folks who don't like idle assets, not realizing the greater benefit is not using it or only having to use it infrequently).

    One other aspect of distributed generation is the automatic separation of the DG when loss of voltage is detected on the grid side. Utilities do not want voltage on their system if they have an outage due to worker safety (and other reasons). Utility crews in hurricane areas will typically investigate if they hear a generator running when the power is out to an area to ensure it is not back feeding the distribution line. As a reminder keep your feet together if you are near a downed power line and hop away, or even better don't go near downed power lines.

  • by mensetmanusman on 10/11/19, 2:19 AM

    People will now experience rolling black outs in California just like in South Africa? Is this just because of the lawsuit?
  • by proee on 10/11/19, 2:56 AM

    This event will increase the sales of Tesla Powerwalls.
  • by driverdan on 10/11/19, 2:00 AM

    If you have work that can be destroyed by a power outage you should plan accordingly. Power outages are a normal thing and can happen at any time, especially in areas with earthquake and fire risks. Not preparing is foolish.
  • by TomMckenny on 10/11/19, 3:02 AM

    Note on the blackout map, the county in the dead center has continuous power and it has a municipal source.

    Likewise the 2000 crisis where LA's DWP customers had power when PG&E did not.

    Although I'm sure it makes me a "socialist" for pointing these things out, it almost seems that having private monopolies in utilities is a bad idea

  • by exabrial on 10/11/19, 2:24 AM

    I mean, don't live in a desert? Seems like an obvious conclusion
  • by unethical_ban on 10/11/19, 4:11 AM

    Surely, PG&E will learn their lesson during this outage and send crews across the state to trim brush along the transmission lines? Strengthen the towers?

    Uncharacteristic edit: Why the downvotes? It must be because you took the comment seriously and think the sentiment grossly naive, or heard the skepticism and thought it unwarranted?