Hacker News new | past | comments | ask | show | jobs | submit login
System separation in the Continental Europe Synchronous Area on 8 January 2021 (entsoe.eu)
269 points by andreasley on Jan 26, 2021 | hide | past | favorite | 131 comments



You know what I'm missing? The almost realtime map from ucte.org which updated every 3 to 5 seconds on my Pentium3 933 with only 512MB Ram in the browser.

I think it needed some plugin, either Java or Flash, but you could zoom in or out, either to the whole of Europe, or down to the substations in the cities and towns.

It was like google maps or similar. AND it had all the transmission lines, power plants, and substations.

And color coded lines, with KW/MW/GW and arrows indicating the direction of energy flow on them.

Anybody could view it without having an account there. At least I did, from time to time.

This way of viewing it in realtime is now gone, or at least not accessible to the general public anymore.

Does anybody know some equivalent?

Sites with bargraphs and charts need not apply.


ENTSO-E is the successor organisation to UCTE [1], with its functions transferred to ENTSO-E in 2009. So if anyone has the data, it would be them. Interesting that the ucte.org website still seems to be online and frozen in time!

[1] https://www.entsoe.eu/news-events/former-associations/#union...


While ENTSO-E does have a map [1], it's not as nice as the one described above.

[1] https://www.entsoe.eu/data/map/


Yes, does not compare, unfortunately.


I hate to commit red-herring but this is another example of how computing is worse today in some ways.


Yeah, I noticed that some years ago. They don't have that offer anymore. Interestingly, looking at the article, the cutouts are in the exact style of that former realtime map.

But static. Not living. That was once there, and available to the general public. Personally, I view that as artificial scarcity of information, regarding energy politics. 'Smokescreening' so to speak.


Is it possible it's no longer available for security reasons?


I don't know. I've thought about that. OTOH if some terrorist would like to know the choke points of the grid, he could deduce that from openstreetmaps also, where the transmission lines and substations are available for viewing.


Yes, and honestly someone determined to do damage like that would be the sort to be patient and methodical enough to find targets with or without the map. But I do understand the instinct to want to at least increase the barrier to destruction, the problem is that it ALSO increases the barrier to understanding (and so protection and enhancement). So, an understandable but bad trade off, IMHO.


I didn't see it from that point of view. It was just interesting to watch for me, out of the corner of my eyes, while doing other stuff on other systems and screens. I did that somewhere between 2005 and 2008, whenever I felt like it :-) With a focus on Germany in general, and my town especially. So I had it mostly zoomed just so that Germany was in view, with small parts of neighbouring countries visible. We didn't have that much solar and wind at the times. But sometimes one could see the direction of flow switch. From/to France for instance, or Denmark. And I think I've seen that to/from Poland and Czech also.

Anyway, you could get a feeling for how things were depending on time of year/day, weather, and so on. Intuitevely, because presented spatially and realtime.

And you don't get that spatial intuitiveness from the substitutes with (often delayed) charts & graphs.

Which makes it impossible to counter bullshit by politicians or other parties with vested interests when they are saying this and that, and you could have said: Ahem, that is not entirely correct, because there and then it was like so, and not what you say!

And the loss of that ability is making me angry!

edit: Regarding the 'intuitiveness' of spatial vs. bars & charts. Even when the possibility to filter for im/export between countries exists, it does not show over which lines it went. Which could be useful in discussions about extending/upgrading lines, or building new ones. It's just not there.


The physical layout of the transmission system is basically impossible to obscure, but does not necessarily give the attacker insight into power flows. Having access to real time power flow data would enable an attacker to design a cascading failure scenario, making an attack MUCH more effective. The last component would be knowing operational response procedures. Which seem to be fairly transparent in Europe? If so I'm surprised more care isn't taken to secure them.


It is probably right, but I have to say that I hate that kind of security through obscurity


Especially when there is this https://de.wikipedia.org/wiki/Stromausfall_in_Europa_im_Nove... which was caused by switching this off, https://de.wikipedia.org/wiki/380-kV-Ems-Freileitungskreuzun... which is rather similar to https://de.wikipedia.org/wiki/Elbekreuzung_2 :-)

Even if there are surveillance cameras on those masts, there aren't on the masts leading to it. So you could ride your bicycle there, have some fun with thermite, and finally enjoy the stars again!1!!

/me giggles (j/k)


Did it look like this?

https://www.researchgate.net/figure/Energy-trade-change-in-t...

Seems like they're referring to it but also the comments suggest the data was from simulation of optimal market exchange, not actual electrical flows


Nope. That is abstract. It looked exactly like the 2 blurred cutouts from the Entsoe article, where they put in the probable causes textmarker style. Interestingly it is hard to find screenshots of good quality or at all. Seems to have been a very special interest of mine?


I used to work on fine-grained internet mapping like on opte.org, so I understand the bug :)

If you ever do find something similar I'd love to know. I have a friend at Kevala, doing this privately in the US. But otherwise, yeah, seems to be a dark art.


Is it possible that not making that level of data available is purposeful from a security point of view?

Western European countries (among others) worry about threats to their "Critical National Infrastructure", whether in person (terrorist with a bomb) or as a cyber attack. Providing a big map with all transmission lines, power plants and substations seems like it would be a big help to an unsophisticated terrorist, showing them what to blow up.


I don't think so, because you can see that on openstreetmap. I mean the transmission lines and substations.


That's a lost art nowadays, doing all that on a Pentium 3. Today, you'd need about 512 GB of RAM and a supercomputer to do the same.

\s


Hrrm. I did that under some Linux and KDE 3.x, maybe even with the KDE Browser called Konqeror. That was usable at the times :-) 2005 to 2008 that must have been.



While that is looking nice, (and I don't know how 'realtime' it is) it lacks the structure of the transmission nets, and the zoomability down to substation level in the towns/cities. I can't remember exactly anymore, it didn't show every transformer in town, but the places where the high voltage overland transmission lines entered into town, and were transformed down to more managable levels were there. I think down to 10.000/50.000 Volts. That was different regionally. For instance Portugal vs. Germany.


Someday someone will reimplement it in React, and you will only need an i9-9900k with 64Gb to run it somewhat smoothly.


Where would someone get the realtime data? Technically it has to exist, otherwise frequency regulation wouldn't work. The fact that they don't show it in the way I described, shows that they don't want to anymore, because they don't need to because of different organizational structure, or the structure has been made that way for ideologicel reasons by politics, i.e. 'smokescreening'.


Direct frequency regulation (primary reserve or FCR) acts directly on the measured frequency at the power plant. But yes, for secondary reserve (aFRR), every grid has to know all its flows, but I don't know of any European grid operator that was transparent about this information. You can deduct some of the cross-border flows from observing the energy markets.


How does this affect timekeeping? Normally, if there is a drop in grid frequency, a small increase will be planned later to maintain a long term average frequency of 50Hz so that clocks that keep time by grid frequency stay accurate [0].

The network split seems to have made this impossible, during the split, the cycle counts for the two regions diverged, and the split ended before this was reconciled. Will people in one of the regions have to adjust their clocks?

[0]: https://en.wikipedia.org/wiki/Wide_area_synchronous_grid#Tim...


My suspicion is that NTP (and GPS) have generally superseded the synchronous grid for timekeeping. It has for every application that I know of.

Simultaneously, I suspect some systems still do rely on grid time.


Computers (or anything containing a proper computer) don't care about grid frequency for time keeping.

On the consumer side, your microwave, coffee machine and even bedside clock will.

Not sure on the industrial side.


> On the consumer side, your microwave, coffee machine and even bedside clock will

I'd be very surprised. It's cheaper to build in a 32.768kHz crystal for timekeeping than try to access the grid frequency from the isolated low-voltage circuit.


It's cheaper to build in a 32.768kHz crystal for timekeeping than try to access the grid frequency from the isolated low-voltage circuit.

Those products don't use isolated power supplies --- normally they use a capacitive dropper. A high-value resistor to an IC pin is sufficient to drive the clock counter, and a resistor is definitely cheaper than a crystal.


It is very true.

A similar event happened in Europe a year ago (also due to problems in former Yugoslavia) and all electric clocks in my house (microwave, oven, alarm clocks) went out of sync temporarily. I was just as surprised as you.

Basically, it's exactly the devices that lose track of time in case of a power outage, and that you need to manually adjust for daylight savings time, that are synchronized to the grid. Devices that use a battery (such as laptops, mobile phones, and CMOS clocks in PCs) must necessarily use some other means.

It doesn't seem to be much correlated with price either: my alarm clock cost over $100 and it was still affected. Surely at that price point they could have afforded a crystal oscillator in the design, if they wanted to, but it seems this just isn't typically done.


At a $100 price point, I'd expected a DCF77 receiver tbh, not a crystal oscillator.


As an industry, it requires a special customs declaration + fire safety measures on boats and planes when you transport an oven with a battery inside. That may be the reason for not using battery-backed ovens, and relying on the power supply.


Just curious what kind of alarm clock costs $100


One that plays soothing sounds, like dogs barking or seagulls fighting over trash, probably.

It might even project the moon phase on the ceiling.


Something like this: https://www.amazon.co.uk/dp/B008LR3KD8/

The idea is that it wakes you up gently by flooding the room with light resembling a natural sunrise. One reason that it might be expensive is that it needs a lot of high luminance hue shifting LEDs, and possibly I overpaid a bit (there do seem to be much cheaper competitors) but it was still a good investment because it does make my mornings a lot more pleasant.


I have a microwave and oven from the same brand, and their digital clocks diverge about 2minutes in 24h. I just don't understand how it's possible to build a device that poorly in 2020 (they are new) and I also wonder which tech they use for timekeeping. Is it possible that one of them uses a crystal and the other uses the 50Hz? That would explain a small difference.


It's easy enough to mistune a 32768Hz clock crystal by shitty electronics engineering, unfortunately :/

50Hz might have temporary variations, but it's controlled so that over a longer period of time, you always get the correct number of cycles, e.g. 180000 in an hour.

This also means that going off the 50Hz power grid theoretically has better (or even perfect) long-term accuracy. Also means that if you adjusted your clock for this separation event, you'll have to adjust it back :D


> I have a microwave and oven from the same brand, and their digital clocks diverge about 2minutes in 24h.

That's much, much worse than a cheap digital watch from 20 years ago.


I know. It's worse than the pendulum clock I have to crank at my grandma's once a week! Progress!


Much worse than a digital watch from the 80s. (Maybe 70s?)


It's either a poorly made bottom of the barrel crystal or they cheaped out even further and used a ceramic resonator. That is the price of race to the bottom globalization. Measuring line frequency is a no go because the safety compliance inceases costs.


US current is 60Hz. EU current is 50Hz. Maybe your over maker did some rounding when adjusting by 5/6 and here you go? Btw, does your oven have a 50-60Hz switch?


>> On the consumer side, your microwave, coffee machine and even bedside clock will

> I'd be very surprised. It's cheaper to build in a 32.768kHz crystal for timekeeping than try to access the grid frequency from the isolated low-voltage circuit.

IIRC, the grid frequency is usually more accurate. Probably because it's carefully monitored and managed by the power authorities like the OP describes. If you build a clock with its own frequency reference, than any error will accumulate and have to be monitored and managed by the user.


Maybe, but I bet those designs are very traditional and date from the time where it was cheaper to use grid frequency.

If not, why don't coffee machines that can start on a timer have a backup battery for their clock? That might be cheap enough too. And vital for those moments when there was a 10 second brownout which leads to your coffee not being ready in the morning...


> why don't coffee machines that can start on a timer have a backup battery for their clock?

As a European the thought wouldn't have come to me. Brownouts just don't happen (see this 0.5% frequency dip making headlines), and a blackout is a once-a-decade event for any given house (probably even less frequent than that).

So your answer is probably that demand for that feature is far from universal, and a coffee machine is much lower stakes than for example an alarm clock.


> Depends. I'm in a rural area, and I get 5-second power cuts every couple of months. It's the main reason I have a UPS under my computer desk.

This. Can't put the coffee maker on an UPS though. Or not on a cost effective UPS.

> So your answer is probably that demand for that feature is far from universal, and a coffee machine is much lower stakes than for example an alarm clock.

Actually for the alarm clock i'm just going to be late. Not having coffee hurts much more!


Depends. I'm in a rural area, and I get 5-second power cuts every couple of months. It's the main reason I have a UPS under my computer desk.


I don't think price is significant factor. It just needs optocoupler, diode and few passives.


You don’t need an optocoupler to measure grid frequency. You can just use a resistor. Dirt cheap, cheaper than a crystal.


Yes, but OP talked about isolated circuits. It's likely that most appliances don't need to be isolated though.


OP was talking about consumer circuits, I think they were mistaken about how likely they are to be isolated.


> your microwave, coffee machine and even bedside clock will.

I guess a ~0.5% deviation for a couple minutes is going to be tolerable here.


The grid is never exactly 50Hz, it varies quite significantly. On the long run, the grid operators try to average to 50Hz as close as possible, just so that clocks continue to operate somewhat accurate.

In fact, it is even in the FAQ of the article:

"The transmission grids of the countries of Continental Europe are electrically tied together to operate synchronously at the frequency of approximately 50 Hz"


Based on the article the control limits are 49.8 to 50.2 Hz (+-0.2hz). That's less than 1%, not very significant.

One of the places I lived in Austin, TX had a range with a clock that used grid frequencies (60hz in the US). It was dead accurate. I only changed it for daylight savings, it never need adjustment.


The US used to guarantee 5,184,000 cycles per day. That’s not, like, 1% or anything comparable, that’s zero (with temporary deviations). Any clock which referenced line frequency would be dead accurate over the long-term. There’s a system called Time Error Correction (TEC) which would do this (refer to WEQ-006 and the like). Obviously this error is measured relative to some reference clock.

I’m not sure what the current status is, but my understanding is that there are efforts to retire this system and allow the speed to drift more.


Oh found it. Looks like this went on for more than a couple minutes:

https://www.theguardian.com/world/2018/mar/08/european-clock...


I was under the impression semaphores use grid power as a clock source. And the case of missing payer in southern Balkan got the timing shifted by a few minutes.


Clocks on our ovens and some microwaves got out of sync during that period. I haven't seen any other device rely on the network frequency for timekeeping - everything time sensitive is already on NTP and GPS time.


OT: Several commenters have mentioned both their oven clocks and microwave clocks getting out of sync. I'm curious as to why people set both of these clocks in the first place.

The microwaves and ovens I've seen do not actually require you to set the clock. None of their cooking features and functionality depends on it. It's just a convenience feature. I doubt that most people actually need two clocks in the kitchen, so why not just set one of them (or neither if you don't need a kitchen clock at all)?


Lots of ovens and microwaves don't allow you to disable the clock, so the alternative to setting them is to have them run out-of-sync, which is much more irritating than having to set them.


Ah...I'd forgotten about the ones that always show the time regardless of whether or not it has been set. I can see how that would get annoying.

I guess I've been lucky in that all the microwaves I've had over the last couple of decades, and the ones we had at work, do not show the time unless you've set it. My current one, which has 7 segment LEDs for the time digits, blanks all the digits, leaving just the colons. My prior one showed "--:--" if time was not set. The one before that had a graphical display and just showed blank where the time would normally go.


Mine has two alternatives: Clock or blinking red light. I choose the clock...


I can give you a personal counter-example: my oven requires the time to be set to run - without it, neither the fan nor any of the two heating elements turns on.

It's an old Bosch HBN202S


I have in my kitchen + living-room one clock based on NTP on a dashboard, one on a WiFi radio and one on the oven.

Depending on where I am I see one of them so all must be synchronized to the millisecond so that children are not late to school (school is nearby, which exponentially raises the risk of them being late because it is just "4 minutes" away).

This is one of the scientific uses of such multiple clocks.


There was this great design for a microwave that a UX designer proposed. One button for +30 seconds, no clock, and a door handle. That's it. Everything else is superfluous.


On most microwaves I encountered one can just press the start button and it will do just that: Start with a 30s countdown and pressing it again will add 30s.


As a person who uses a microwave a lot this sounds terrible, I could live with 30s resolution, but not having adjustable power level would not be acceptable. Cooking anything on 100% power is just a recipe for uneven heating.


Based on the other commenter maybe two buttons: start/+30 seconds and power level in 25% increments.


I just got rid of a GE microwave which required me to set both the time and date before heating food after a power outage. I never did figure out what the date was for since it was never displayed and the clock did not automatically adjust for daylight savings time.

On top of that it had a dedicated button for toggling between AM and PM when setting the clock, which served no purpose whatsoever after that point. I wish I knew what the designer was thinking when they came up with that.


Literally because it annoys me more having it not set properly


In Europe, many countries have radio signals that many clocks use to sync their time that is based on some atomic clocks. For example, in Germany these are called Funkuhr and the signal comes from an atomic clock in Braunschweig if I'm not mistaken.


We have this in North America as well. The station is called WWVB.

https://www.nist.gov/time-distribution/radio-station-wwvb


Infosec slant: such investigations are made way easier by precise timestamping (everything happens in a few milliseconds), but the source of truth (GPS or other GNSS) is usually pretty easily spoofable if you only intend to move the time a few milliseconds. Galileo has a project addressing this (https://ec.europa.eu/transparency/regexpert/?do=groupDetail....) but AFAIK it's not in the signal yet. And once it is, it will take years for the devices doing the satellite time to PTP to be replaced/updated.


Surely this is only true to a minimal extent in world where you can already buy timeservers that support multiple independent time sources out of the box (multiple satellite Galileo,GPS etc. + terrestrial radio e.g. DCF77).

IIRC if you're dead serious and money is no object, all the major national time labs also allow you to run a private leased line to their facility for direct checks.


As of now all GNSS constellation are somewhat spoofable, especially if you only want to drift a clock a few milliseconds late... And a lot of timeservers are still GPS only so the attack is pretty easy to perform.

The private line idea is maybe doable (but which protocol? NTP doesn't offer the needed precision), but as all substations would need one, my guess is that eventually money would be an object...


> (but which protocol? NTP doesn't offer the needed precision)

You might enjoy reading the documentation for NPLTime [1] which uses PTP over telecom fibre networks.

Of course, their target market is banks with record keeping requirements so it probably costs a fortune.

[1] https://www.npl.co.uk/npltime


AFAIK the continental synchronous area frequency is long-term-synchronized to atomic time (i.e. the cumulative grid time must stay within X seconds of the reference clock).

Looking at the (partial) graph suggests that the south-east part ran faster for the entire duration of the disconnection.

Does this mean grid-synced clocks in the south-east part are now permanently ahead of the atomic reference, or are they planning a mitigation (which I assume would have to mean disconnecting again and running the SE part slower for a bit)?


> Does this mean grid-synced clocks in the south-east part are now permanently ahead of the atomic reference, or are they planning a mitigation (which I assume would have to mean disconnecting again and running the SE part slower for a bit)?

That's the standard procedure. Maintaining an exact frequency can be hard, but counting cycles is easy. So the frequency will slew to slowly compensate back.

Edit: Here's a Tom Scott video on a UK–Continental divergence that happened a while back: https://www.youtube.com/watch?v=bij-JjzCa7o


UK is connected through DC lines, so frequency is independent. In this case they would have to separate the grid again to be able to correct this.


What puzzles me is why everyone talks about clocks..!

Clock synchronisation is a nice side effect, but what amazes me is how it is possible to keep this thing swinging somewhat synchronously year after year, and how huge chunks of it doesn't crash and burn when events like this happens.

(I have worked as a consultant for a power supplier, I was more puzzled afterwards : )


> what amazes me is how it is possible to keep this thing swinging somewhat synchronously year after year

There's nothing to do to keep them in sync: every single[1] alternator connected to the electric grid is rotating at the same speed, which is driven by the frequency of the electricity. This is a physical effect.

The big challenge isn't to keep things in sync, it's to keep the supply equals to the demand. (when we don't, the frequency goes up or down depending on which is higher than the other)

[1]: in thermal stations only, wind turbines are not directly connected to the grid because their rotation speed depends on the wind…


> There's nothing to do to keep them in sync: every single[1] alternator connected to the electric grid is rotating at the same speed, which is driven by the frequency of the electricity. This is a physical effect.

You either know a lot less than me or you know more and understand it better. I honestly can't tell : )

It feeks like my brain collapses just from starting to think about thousands of kilometers of grid, thousands of power stations, millions of consumers etc. (Not that the amount of them matters, just the supply or demand they deliver/drive.)

In the interest of learning, here is my model that I map the things I do understand to:

The way I (as a hardware/software engineer who started in electronics and had a brief course on power electrics) reason about it is torque, as if they are kind of sharing a common shaft and applying torque to it while consumers are braking it.

This simplified mental model would explain - kind of - how it can stay in sync, only in reality we are talking not a linear axle or shaft but this continent-wide grid where gigawatts of supply and demand can occur within minutes or even less and the phase differs by quite a lot over the span of the continent. (In my mental model this is the axle twisting.)

Also at 50Hz the wavelength of light in vacuum should be around 6000km if I typed correctly and DDG understood correctly. A rule of thumb we learned (in high frequency electronics back in electronics engineering) was that once you cross a tenth of that the normal rules doesn't quite apply. If this can be applied to power grids (an I think it can) it becomes even more complex I guess. (The simplification we can apply in "small" circuits is that we can pretend evey point in the circuit is at the same point of the phase at the same time.)

Keep in mind everyone: these are just my models. I'll be delighted if littlesymaar (or someone else) knows this extremely well and manages to enlighten me because it would be fun to really "get" it.

Edits: A lot.

Also: Just operating one power plant can be complex: I remember one presentation from former students or something about how important it was in Eastern Europe back then to be ready to cut immediately if neighbouring plants failed or cut so that your plant wouldn't suddenly oversupply and burn out.


Something that might help: It isn't one axle twisting. Any one source or sink on the grid can model themselves as a single shaft coupled to the rest of the grid, but for conceptualizing it as a whole you really do have to think of it as a system of coupled machines.

One thing that might help with the stability intuition: The generators themselves are synchronous machines, but they have parasitic induction machines deliberately installed in the form of damper bars. Those damper bars mean that the vibrating mode between the generator rotor and the stator field is well-damped. Similarly, the vast majority of total load is in the form of induction motors which naturally have a damped response between the motor rotor and stator field. So resonating patterns and shocks are quite difficult to set up. Even sharp step inputs are attenuated to be not-sharp over short distances.

In the US, one common test for grid step response is dropping an entire nuclear power plant. Not one reactor: the entire site. So several GW of electric supply is dropped instantaneously. The limit typically isn't any kind of oscillation, its the ability of primary frequency reserve to pick up the slack before you start triggering under-frequency trips.


I think the idea of a common shaft, only provided by electricity, is a good approximation.

But the shaft of such length is not very "rigid", and the task is to keep it from wringing and breaking by keeping all parts of it rotating at the same speed.

(As a side note: have you heard of selsyns?)


Tell us more about that :)


I cannot really but I have put my lack of knowledge on public display here for someone else to help me and you : https://news.ycombinator.com/item?id=25917037

: )


Thanks. I've already read that ;) But let me say this. About 10 to 15 years ago I've been more interested in transmission lines because of "Energiewende" and such, came across HVDC, how that would enable smart grids, possibly protection against solar/geomagnetic storms because of faster switching times, and so on. I looked into how they built those HVDC-lines in China, how that is not new at all, but eased by modern solid-state power semiconductors, and so on. And looked further into patents, modulation schemes, even many 'papers'(pdf) and also very technical books(skimmed those only).

Anyway, what I learned was that the grid has many very nonlinear phenomena which are extremely hard to model and predict, because of the dynamics. Much more complicated than the simplistic answers some here have given you. So...relax?

:-)


Relevant Wikipedia article: https://en.wikipedia.org/wiki/Synchronous_grid_of_Continenta...

Must admit I’ve never heard of this before :-)


Funny how the folks on this site will still wade into an article even with a completely unrecognizable (yet curious) title like this one.


I was wondering if someone could explain the countermeasures for such an event. Obviously, as the article states, producers are being shut off in the regions with surplus, while drains, who can afford to shut off, are shut off in the deficit regions.

Is this an automatic process? Or is it more like someone from the company's energy provider calls them and tells them to shut off some devices? And is there not a potential problem, that if too many shut of at once, you now have a surplus again? Or is it coordinated by one single entity?


The company I work for provides Demand Response to the Irish grid operator.

In cases like this, our systems would detect the frequency deviation, and shut off loads within 100 milliseonds to reduce the demand on the grid. This helps in cases where demand is greater than supply.

The entire system is automated - the required time frames are so quick that you don't have time for humans to be involved. By the time we're aware that an event has occurred, we've already reduced demand on the grid.

Handling high frequency events where supply is greater than demand is tricker. Sites that have long running generation can be instructed to shut down their generation, but large-scale batteries are probably the best solution in these cases. They can be switched quickly to start charging (if they have spare capacity).

As you've identified, one potential issue is that you can end up over-responding to the event and move from a low frequency event to a high frequency event.

The way we do in in Ireland is that our response is proportional to the frequency nadir. Not everything is tripped off at the same time.

As other posters have noted, the actual frequency deviations that occurred are not that big. 49.7 Hz is not that low compared to normal grid frequency. In fact, some of our systems wouldn't even activate at this level. They would see it, but wouldn't trip off any loads.


I’m curious about this. What kinds of load is your company is running? How do you take them offline and online so quickly without affecting production or leading to long restart cycles?


We control a mixture of loads, but for Fast Frequency Response (FFR), it's mainly compressor/ motors, with the occassional mains breaker thrown in. Large cement factories, refrigeration companies etc.

As for how we control them so quickly, it's usually via a direct connection to a breaker of some description, or where there's a SCADA system that can trip out the loads in the required time frames. Before the site can participate in these services, we perform extensive testing to ensure they meet the required time constraints.

Our system monitors the grid voltage and current at 8 kHz, and we down sample that to 50 Hz (average grid frequency). We can detect a frequency event across 3 phases within 60 milliseconds. (We check all three phases for multiple cycles to reduce any false positives.)

When we trip out the loads, production stops. We'll notify the client why their loads have been turned off, but the SMS message will usually arrive a few seconds after the trip.

The clients that participate in these services get paid quite well to make their loads available and they're aware of the process. They agree to turn off demand without notice when there's a frequency event.

It's not suitable for every site. We work quite a lot with pharma companies for "regular" demand response, but very few can do FFR. Shutting down a pharma plant with no notice can cost a lot of money in wasted product and down time as they clear the wasted product off the production lines.


The grid frequency is a measure of energy balance in the grid. So anyone can measure anywhere in the frequency domain whether there is a surplus or deficit of generation. This is due to the fact that the grid itself can't store any energy, it has to be balanced all the time.

Imagine the massive spinning generators as a big mass that slow down just a little bit when you switch on a light and then that generator has to add more steam (or open hydro valve or whatever).

So anyone with an accurate enough measuring device can exactly monitor the state of the grid. We use this device [0] for example.

There are generally frequency containment reserves (FCR) that consist of different ways of generation and have their different reaction times, power and energy capacities.

Hydro for example can react in about 15 seconds, battery inverters in milliseconds. Gas turbines in minutes, coal fired plants in hours.

You can also shed energy by switching off loads (Demand side response).

The system operator is responsible for grid balancing in the short term, they have direct facilities under their control and they have contracts with generators and consumers. And there are markets to bid your generation and flexibility.

The markets in the Nordics for example are:

- FCR-N (Frequency containment Reserve - Normal operations)

- between 49.90 - 49.99 and 50.01 - 50.10 (reaction time up to 20s)

- FCR-D (Disturbance) - between 49.7-49.90 and 50.10 - 50.30 (reaction time up to 2 seconds IIRC)

- FFR (Fast Frequency response) - below 49.7 - reaction time 0.6s IIRC

Once a day you bid your capacity for the next 24h (for each hour) and then you measure the grid frequency yourself and when you detect a deviation you activate your response. You get paid for availability and activation separately. There is a ton of qualification and logging you need to do to be able to participate but the activation message is the grid frequency itself, no further communication needed.

Outside frequency regulation there is energy markets where generation and consumption is agreed 24h ahead.

[0] https://www.gobmaier.de/


> Imagine the massive spinning generators as a big mass that slow down just a little bit when you switch on a light and then that generator has to add more steam (or open hydro valve or whatever).

I have always wondered, would that still hold true if the grid was fully solar-based? There would be no rotating mass in that case.


Yes, although a significant issue with that is there is no spinning mass to "borrow" inertia from, so while modern inverters at solar sites are good at shaping output phases, they have very little capability to absorb significant frequency deviations.

Grid scale battery systems are often used for voltage or frequency stability as opposed to deep discharging as generation offsets, although that will change eventually if batteries get better enough or really cheap LNG stops being a thing.


Yes indeed, inverter based generators (solar and batteries) don't have any inertia. But as long as the grid is still AC the frequency will be the same and the inverters would need to compensate internally. Especially with solar where a cloud going over a solar farm can easily knock off a few MW in seconds the volatility of the grid will increase and need for storage (batteries, hydro, etc) alongside Demand Side Response is getting much bigger.


It does hold true for some systems. Wind turbine generator controls can provide virtual inertia over very short time scales (a second or few) by exchanging energy with the turbine rotor. You can also provide primary frequency reserve in the negative direction (load step-off) using exactly the same ramp control that steam plants use. Typical ramp rate is 100% of rated power for a 5% change in frequency.

However, in order to provide primary frequency reserve in the other direction, you do need additional local storage. You don't need very much. Just 10% of rated power for 15 minutes gets you to very deep renewable penetration.

The trouble isn't with the technology, its with the economics. Once you set a sufficiently high price for frequency support and primary frequency reserve, suppliers will show up.


These are called ancillary services. There are 3 levels: frequency control (for example dams), primary reverse (fast acting power, in less than 10s), and secondary reserve (slow acting: < 15min).

Participants can either be positive (they consume more energy, for example PHES systems), or negative (they inject or consume less energy).

All these ancillary services are paid (annual auction + per case). Nowadays, it's getting bigger with VPP: virtual power plants, which aggregate small loads (i.e. small ~1MW generators) in order to propose a bigger load to TSOs.

It's all automatic

If you're interested: https://www.swissgrid.ch/en/home/customers/ancillary-service...


On top of that, newer solar inverters MUST (at least in the EU) reduce power if frequency rises in order to automatically shed power in cases of extreme supply


"Shutting off drains" is known as demand side response or DSR. It's often cheaper, faster, and more environmentally friendly to pay industrial customers to cut their demand by x MW than to fire up x MW of ancillary generation. And as far as grid frequency/balance is concerned, has exactly the same effect.

Often such customers will have flexible demand in the form of non-critical heating or cooling, pumps that only need to run some of the time, etc, which they are very happy to turn off temporarily in return for extra income.

This is indeed an automatic process, triggered in near-real time in response to signals from the grid.


It is an automatic process, coordinated - as far as I'm aware - by two national energy grid providers (Switzerland and Germany) who have been selected for this role.


The response is automated and every system operator provides some ressources to react: https://en.wikipedia.org/wiki/Operating_reserve


They talk about it in the article as contracted interruptible services that got shut down in France and Italy to reduce the draw on the grid.


Something similar happened in 2018. However, it was not a technical problem but a conflict between the power grid operators in Serbia and Kosovo.

https://www.dw.com/en/clocks-in-europe-are-running-late-beca...

I believe that if you adjusted your clock during the conflict you had to adjust it again when it was resolved as the resolution was to increase the frequency for a period to reverse the loss of frequency.



Seems like an excellent example that preventing a single point of failure means nothing if your system doesn't have the extra capacity to handle a single failure.


There was multiple failures, not single.

The grid was separated but it didn't cause more power outages.


Aha so this is what caused that big frequency drop people had already reported. Interesting!


I wonder what are the benefits of keeping the grid synchronous?

Would it be possible to have multiple smaller grids, still interconnected, but without being kept in sync?

I'm not sure if what I'm saying is possible or efficient, but converting AC to DC, transmitting the energy, then converting DC to AC so the frequency becomes irrelevant.


> I wonder what are the benefits of keeping the grid synchronous?

> Wide area synchronous networks improve reliability and permit the pooling of resources. Also, they can level out the load, which reduces the required generating capacity, allow more environmentally-friendly power to be employed; and allow more diverse power generation schemes and permit economies of scale. [0]

[0] https://en.m.wikipedia.org/wiki/Wide_area_synchronous_grid


It is absolutely possible and exactly how connections between the different grids are done, e.g. between the power grids of North America or between the European grid and Russia.

HVDC of course has the disadvantage of extra conversion equipment. HV circuit breakers are also significantly more complex as an arc will form as long as current is flowing. With AC this happens at the zero automatically, nothing like this with DC.


Does not even need to be between countries for example east and west of Denmark are on two different grids which only recently got connected 10 years ago. (Large HVDC under the great belt) I am pretty sure the map on the site is "wrong" as Sjælland (Zealand) the large eastern island of Denmark is on the nordic grid, and so is Bornholm which nearly exclusivly gets it power from Sweden.


The map on Wikipedia is better, as it divides Denmark correctly, and also excludes several other large European islands that are not connected to the grid.

https://en.wikipedia.org/wiki/Synchronous_grid_of_Continenta...


You can see on this map the purple DC lines between England and France and Ireland https://www.entsoe.eu/data/map/

I am amazed the grid goes all the way to Iraq


Multiple small grids (probably not interconnected) would be like microgrids.

In general, interconnection allows for the sharing of generators. My 30 year old coal unit might not need to run because I can buy electricity from your cheaper gas plant.


For one, it's used for timekeeping.


That's more of a side-effect, not the goal of a synchronous grid.


There was a incident in Turkey (31 march 2015) that caused a country wide blackout. From what I understand the west side and the east side loose sync. The west side had under supply and the east side had over supply.


If you design an electricity grid to never partition, then you need to keep a substantial headroom of transmission equipment unused to prevent cascading failures leading to a partition in case of failure of just one or two sites.

Instead, an electricity grid should be able to survive any partition like this while still keeping all frequencies within the nominal range. The total cost of such a system is lower in most cases (you need more idling generation capacity or droppable load, but less transmission capacity)

The EU system here failed to do either here.


The frequencies were in nominal range. The lowest was around 49.7, the highest about 50.6, both of which are within the allowed range of frequency (47.5 - 52.5 is the absolute emergency range, 49-51 is the nominal range, before you start capping power to or from customers).

Romania had a power blackout for a bit, but everything worked as intended and the romanian grid blacking out didn't pull in the rest of the european grid.

A partition between power grid is acceptable to preserve the largest amount of the grid remaining functional. Marrying the partitions together is bothersome but not something that takes forever (took 1 hour in this case).

While 49.7 is very low, most of your devices will not notice and will be fine. Most industrial equipment is largely already coded to handle frequency shifts in favor of keeping the grid stable.


I understand that some equipment, such as consumer grade solar panels, already disconnects at 49.8, and one early analysis I read[0] said that that in fact exacerbated the problem somewhat on 8 January.

[0] https://www.hoogspanningsnet.com/ , entry of 08 januari 2021 - similar sentiments were expressed in articles linked from there


Power producers != Consumers for small devices like consumer grade solar panels, especially if the inverter is too cheap to handle a frequency excursion.


I'm not sure this counts as a failure.

The load shedding mechanisms worked, supportive power generation was automatically activated and there was no widespread blackout.

You are right that it appears like there was insufficient capacity at the north-west/south-east separation point and I guess that is going to be investigated but apart from that everything looks like it worked.


The system survived the partitions, there was no widespread blackout. Further, the system is not organized by the EU, but the ENTSO-E and contains e.g. countries from North Afrika.


There was no blackout _yet_, but it was very close.

According to the German Bundesnetzagentur (the equivalent of the FTC/EIA), the number of times where they have to intervene with the grid due to grid instability is constantly rising due to Germany shutting down nuclear and coal plants.

> https://www.bundesnetzagentur.de/DE/Sachgebiete/Elektrizitae...


> There was no blackout _yet_, but it was very close.

Where do you get that from? None of the sources reported a close blackout, as far as I understood it there was a lot of emergency capacity left. We weren't even in the emergency frequency range, as the other commenter pointed out.

Even the linked article just states that those interventions got more often after shutting down coal+nuclear, but it's not critical, it _only_ costs money to compensate the operators: https://de.wikipedia.org/wiki/Redispatch_(Stromnetz)

It's probably much less money than all the nuclear subsidies.


According to this chart there has been an insignificant reduction in installed capacity.

https://energy-charts.info/charts/installed_power/chart.htm?...

Nuclear power went from 20GW to 10GW. Hard coal went from 28GW to 22GW.

Ok, but gas went up from 23GW to 29GW. Brown coal stayed the same.

Renewables went up by by 50GW for PV and 50GW for wind. Consider that wind often hits a 50% capacity factor. That is 25GW in additional power just from wind alone.

Some of those plants may not be running continuously but they are still useful for emergency responses.


It is a choice, spend a fortune on nuclear power that nobody wants in their own backyard, or deal with a less stable grid due to solar and wind energy.


Sigh. It's not a dichotomy at all.

It's perfectly possible to have a stable grid even with solar and wind.

The problem is that neither nuclear nor coal power stations are load following and "base load" has lowered over the years (industry has moved to Asia, devices became more efficient, etc.).

Stand-by power (like natural gas powered generators) hasn't been built up the way the way it should have been. Same goes for smart grid technologies and buffer storage; not to mention the maniacs (especially in South Germany) who basically protest everything - from nuclear power, to wind power, to required infrastructure like north-to-south high-voltage transmission lines.

It's way too oversimplified to reduce the issue to just wind and solar.




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: