I have been working in the same field, quite a few years in my own forecasting company and then later on as a side project.
Not wanting to steal the thunder of Deepmind, but I feel should mention my site which is currently beta as in "works for a few electricity power load and supply forecasting scenarios" and quite well so. Also, it's out there, cheap and fast.
Unfortunately, it's only in German but the basic rule is:
- upload an excel (or csv) file
- wait a minute or so (for a typical dataset of say 3 years of hourly data)
- get back the results
There are a few design rules how the excel file should look like, e.g.,
- a date/time series per sheet (it accepts most English and European formats) and as many number series as you want (always use a decimal point instead of comma here)
- one series has to have consecutive missing numbers at the end, this will be the target value
Actually, I use both but my production models are in Clojure.
I often end up implementing minor things myself using lower level abstractions (e.g., Linear Regressions or PCA with whitening using Matrix libraries) and I check the results and/or try new things using scikit-learn.
So in general, I'd say I do the programming (outputing intermediate CSVs, tests, web service, thread handling, UI, ...) in Clojure(Script), and try other approaches (e.g., other models/parameters/...) in Python.
I'm quite happy with this pipeline but probably to some extent because I really love to understand how things work and nothing pushes you to learn as much as a missing function in your ML library :-)
I have previously built such a tool for retail sales forecasting (based on previous sales). The stack used R + Rshiny (web app). Method FB's `prophet` as it seemed faster and more accounting of holidays and other variations.
You can still do cutting edge work with 3 small layers, for example most of the advances in robotic control (using neural networks) use networks with similar sizes.
> Using a neural network trained on widely available weather forecasts and historical turbine data, we configured the DeepMind system to predict wind power output 36 hours ahead of actual generation.
Interesting. Something I've long wondered about:
- Many wind farms are located in very specific places that have measured high winds and are otherwise good places to locate a wind farm.
- Global climate change is changing local climates too. Are the studies that were done 10-20 years ago to discover ideal wind farm locations no longer as accurate as they used to be?
- How are wind farm locations decided today? Is there is skill in predicting where a wind farm might be optimally located in 5 years rather than where it should be located today? Or if not (since probably not) what are the right risk mitigation actions to take if you are planning a wind farm but unsure about location?
- Since Machine Learning is useful in improving the value of the existing wind power, could it also be useful in this endeavor, finding optimal wind power generation locations?
- Lastly, is this something a person could research at home with open data and code? Just curious. :)
Is this a substantial improvement over existing neural network models? I’ve seen models that were being used to trade renewables on PJM’s market that seemed at least as good as these. And, similar models are currently being used to manage DERs in micro-grids.
I don't mean to downplay these results. I’m not an expert. Just mentioning it in case someone at deepmind sees this and wants to reach out to PJM. I believe those models were created almost 10 years ago.
It is probably not. Their neural network to optimize their warehouse efficiency had 6 neurons. The neural network used here is probably pretty small too
If anyone is interested in this type of work, our team at Amperon is hiring data engineers, data scientists, and front end engineers. We’re funded by SV Angel and Notation Capital and we’ve already got traction in ERCOT (Texas).
How exactly does this need machine learning? Naively I would assume that it's somewhat directly correlated with wind speed (and little more, maybe direction) and a very simple model based on that data will give you good predictions.
Weather forecasting is difficult problem but it's problem that as far as I know has been worked on intensely for quite a while.
Weather forecasting is based on physics-based models that, so far as I know, are essentially correct with the fundamental barrier to prediction being a combination of random effects and the difficulty of solving the complex equations numerically.
Deep learning involves training a heuristically approximate a system based on past data. It has been used to emulate various human-learning behaviors like recognizing images or recognizing good or bad position in games. Essentially, the visible successes are in "we don't know but can extrapolate roughly from data" (or we "know" but can't easily program it, in the case of image recognition). So I'd be surprised if deep learning do a good or better job in situation where we do know how things work, we already running algorithm which is correct - the situation with weather.
I was super excited when I read the headline and saw that deepmind published the article. I was hoping for something along the lines of applying reinforcement learning on the turbine or park level to optimize power output. Finding the optimal pitch angles, better yawing algorithms (yaw misalignemnts are one of the most common causes of underperforming wind turbines), or finding optimal yaw directions for the entire wind park (downwind wake effects can have a negative impact on other turbines)...
Unfortunately this is really only about predicting wind speeds to then forecast power output. IIRC, grid codes in some countries are starting to require 24 hour, 15 minute interval forecasts for renewables? I've also seen this feature being advertised by other companies over the years as well, although I don't recall seeing any numbers on accuracy for any of the models.
> Our hope is that this kind of machine learning approach can strengthen the business case for wind power and drive further adoption of carbon-free energy on electric grids worldwide. Researchers and practitioners across the energy industry are developing novel ideas for how society can make the most of variable power sources like solar and wind. We’re eager to join them in exploring general availability of these cloud-based machine learning strategies.
Would be nice to know a bit more detail? Which power grids was this approach tested on? How does this approach work exactly? Are we talking about DNN on time series? The graphs also really need labels to inform.
>> ... join them in exploring general availability of these cloud-based machine learning strategies.
... is probably the operative part. Possibly uncharitable, but this appears to be ‘thought leadership’ marketing generally supporting the idea of GCP managed services.
Of course it can, but only on the margin as this article suggests. The real potential is in figuring out where to put them, what to put in the spots, and most importantly, who to choose to build them and support them given the aforementioned variables. There may also be some advances to be made in the actual design, but I do not believe machine learning is going to help so much there.
Meteorologists have long used a metric, forecast skill, to evaluate different techniques. It would be good if the researchers used standard benchmarks.
ML is probably much more useful in understanding the demand for power, the reaction to weather and current events. But even still, it seems a stretch.
Perhaps some people are missing the point here?. The post is not about the technical challenge/novel but rather how existing ML techniques can play a part in making reusable energy more efficient.
Not wanting to steal the thunder of Deepmind, but I feel should mention my site which is currently beta as in "works for a few electricity power load and supply forecasting scenarios" and quite well so. Also, it's out there, cheap and fast.
https://ausblick.cryptoport.net
Unfortunately, it's only in German but the basic rule is:
- upload an excel (or csv) file
- wait a minute or so (for a typical dataset of say 3 years of hourly data)
- get back the results
There are a few design rules how the excel file should look like, e.g.,
- a date/time series per sheet (it accepts most English and European formats) and as many number series as you want (always use a decimal point instead of comma here)
- one series has to have consecutive missing numbers at the end, this will be the target value
Other than that, it should just work.
Feel free to comment.
PS: The examples might be the best explanation - https://ausblick.cryptoport.net/examples