by bretthoerner on 11/14/23, 3:42 PM with 290 comments
by meteo-jeff on 11/14/23, 5:28 PM
Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable.
by robertlagrant on 11/14/23, 4:52 PM
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
by xnx on 11/14/23, 4:04 PM
by EricLeer on 11/15/23, 10:39 AM
by serjester on 11/14/23, 5:24 PM
> ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute resources and higher resolution data.
I can't wait to see how far other people take this.
by dauertewigkeit on 11/15/23, 6:34 PM
Full disclosure: I think DeepMind often publish these bombastic headlines about their models which often don't live up to their hype, or at least that was my personal experience. They have a good PR team, anyway.
by stabbles on 11/14/23, 8:28 PM
There are really convenient apps that show an animated map with radar data of rain, historical data + prediction (typically).
The prediction is always completely bonkers.
You can eyeball it better.
No wonder "AI" can improve that. Even linear extrapolation is better.
Yes, local rain prediction is a different thing from global forecasting.
[1] https://www.buienradar.nl [2] https://www.meteoschweiz.admin.ch/service-und-publikationen/...
by lispisok on 11/14/23, 4:54 PM
Weather forecasting has been moving focus towards ensembles to account for uncertainty in forecasts. I see a future of large ensembles of ML models being ran hourly incorporating the latest measurements
by freedomben on 11/14/23, 5:17 PM
by Vagantem on 11/14/23, 10:00 PM
by matsemann on 11/15/23, 3:42 PM
I'm all for better weather data, it's quite critical up in the mountains, so that's why my question about how reliable it is in life&death situations.
by Gys on 11/14/23, 4:55 PM
by crazygringo on 11/14/23, 11:55 PM
Apple moved from using The Weather Channel to their own forecasting a year ago [1].
Using AI to produce better weather forecasts is exactly the kind of thing that is right up Google's alley -- I'm very happy to see this, and can't wait for this to get built into our weather apps.
by layoric on 11/14/23, 10:43 PM
My experience in this space is that I was first employee at Solcast building a live 'nowcast' system for 4+ years (left ~2021) targeting solar radiation and cloud opacity initially, but expanding into all aspects of weather, focusing on the use of the newer generation of satellites, but also heavily using NWP models like ECMWF. Last I knew,nowcasts were made in minutes on a decent size cluster of systems, and has been shown in various studies and comparisons to produce extremely accurate data (This article claims 'the best' without links which is weird..), be interesting on how many TPUsv4 were used to produce these forecasts and how quickly? Solcast used ML as a part of their systems, but when it comes down to it, there is a lot more operationally to producing accurate and reliable forecasts, eg it would be arrogant to say the least to switch from something like ECMWF to this black box anytime soon.
Something I said as just before I left Solcast was that their biggest competition would come from Amazon/Google/Microsoft and not other incumbent weather companies. They have some really smart modelers, but its hard to compete with big tech resources. I believe Amazon has been acquiring power usage IoT related companies over the past few years, I can see AI heavily moving into that space as well.. for better or worse.
by hackitup7 on 11/15/23, 5:21 PM
by pyb on 11/14/23, 8:06 PM
by brap on 11/14/23, 8:37 PM
I would imagine we probably have a solid mathematical model of how weather behaves, so given enough resources to measure and calculate, could you, in theory, predict the daily weather going 10 years into the future? Or is there something inherently “random” there?
by tony_cannistra on 11/15/23, 4:25 PM
The Allen Institute has worked on it for a while, and has hired quite a few PhDs (https://allenai.org/climate-modeling).
by thriftwy on 11/15/23, 7:05 PM
by jauntywundrkind on 11/14/23, 6:00 PM
Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely at great pain)...
by rldjbpin on 11/16/23, 9:05 AM
the resolution, while seemingly impressive, is very imprecise compared to the SOTA in the theoretical modelling side.
this discredits the computational claims made by the paper for me. i understand that the current simulations can go down to meter scale, but i wonder what the compuational requirements are when you calculate for this resolution.
by comment_ran on 11/14/23, 7:25 PM
If it is possible, then I will try using the sensor to measure my velocity at some place where I live, and I can run the model and see how the results look like. I don't know if it's going to accurately predict the future or within a 10% error bar range.
by joegibbs on 11/15/23, 1:27 AM
by user_7832 on 11/14/23, 5:59 PM
To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?
by max_ on 11/14/23, 7:56 PM
by carabiner on 11/14/23, 7:31 PM
Any way to run this at even higher resolution, like 1 km? Could this resolve terrain forced effects like lenticular clouds on mountain tops?
by miserableuse on 11/14/23, 7:59 PM
by devit on 11/15/23, 10:32 PM
by hammad93 on 11/15/23, 1:52 AM
by syntaxing on 11/14/23, 6:50 PM
by tchvil on 11/15/23, 5:00 PM
by whoislewys_1 on 11/15/23, 3:00 AM
Is it inevitable that all market alpha gets mined by AI?
by haolez on 11/14/23, 7:59 PM
by csours on 11/14/23, 10:35 PM
by dnlkwk on 11/14/23, 8:23 PM
by supdudesupdude on 11/14/23, 9:56 PM
by knicholes on 11/14/23, 10:04 PM
by rottc0dd on 11/15/23, 8:31 AM
by simonebrunozzi on 11/14/23, 8:10 PM
by sagarpatil on 11/14/23, 6:18 PM
by max_ on 11/14/23, 7:46 PM
Does GraphCast come close to them?
by isaacfrond on 11/15/23, 3:23 PM
You'd think predicting the weather is mostly a matter of fast computation. The physical rules are well understood, so to get a better estimate use a finer mesh in your finite element computation and use a smaller time scale in estimating your differential equations.
Neural networks are notoriously bad at exact approximation. I mean you can never beat a calculator when the issue is doing calculations.
So apparently the AI found some shortcut for doing the actual computational work. That is also surprising as weather is a chaotic system. Shortcuts should not exist.
Long story short, I don't get what's going on here.
by mg on 11/15/23, 3:36 PM
OpenAI has become one of the fastest growing companies of all time. And much of it is based on Google's "Attention is all you need" and other papers.
Since Microsoft added the Dall-E 3 image creator to Bing, Bing saw a huge inflow of new users. Dall-E is also a technology rooted in Google papers.
I wonder how Google thinks about this internally.
by max_ on 11/14/23, 7:44 PM
Why? Other AI studios seem to work on gimmicks while DeepMind seems to work on genuinely useful AI applications [0].
Thanks for the good work!
[0] Not to say that Chat GPT & Midjourney are not useful, I just find DeepMinds quality of research more interesting.
by alberth on 11/15/23, 5:33 PM
Aren't existing weather forecasting models, already a form of "AI"?
I'm no AI/ML expert, but isn't the real story here is that a new model (like GPT-4.0) is better than the previous/existing model (GPT-3.5).
It's just grabs way more attention calling the new model "AI" (vs not referring to the old as such).
by hexo on 11/15/23, 5:04 PM
by acolderentity on 11/15/23, 4:44 PM
by cryptoz on 11/14/23, 5:50 PM
by tokai on 11/15/23, 4:36 PM
by drcongo on 11/15/23, 5:24 PM
by amluto on 11/14/23, 5:45 PM
On top of this, surely the actual observations that feed into the model are terrible -- they come from weather stations, sounding rockets, balloons, radar, etc, none of which seem likely to be especially accurate in all locations. Except that, where a weather station exists, the output of that station is the observation that people care about -- unless you're in an airplane, you don't personally care about the geopotential, but you do care about how windy it is, what the temperature and humidity are, and how much precipitation there is.
ISTM these dynamics ought to be better captured by learning them from actual observations than from trying to map physics both ways onto the rather limited datasets that are available. And a trained model could also learn about the idiosyncrasies of the observation and the extra bits of forcing (buildings, etc) that simply are not captured by the inputs.
(Heck, my personal in-my-head neural network can learn a mapping from NWS forecasts to NWS observations later in the same day that seems better than what the NWS itself produces. Surely someone could train a very simple model that takes NWS forecasts as inputs and produces its estimates of NWS observations during the forecast period as outputs, thus handling things like "the NWS consistently underestimates the daily high temperature at such-and-such location during a summer heat wave.")
by greatpostman on 11/15/23, 3:27 PM