{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "06a01116-fbbe-49f0-9dc6-c701edb23cb7", "metadata": { "id": "06a01116-fbbe-49f0-9dc6-c701edb23cb7" }, "outputs": [], "source": [ "import xarray as xr" ] }, { "cell_type": "markdown", "id": "600ee6e5-5aa5-40b7-97b3-2645b91a4e2a", "metadata": { "id": "600ee6e5-5aa5-40b7-97b3-2645b91a4e2a" }, "source": [ "\n", "# WeatherBench 2 Data Guide\n", "\n", "One core part of WeatherBench 2 are ready-to-use, cloud-based datasets. This page lists and describes all the available datasets.\n", "\n", "The datasets are stored in this public Google Cloud bucket: [`gs://weatherbench2/datasets`](https://console.cloud.google.com/storage/browser/weatherbench2).\n", "\n", "**Please also check the LICENSE files for each dataset in the respective GCS buckets.** Some datasets allow commercial use. Others only permit research use.\n", "\n", "### A note on resolutions\n", "\n", "We provide the datasets at different resolutions. All files will have the number of longitude X latitude grid points in their filename, e.g. `64x32`. For the WeatherBench 2 paper, all evaluation was done at `240x121` = 1.5 degree resolution. All datasets were regridded using first-order conservative regridding, i.e., with weights proportional to the area of overlap between grid cells on the original and desired grids.\n", "\n", "The `1440x721` (= 0.25 degrees) and `240x121` files contain the poles, i.e. -90 and 90 degree latitude, denoted with `with_poles`. `64x32` files do not contain the poles to ensure equal spacing.\n", "\n", "## Ground-truth datasets\n", "\n", "### ERA5\n", "\n", "Our ERA5 datasets were downloaded from the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/) and have a time range from 1959 to 2023 (incl.). The data here have been downsampled to 6h and 13 levels, even though a raw hourly dataset with 37 levels is also available at `gs://weatherbench2/datasets/era5/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr`\n", "\n", "Location: [`gs://weatherbench2/datasets/era5/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/era5/)\n", "\n", "Files:\n", "* `1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr`\n", "* `1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr`\n", "* `1959-2023_01_10-6h-240x121_equiangular_with_poles_conservative.zarr`\n", "* `1959-2023_01_10-6h-64x32_equiangular_conservative.zarr`\n", "\n", "*Note: Older version of the ERA5 files exist in the bucket to ensure continuity.*\n", "\n", "See output below for a list of variables. The file also contains several derived variables which were computed using these [methods](https://weatherbench2.readthedocs.io/en/latest/api.html#derived-variables)." ] }, { "cell_type": "code", "execution_count": null, "id": "972c9516-847b-4813-9618-1ce54b6dad40", "metadata": { "id": "972c9516-847b-4813-9618-1ce54b6dad40", "outputId": "50d20543-29e3-4c76-8ef2-9f97909387ce" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                                           (time: 93544,\n",
       "                                                       latitude: 721,\n",
       "                                                       longitude: 1440,\n",
       "                                                       level: 13)\n",
       "Coordinates:\n",
       "  * latitude                                          (latitude) float32 90.0...\n",
       "  * level                                             (level) int64 50 ... 1000\n",
       "  * longitude                                         (longitude) float32 0.0...\n",
       "  * time                                              (time) datetime64[ns] 1...\n",
       "Data variables: (12/62)\n",
       "    10m_u_component_of_wind                           (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind                           (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed                                    (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    2m_dewpoint_temperature                           (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature                                    (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    above_ground                                      (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    ...                                                ...\n",
       "    volumetric_soil_water_layer_1                     (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    volumetric_soil_water_layer_2                     (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    volumetric_soil_water_layer_3                     (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    volumetric_soil_water_layer_4                     (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    vorticity                                         (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                                        (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 93544,\n", " latitude: 721,\n", " longitude: 1440,\n", " level: 13)\n", "Coordinates:\n", " * latitude (latitude) float32 90.0...\n", " * level (level) int64 50 ... 1000\n", " * longitude (longitude) float32 0.0...\n", " * time (time) datetime64[ns] 1...\n", "Data variables: (12/62)\n", " 10m_u_component_of_wind (time, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, latitude, longitude) float32 dask.array\n", " 2m_dewpoint_temperature (time, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, latitude, longitude) float32 dask.array\n", " above_ground (time, level, latitude, longitude) float32 dask.array\n", " ... ...\n", " volumetric_soil_water_layer_1 (time, latitude, longitude) float32 dask.array\n", " volumetric_soil_water_layer_2 (time, latitude, longitude) float32 dask.array\n", " volumetric_soil_water_layer_3 (time, latitude, longitude) float32 dask.array\n", " volumetric_soil_water_layer_4 (time, latitude, longitude) float32 dask.array\n", " vorticity (time, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/era5/1959-2023_01_10-wb13-6h-1440x721_with_derived_variables.zarr')" ] }, { "cell_type": "markdown", "id": "8ead4f44-d62b-4712-9057-bee2b1d589cb", "metadata": { "id": "8ead4f44-d62b-4712-9057-bee2b1d589cb" }, "source": [ "### ERA5 Climatology\n", "\n", "A climatology is used for e.g. computing anomaly metrics such as the ACC. For WeatherBench 2, the climatology was computed using a running window for smoothing (see paper and script) for each day of year and sixth hour of day. We have computed climatologies for 1990-2017 and 1990-2019.\n", "\n", "Location: [`gs://weatherbench2/datasets/era5-hourly-climatology/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/era5-hourly-climatology/)\n", "\n", "Files:\n", "* `1990-2017_6h_1440x721.zarr`\n", "* `1990-2017_6h_512x256_equiangular_conservative.zarr`\n", "* `1990-2017_6h_240x121_equiangular_with_poles_conservative.zarr`\n", "* `1990-2017_6h_64x32_equiangular_conservative.zarr`\n", "* `1990-2019_6h_1440x721.zarr`\n", "* `1990-2019_6h_512x256_equiangular_conservative.zarr`\n", "* `1990-2019_6h_240x121_equiangular_with_poles_conservative.zarr`\n", "* `1990-2019_6h_64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "1047f416-37cd-49c9-9ca6-c7158efb7f46", "metadata": { "id": "1047f416-37cd-49c9-9ca6-c7158efb7f46", "outputId": "79e4ac7f-f59b-45d6-cb27-6cc7495d0e6b" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                                         (hour: 4, dayofyear: 366,\n",
       "                                                     latitude: 721,\n",
       "                                                     longitude: 1440, level: 13)\n",
       "Coordinates:\n",
       "  * dayofyear                                       (dayofyear) int64 1 ... 366\n",
       "  * hour                                            (hour) int64 0 6 12 18\n",
       "  * latitude                                        (latitude) float32 90.0 ....\n",
       "  * level                                           (level) int64 50 ... 1000\n",
       "  * longitude                                       (longitude) float32 0.0 ....\n",
       "Data variables: (12/52)\n",
       "    10m_u_component_of_wind                         (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind                         (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed                                  (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    2m_dewpoint_temperature                         (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature                                  (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    ageostrophic_wind_speed                         (hour, dayofyear, level, latitude, longitude) float32 dask.array<chunksize=(3, 3, 1, 721, 1440), meta=np.ndarray>\n",
       "    ...                                              ...\n",
       "    volumetric_soil_water_layer_1                   (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    volumetric_soil_water_layer_2                   (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    volumetric_soil_water_layer_3                   (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    volumetric_soil_water_layer_4                   (hour, dayofyear, latitude, longitude) float32 dask.array<chunksize=(3, 3, 721, 1440), meta=np.ndarray>\n",
       "    vorticity                                       (hour, dayofyear, level, latitude, longitude) float32 dask.array<chunksize=(3, 3, 1, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                                      (hour, dayofyear, level, latitude, longitude) float32 dask.array<chunksize=(3, 3, 1, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (hour: 4, dayofyear: 366,\n", " latitude: 721,\n", " longitude: 1440, level: 13)\n", "Coordinates:\n", " * dayofyear (dayofyear) int64 1 ... 366\n", " * hour (hour) int64 0 6 12 18\n", " * latitude (latitude) float32 90.0 ....\n", " * level (level) int64 50 ... 1000\n", " * longitude (longitude) float32 0.0 ....\n", "Data variables: (12/52)\n", " 10m_u_component_of_wind (hour, dayofyear, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (hour, dayofyear, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (hour, dayofyear, latitude, longitude) float32 dask.array\n", " 2m_dewpoint_temperature (hour, dayofyear, latitude, longitude) float32 dask.array\n", " 2m_temperature (hour, dayofyear, latitude, longitude) float32 dask.array\n", " ageostrophic_wind_speed (hour, dayofyear, level, latitude, longitude) float32 dask.array\n", " ... ...\n", " volumetric_soil_water_layer_1 (hour, dayofyear, latitude, longitude) float32 dask.array\n", " volumetric_soil_water_layer_2 (hour, dayofyear, latitude, longitude) float32 dask.array\n", " volumetric_soil_water_layer_3 (hour, dayofyear, latitude, longitude) float32 dask.array\n", " volumetric_soil_water_layer_4 (hour, dayofyear, latitude, longitude) float32 dask.array\n", " vorticity (hour, dayofyear, level, latitude, longitude) float32 dask.array\n", " wind_speed (hour, dayofyear, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/era5-hourly-climatology/1990-2019_6h_1440x721.zarr')" ] }, { "cell_type": "markdown", "id": "4bd47ee8-44ca-4315-ad48-b6eecb6cc139", "metadata": { "id": "4bd47ee8-44ca-4315-ad48-b6eecb6cc139" }, "source": [ "### IFS HRES t=0 \"Analysis\"\n", "\n", "To evaluate IFS forecasts, we use the IFS analysis as the ground truth. Note that here we use the initial conditions of the HRES forecasts, i.e. the forecasts at lead time zero as analysis. This is not exactly the same as the analysis dataset provided by ECMWF (see paper for details).\n", "\n", "Location: [`gs://weatherbench2/datasets/hres_t0/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/hres_t0/)\n", "\n", "Files:\n", "* `2016-2022-6h-1440x721.zarr`\n", "* `2016-2022-6h-512x256_equiangular_conservative.zarr`\n", "* `2016-2022-6h-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2016-2022-6h-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "72809eb7-e624-4377-9d45-b21353c8e340", "metadata": { "id": "72809eb7-e624-4377-9d45-b21353c8e340", "outputId": "55b7de77-41f3-41e3-d472-ecda410b79a8" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                  (time: 10268, latitude: 721, longitude: 1440,\n",
       "                              level: 13)\n",
       "Coordinates:\n",
       "  * latitude                 (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n",
       "  * level                    (level) int32 50 100 150 200 ... 700 850 925 1000\n",
       "  * longitude                (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * time                     (time) datetime64[ns] 2016-01-01 ... 2023-01-10T...\n",
       "Data variables: (12/14)\n",
       "    10m_u_component_of_wind  (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind  (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed           (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature           (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential             (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure  (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    ...                       ...\n",
       "    temperature              (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr  (time, latitude, longitude) float32 dask.array<chunksize=(1, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind      (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind      (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    vertical_velocity        (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed               (time, level, latitude, longitude) float32 dask.array<chunksize=(1, 13, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 10268, latitude: 721, longitude: 1440,\n", " level: 13)\n", "Coordinates:\n", " * latitude (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n", " * level (level) int32 50 100 150 200 ... 700 850 925 1000\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * time (time) datetime64[ns] 2016-01-01 ... 2023-01-10T...\n", "Data variables: (12/14)\n", " 10m_u_component_of_wind (time, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, latitude, longitude) float32 dask.array\n", " geopotential (time, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, latitude, longitude) float32 dask.array\n", " ... ...\n", " temperature (time, level, latitude, longitude) float32 dask.array\n", " total_precipitation_6hr (time, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, level, latitude, longitude) float32 dask.array\n", " vertical_velocity (time, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/hres_t0/2016-2022-6h-1440x721.zarr')" ] }, { "cell_type": "markdown", "id": "b0fc6e03-9d0e-4cb3-bd1b-4b8de5876e5c", "metadata": { "id": "b0fc6e03-9d0e-4cb3-bd1b-4b8de5876e5c" }, "source": [ "## Forecast datasets\n", "\n", "### IFS HRES\n", "\n", "Here, we provide the 00 and 12 UTC initializations of HRES.\n", "\n", "Location: [`gs://weatherbench2/datasets/hres/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/hres/)\n", "\n", "Files:\n", "* `2016-2022-0012-1440x721.zarr`\n", "* `2016-2022-0012-512x256_equiangular_conservative.zarr`\n", "* `2016-2022-0012-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2016-2022-0012-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "65df0925-2ab3-46a2-a8ef-990d73f92b29", "metadata": { "id": "65df0925-2ab3-46a2-a8ef-990d73f92b29", "outputId": "8e02f8e7-e376-49c8-b311-8462baed87b4" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                   (time: 5134, prediction_timedelta: 41,\n",
       "                               latitude: 721, longitude: 1440, level: 13)\n",
       "Coordinates:\n",
       "  * latitude                  (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n",
       "  * level                     (level) int32 50 100 150 200 ... 700 850 925 1000\n",
       "  * longitude                 (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * prediction_timedelta      (prediction_timedelta) timedelta64[ns] 00:00:00...\n",
       "  * time                      (time) datetime64[ns] 2016-01-01 ... 2023-01-10...\n",
       "Data variables: (12/16)\n",
       "    10m_u_component_of_wind   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential              (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    ...                        ...\n",
       "    total_precipitation_24hr  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind       (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind       (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    vertical_velocity         (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 5134, prediction_timedelta: 41,\n", " latitude: 721, longitude: 1440, level: 13)\n", "Coordinates:\n", " * latitude (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n", " * level (level) int32 50 100 150 200 ... 700 850 925 1000\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 00:00:00...\n", " * time (time) datetime64[ns] 2016-01-01 ... 2023-01-10...\n", "Data variables: (12/16)\n", " 10m_u_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " ... ...\n", " total_precipitation_24hr (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " total_precipitation_6hr (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " vertical_velocity (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/hres/2016-2022-0012-1440x721.zarr')" ] }, { "cell_type": "markdown", "id": "208e4661-caf9-440d-9f95-38ad4b8afe41", "metadata": { "id": "208e4661-caf9-440d-9f95-38ad4b8afe41" }, "source": [ "### IFS ENS\n", "\n", "Downloading the full ensemble takes a very long time. We downloaded ensemble data from the [TIGGE archive](https://apps.ecmwf.int/datasets/data/tigge/levtype=sfc/type=cf/) for 2018 to 2022 (incl.).\n", "\n", "All data from the TIGGE archive can only be used for research purposes. Please check the license for more specific constraints.\n", "\n", "Location: [`gs://weatherbench2/datasets/ifs_ens/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/ifs_ens/)\n", "\n", "Files:\n", "* `2018-2022-1440x721.zarr`\n", "* `2018-2022-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2018-2022-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "bf1c57b3-3408-4c25-a643-e0486156c186", "metadata": { "id": "bf1c57b3-3408-4c25-a643-e0486156c186", "outputId": "17d39fa7-8252-4eb2-bf38-34cc9ca52c13" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                   (time: 3652, number: 50,\n",
       "                               prediction_timedelta: 61, latitude: 721,\n",
       "                               longitude: 1440, level: 3)\n",
       "Coordinates:\n",
       "  * latitude                  (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n",
       "  * level                     (level) int32 500 700 850\n",
       "  * longitude                 (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * number                    (number) int32 1 2 3 4 5 6 7 ... 45 46 47 48 49 50\n",
       "  * prediction_timedelta      (prediction_timedelta) timedelta64[ns] 00:00:00...\n",
       "  * time                      (time) datetime64[ns] 2018-01-01 ... 2022-12-31...\n",
       "Data variables: (12/15)\n",
       "    10m_u_component_of_wind   (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind   (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed            (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature            (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential              (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure   (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    ...                        ...\n",
       "    total_precipitation       (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_24hr  (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr   (time, number, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind       (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind       (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 50, 1, 3, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 3652, number: 50,\n", " prediction_timedelta: 61, latitude: 721,\n", " longitude: 1440, level: 3)\n", "Coordinates:\n", " * latitude (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n", " * level (level) int32 500 700 850\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * number (number) int32 1 2 3 4 5 6 7 ... 45 46 47 48 49 50\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 00:00:00...\n", " * time (time) datetime64[ns] 2018-01-01 ... 2022-12-31...\n", "Data variables: (12/15)\n", " 10m_u_component_of_wind (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " ... ...\n", " total_precipitation (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " total_precipitation_24hr (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " total_precipitation_6hr (time, number, prediction_timedelta, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, number, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/ifs_ens/2018-2022-1440x721.zarr')" ] }, { "cell_type": "markdown", "id": "227bfb7b-4dce-419b-a53b-1b55866c67c7", "metadata": { "id": "227bfb7b-4dce-419b-a53b-1b55866c67c7" }, "source": [ "### IFS ENS mean\n", "\n", "We also compute the ensemble mean and save it as a separate file, e.g. to use it as a deterministic baseline.\n", "\n", "Location: [`gs://weatherbench2/datasets/ens/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/ens/)\n", "\n", "Files:\n", "* `2018-2022-1440x721_mean.zarr`\n", "* `2018-2022-240x121_equiangular_with_poles_conservative_mean.zarr`\n", "* `2018-2022-64x32_equiangular_conservative_mean.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "afbc146f-f6e1-40d8-ae72-cdd601755eb3", "metadata": { "id": "afbc146f-f6e1-40d8-ae72-cdd601755eb3", "outputId": "e6b86a67-b4df-436c-8e22-21e9c28b440c" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                   (time: 3652, prediction_timedelta: 61,\n",
       "                               latitude: 721, longitude: 1440, level: 3)\n",
       "Coordinates:\n",
       "  * latitude                  (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n",
       "  * level                     (level) int32 500 700 850\n",
       "  * longitude                 (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * prediction_timedelta      (prediction_timedelta) timedelta64[ns] 00:00:00...\n",
       "  * time                      (time) datetime64[ns] 2018-01-01 ... 2022-12-31...\n",
       "Data variables: (12/15)\n",
       "    10m_u_component_of_wind   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential              (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    ...                        ...\n",
       "    total_precipitation       (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_24hr  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr   (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind       (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind       (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 3652, prediction_timedelta: 61,\n", " latitude: 721, longitude: 1440, level: 3)\n", "Coordinates:\n", " * latitude (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n", " * level (level) int32 500 700 850\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 00:00:00...\n", " * time (time) datetime64[ns] 2018-01-01 ... 2022-12-31...\n", "Data variables: (12/15)\n", " 10m_u_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " ... ...\n", " total_precipitation (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " total_precipitation_24hr (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " total_precipitation_6hr (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/ifs_ens/2018-2022-1440x721_mean.zarr')" ] }, { "cell_type": "markdown", "id": "861126df-edc3-4899-bbd7-b808bd3e6717", "metadata": { "id": "861126df-edc3-4899-bbd7-b808bd3e6717" }, "source": [ "### ERA5 forecast\n", "\n", "As an apples-to-apples baseline for ML forecasts trained and evaluated with ERA5, we downloaded the ERA5 forecasts, a set of experimental forecasts run by ECMWF using the ERA5 IFS version, starting from ERA5 initial conditions. We downloaded data for 2018 and 2020.\n", "\n", "Location: [`gs://weatherbench2/datasets/era5-forecasts/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/era5-forecast/)\n", "\n", "Files:\n", "* `2018-1440x721.zarr/`\n", "* `2018-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2018-64x32_equiangular_conservative.zarr`\n", "* `2020-1440x721.zarr/`\n", "* `2020-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2020-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "ad1d841e-db53-47be-a194-21ad18125609", "metadata": { "id": "ad1d841e-db53-47be-a194-21ad18125609", "outputId": "8f8ce191-5611-4ac9-b699-55dab000209e" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                  (time: 732, prediction_timedelta: 31,\n",
       "                              latitude: 721, longitude: 1440, level: 3)\n",
       "Coordinates:\n",
       "  * latitude                 (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n",
       "  * level                    (level) int32 500 700 850\n",
       "  * longitude                (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * prediction_timedelta     (prediction_timedelta) timedelta64[ns] 00:00:00 ...\n",
       "  * time                     (time) datetime64[ns] 2020-01-01 ... 2020-12-31T...\n",
       "Data variables:\n",
       "    10m_u_component_of_wind  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed           (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature           (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential             (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    specific_humidity        (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    temperature              (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind      (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind      (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    vertical_velocity        (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed               (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 3, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 732, prediction_timedelta: 31,\n", " latitude: 721, longitude: 1440, level: 3)\n", "Coordinates:\n", " * latitude (latitude) float32 -90.0 -89.75 ... 89.75 90.0\n", " * level (level) int32 500 700 850\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 00:00:00 ...\n", " * time (time) datetime64[ns] 2020-01-01 ... 2020-12-31T...\n", "Data variables:\n", " 10m_u_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " specific_humidity (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " temperature (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " vertical_velocity (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/era5-forecasts/2020-1440x721.zarr')" ] }, { "cell_type": "markdown", "id": "b294d36c-66b5-4515-b226-27007539495c", "metadata": { "id": "b294d36c-66b5-4515-b226-27007539495c" }, "source": [ "### Keisler (2022)\n", "\n", "Ryan Keisler provided us with forecast using the Graph Neural Network from his [2022 paper](https://arxiv.org/abs/2202.07575).\n", "\n", "Location: [`gs://weatherbench2/datasets/keisler/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/keisler/)\n", "\n", "Files:\n", "* `2020-360x181.zarr`\n", "* `2020-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2020-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "7bc21904-66fa-4cbe-b5bf-9b38fe7c4ca0", "metadata": { "id": "7bc21904-66fa-4cbe-b5bf-9b38fe7c4ca0", "outputId": "1cb2f0cf-7286-40a6-a4a6-5868892b732c" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:               (level: 3, time: 732, prediction_timedelta: 41,\n",
       "                           latitude: 181, longitude: 360)\n",
       "Coordinates:\n",
       "  * latitude              (latitude) float64 90.0 89.0 88.0 ... -89.0 -90.0\n",
       "  * level                 (level) int64 500 700 850\n",
       "  * longitude             (longitude) float64 0.0 1.0 2.0 ... 357.0 358.0 359.0\n",
       "  * prediction_timedelta  (prediction_timedelta) timedelta64[ns] 00:00:00 ......\n",
       "  * time                  (time) datetime64[ns] 2020-01-01 ... 2020-12-31T12:...\n",
       "Data variables:\n",
       "    geopotential          (level, time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 41, 181, 360), meta=np.ndarray>\n",
       "    specific_humidity     (level, time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 41, 181, 360), meta=np.ndarray>\n",
       "    temperature           (level, time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 41, 181, 360), meta=np.ndarray>\n",
       "    u_component_of_wind   (level, time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 41, 181, 360), meta=np.ndarray>\n",
       "    v_component_of_wind   (level, time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 41, 181, 360), meta=np.ndarray>\n",
       "    wind_speed            (level, time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 41, 181, 360), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (level: 3, time: 732, prediction_timedelta: 41,\n", " latitude: 181, longitude: 360)\n", "Coordinates:\n", " * latitude (latitude) float64 90.0 89.0 88.0 ... -89.0 -90.0\n", " * level (level) int64 500 700 850\n", " * longitude (longitude) float64 0.0 1.0 2.0 ... 357.0 358.0 359.0\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 00:00:00 ......\n", " * time (time) datetime64[ns] 2020-01-01 ... 2020-12-31T12:...\n", "Data variables:\n", " geopotential (level, time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " specific_humidity (level, time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " temperature (level, time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " u_component_of_wind (level, time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " v_component_of_wind (level, time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " wind_speed (level, time, prediction_timedelta, latitude, longitude) float32 dask.array" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/keisler/2020-360x181.zarr')" ] }, { "cell_type": "markdown", "id": "df3113b5-908a-495f-951d-52c953278ade", "metadata": { "id": "df3113b5-908a-495f-951d-52c953278ade" }, "source": [ "\n", "### Pangu-Weather\n", "\n", "We ran the [Pangu](https://www.nature.com/articles/s41586-023-06185-3) model using the code available on [GitHub](https://github.com/198808xc/Pangu-Weather). \n", "\n", "Location: [`gs://weatherbench2/datasets/pangu/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/pangu/)\n", "\n", "Files:\n", "* `2018-2022_0012_0p25.zarr`\n", "* `2018-2022_0012_240x121_equiangular_with_poles_conservative.zarr`\n", "* `2018-2022_0012_64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "faf8ecea-64f5-4cb0-bd95-1f59a7a23914", "metadata": { "id": "faf8ecea-64f5-4cb0-bd95-1f59a7a23914", "outputId": "f7301ad7-9c6c-4244-c3ee-3082682dbd76" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                  (time: 3652, prediction_timedelta: 40,\n",
       "                              latitude: 721, longitude: 1440, level: 13)\n",
       "Coordinates:\n",
       "  * latitude                 (latitude) float32 90.0 89.75 89.5 ... -89.75 -90.0\n",
       "  * level                    (level) int64 1000 925 850 700 ... 200 150 100 50\n",
       "  * longitude                (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * prediction_timedelta     (prediction_timedelta) timedelta64[ns] 06:00:00 ...\n",
       "  * time                     (time) datetime64[ns] 2018-01-01 ... 2022-12-31T...\n",
       "Data variables:\n",
       "    10m_u_component_of_wind  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed           (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature           (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential             (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    specific_humidity        (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    temperature              (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind      (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind      (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed               (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 3652, prediction_timedelta: 40,\n", " latitude: 721, longitude: 1440, level: 13)\n", "Coordinates:\n", " * latitude (latitude) float32 90.0 89.75 89.5 ... -89.75 -90.0\n", " * level (level) int64 1000 925 850 700 ... 200 150 100 50\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 06:00:00 ...\n", " * time (time) datetime64[ns] 2018-01-01 ... 2022-12-31T...\n", "Data variables:\n", " 10m_u_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " specific_humidity (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " temperature (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/pangu/2018-2022_0012_0p25.zarr')" ] }, { "cell_type": "markdown", "id": "4a2bc296-ecc3-4688-908e-b47f91199498", "metadata": { "id": "4a2bc296-ecc3-4688-908e-b47f91199498" }, "source": [ "\n", "### Pangu-Weather (operational)\n", "\n", "We also ran Pangu in a quasi-operational setup with IFS HRES initial conditions.\n", "\n", "Location: [`gs://weatherbench2/datasets/pangu_hres_init/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/pangu_hres_init/)\n", "\n", "Files:\n", "* `2020_0012_0p25.zarr`\n", "* `2020_0012_240x121_equiangular_with_poles_conservative.zarr`\n", "* `2020_0012_64x32_equiangular_conservative.zarr`\n", "* `2021_0012_0p25.zarr`\n", "* `2021_0012_240x121_equiangular_with_poles_conservative.zarr`\n", "* `2021_0012_64x32_equiangular_conservative.zarr`\n", "* `2022_0012_0p25.zarr`\n", "* `2022_0012_240x121_equiangular_with_poles_conservative.zarr`\n", "* `2022_0012_64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "99fdd231-8bc4-4c27-9f61-9b311afd14b8", "metadata": { "id": "99fdd231-8bc4-4c27-9f61-9b311afd14b8", "outputId": "d253ae34-2c88-4ac1-8951-7a52fa0cb7bf" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                  (time: 732, prediction_timedelta: 40,\n",
       "                              latitude: 721, longitude: 1440, level: 13)\n",
       "Coordinates:\n",
       "  * latitude                 (latitude) float32 90.0 89.75 89.5 ... -89.75 -90.0\n",
       "  * level                    (level) int64 1000 925 850 700 ... 200 150 100 50\n",
       "  * longitude                (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n",
       "  * prediction_timedelta     (prediction_timedelta) timedelta64[ns] 06:00:00 ...\n",
       "  * time                     (time) datetime64[ns] 2020-01-01 ... 2020-12-31T...\n",
       "Data variables:\n",
       "    10m_u_component_of_wind  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed           (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature           (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential             (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    specific_humidity        (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    temperature              (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind      (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind      (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed               (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 732, prediction_timedelta: 40,\n", " latitude: 721, longitude: 1440, level: 13)\n", "Coordinates:\n", " * latitude (latitude) float32 90.0 89.75 89.5 ... -89.75 -90.0\n", " * level (level) int64 1000 925 850 700 ... 200 150 100 50\n", " * longitude (longitude) float32 0.0 0.25 0.5 ... 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 06:00:00 ...\n", " * time (time) datetime64[ns] 2020-01-01 ... 2020-12-31T...\n", "Data variables:\n", " 10m_u_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " specific_humidity (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " temperature (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/pangu_hres_init/2020_0012_0p25.zarr')" ] }, { "cell_type": "markdown", "id": "55cafb20-8b04-4c77-ab9c-f2390bc5c6b9", "metadata": { "id": "55cafb20-8b04-4c77-ab9c-f2390bc5c6b9" }, "source": [ "\n", "### GraphCast\n", "\n", "[GraphCast](https://www.science.org/doi/10.1126/science.adi2336) forecasts are available for 2018 and 2020. As described in the paper, the 2018 forecast were created with a model trained with data up to and including 2017, while the 2020 forecasts were created with a model trained with data up to and including 2019. These forecasts are initialized using ERA5.\n", "\n", "Location: [`gs://weatherbench2/datasets/graphcast/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/graphcast/)\n", "\n", "* Files (see directory above for exact file names for each year):\n", "* `date_range_YYYY-11-16_XXXX-02-01_12_hours_derived.zarr`\n", "* `date_range_YYYY-11-16_XXXX-02-01_12_hours-240x121_equiangular_with_poles_conservative.zarr`\n", "* `date_range_YYYY-11-16_XXXX-02-01_12_hours-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "23760485-bd29-45e2-bacc-51e2d934a55f", "metadata": { "id": "23760485-bd29-45e2-bacc-51e2d934a55f", "outputId": "6e9dbc9a-e53a-4c94-a5e9-4533e8c4b109" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                   (time: 886, prediction_timedelta: 40, lat: 721,\n",
       "                               lon: 1440, level: 37)\n",
       "Coordinates:\n",
       "  * lat                       (lat) float32 -90.0 -89.75 -89.5 ... 89.75 90.0\n",
       "  * level                     (level) int64 1 2 3 5 7 ... 900 925 950 975 1000\n",
       "  * lon                       (lon) float32 0.0 0.25 0.5 ... 359.2 359.5 359.8\n",
       "  * prediction_timedelta      (prediction_timedelta) timedelta64[ns] 06:00:00...\n",
       "  * time                      (time) datetime64[ns] 2019-11-16 ... 2021-01-31...\n",
       "Data variables: (12/14)\n",
       "    10m_u_component_of_wind   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed            (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature            (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential              (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 37, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    ...                        ...\n",
       "    total_precipitation_24hr  (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind       (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 37, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind       (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 37, 721, 1440), meta=np.ndarray>\n",
       "    vertical_velocity         (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 37, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 37, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 886, prediction_timedelta: 40, lat: 721,\n", " lon: 1440, level: 37)\n", "Coordinates:\n", " * lat (lat) float32 -90.0 -89.75 -89.5 ... 89.75 90.0\n", " * level (level) int64 1 2 3 5 7 ... 900 925 950 975 1000\n", " * lon (lon) float32 0.0 0.25 0.5 ... 359.2 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 06:00:00...\n", " * time (time) datetime64[ns] 2019-11-16 ... 2021-01-31...\n", "Data variables: (12/14)\n", " 10m_u_component_of_wind (time, prediction_timedelta, lat, lon) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, lat, lon) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, lat, lon) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, lat, lon) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, lat, lon) float32 dask.array\n", " ... ...\n", " total_precipitation_24hr (time, prediction_timedelta, lat, lon) float32 dask.array\n", " total_precipitation_6hr (time, prediction_timedelta, lat, lon) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " vertical_velocity (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, lat, lon) float32 dask.array" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/graphcast/2020/date_range_2019-11-16_2021-02-01_12_hours_derived.zarr')" ] }, { "cell_type": "markdown", "id": "3b021028-6191-47f9-9c94-704e15ddcc78", "metadata": { "id": "3b021028-6191-47f9-9c94-704e15ddcc78" }, "source": [ "\n", "### GraphCast (Operational)\n", "\n", "These GraphCast forecasts are initialized using operational IFS HRES analyses. The model details differ from the ERA5 version above. See `graphcast_operational` on the [GraphCast GitHub repository](https://github.com/google-deepmind/graphcast).\n", "\n", "Location: [`gs://weatherbench2/datasets/graphcast_hres_init/2020/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/graphcast_hres_init/2020/)\n", "\n", "* Files (see directory above for exact file names for each year):\n", "* `date_range_2019-11-16_2021-02-01_12_hours_derived.zarr`\n", "* `date_range_2019-11-16_2021-02-01_12_hours-240x121_equiangular_with_poles_conservative.zarr`\n", "* `date_range_2019-11-16_2021-02-01_12_hours-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "0e25325a-c907-4825-8351-237db6f7e4ee", "metadata": { "id": "0e25325a-c907-4825-8351-237db6f7e4ee", "outputId": "e70acbc1-ef5b-42d3-ede3-7a0e5672a1dc" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                   (time: 732, prediction_timedelta: 40, lat: 721,\n",
       "                               lon: 1440, level: 13)\n",
       "Coordinates:\n",
       "  * lat                       (lat) float32 -90.0 -89.75 -89.5 ... 89.75 90.0\n",
       "  * level                     (level) int32 50 100 150 200 ... 700 850 925 1000\n",
       "  * lon                       (lon) float32 0.0 0.25 0.5 ... 359.2 359.5 359.8\n",
       "  * prediction_timedelta      (prediction_timedelta) timedelta64[ns] 06:00:00...\n",
       "  * time                      (time) datetime64[ns] 2020-01-01 ... 2020-12-31...\n",
       "Data variables: (12/14)\n",
       "    10m_u_component_of_wind   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed            (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature            (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    geopotential              (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    ...                        ...\n",
       "    total_precipitation_24hr  (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr   (time, prediction_timedelta, lat, lon) float32 dask.array<chunksize=(1, 1, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind       (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind       (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    vertical_velocity         (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                (time, prediction_timedelta, level, lat, lon) float32 dask.array<chunksize=(1, 1, 13, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 732, prediction_timedelta: 40, lat: 721,\n", " lon: 1440, level: 13)\n", "Coordinates:\n", " * lat (lat) float32 -90.0 -89.75 -89.5 ... 89.75 90.0\n", " * level (level) int32 50 100 150 200 ... 700 850 925 1000\n", " * lon (lon) float32 0.0 0.25 0.5 ... 359.2 359.5 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 06:00:00...\n", " * time (time) datetime64[ns] 2020-01-01 ... 2020-12-31...\n", "Data variables: (12/14)\n", " 10m_u_component_of_wind (time, prediction_timedelta, lat, lon) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, lat, lon) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, lat, lon) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, lat, lon) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, lat, lon) float32 dask.array\n", " ... ...\n", " total_precipitation_24hr (time, prediction_timedelta, lat, lon) float32 dask.array\n", " total_precipitation_6hr (time, prediction_timedelta, lat, lon) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " vertical_velocity (time, prediction_timedelta, level, lat, lon) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, lat, lon) float32 dask.array" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/graphcast_hres_init/2020/date_range_2019-11-16_2021-02-01_12_hours_derived.zarr')" ] }, { "cell_type": "markdown", "id": "4162d0ed-bfcf-43cb-95b7-ffeae4b91897", "metadata": { "id": "4162d0ed-bfcf-43cb-95b7-ffeae4b91897" }, "source": [ "\n", "### Spherical CNN\n", "\n", "Forecasts using a [Spherical CNN](https://arxiv.org/abs/2306.05420) are available for 2020.\n", "\n", "Location: [`gs://weatherbench2/datasets/sphericalcnn/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/sphericalcnn/)\n", "\n", "Files:\n", "* `2020-240x121_equiangular_with_poles.zarr`\n", "* `2020-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "0e818d53-a344-4a89-bb61-fa3bad10a542", "metadata": { "id": "0e818d53-a344-4a89-bb61-fa3bad10a542", "outputId": "535c1608-8b63-447b-9be3-a57c7dad82f7" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:               (time: 178, prediction_timedelta: 40, level: 13,\n",
       "                           longitude: 240, latitude: 121)\n",
       "Coordinates:\n",
       "  * latitude              (latitude) float64 -90.0 -88.5 -87.0 ... 88.5 90.0\n",
       "  * level                 (level) int64 50 100 150 200 250 ... 700 850 925 1000\n",
       "  * longitude             (longitude) float64 0.0 1.5 3.0 ... 355.5 357.0 358.5\n",
       "  * prediction_timedelta  (prediction_timedelta) timedelta64[ns] 0 days 06:00...\n",
       "  * time                  (time) datetime64[ns] 2020-01-01 ... 2020-12-20\n",
       "Data variables:\n",
       "    geopotential          (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(10, 40, 13, 240, 121), meta=np.ndarray>\n",
       "    specific_humidity     (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(10, 40, 13, 240, 121), meta=np.ndarray>\n",
       "    temperature           (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(10, 40, 13, 240, 121), meta=np.ndarray>\n",
       "    u_component_of_wind   (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(10, 40, 13, 240, 121), meta=np.ndarray>\n",
       "    v_component_of_wind   (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(10, 40, 13, 240, 121), meta=np.ndarray>\n",
       "    wind_speed            (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(10, 40, 13, 240, 121), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 178, prediction_timedelta: 40, level: 13,\n", " longitude: 240, latitude: 121)\n", "Coordinates:\n", " * latitude (latitude) float64 -90.0 -88.5 -87.0 ... 88.5 90.0\n", " * level (level) int64 50 100 150 200 250 ... 700 850 925 1000\n", " * longitude (longitude) float64 0.0 1.5 3.0 ... 355.5 357.0 358.5\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] 0 days 06:00...\n", " * time (time) datetime64[ns] 2020-01-01 ... 2020-12-20\n", "Data variables:\n", " geopotential (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_humidity (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " temperature (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, longitude, latitude) float32 dask.array" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/sphericalcnn/2020-240x121_equiangular_with_poles.zarr')" ] }, { "cell_type": "markdown", "id": "d70d2ff3-e38a-4f2b-af3d-c87022e6471a", "metadata": { "id": "d70d2ff3-e38a-4f2b-af3d-c87022e6471a" }, "source": [ "\n", "### FuXi\n", "\n", "Forecasts using [FuXi](https://www.nature.com/articles/s41612-023-00512-1) model are available for 2020.\n", "\n", "Location: [`gs://weatherbench2/datasets/fuxi/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/fuxi/)\n", "\n", "Files:\n", "* `2020-1440x721.zarr`\n", "* `2020-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2020-64x32_equiangular_conservative.zarr`\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cb7a2929-dafd-4c80-8a64-af0faa42f967", "metadata": { "id": "cb7a2929-dafd-4c80-8a64-af0faa42f967", "outputId": "ac9fdf1e-404f-4a8d-b8a1-fff45884b926" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                            (time: 702, prediction_timedelta: 60,\n",
       "                                        latitude: 721, longitude: 1440, level: 2)\n",
       "Coordinates:\n",
       "  * latitude                           (latitude) float64 -90.0 -89.75 ... 90.0\n",
       "  * level                              (level) int32 500 850\n",
       "  * longitude                          (longitude) float64 0.0 0.25 ... 359.8\n",
       "  * prediction_timedelta               (prediction_timedelta) timedelta64[ns] ...\n",
       "  * time                               (time) datetime64[ns] 2020-01-01 ... 2...\n",
       "Data variables:\n",
       "    10m_u_component_of_wind            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    10m_v_component_of_wind            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    10m_wind_speed                     (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    2m_temperature                     (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    geopotential                       (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 60, 2, 721, 1440), meta=np.ndarray>\n",
       "    mean_sea_level_pressure            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    temperature                        (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 60, 2, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_24hr_from_6hr  (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    total_precipitation_6hr            (time, prediction_timedelta, latitude, longitude) float32 dask.array<chunksize=(1, 60, 721, 1440), meta=np.ndarray>\n",
       "    u_component_of_wind                (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 60, 2, 721, 1440), meta=np.ndarray>\n",
       "    v_component_of_wind                (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 60, 2, 721, 1440), meta=np.ndarray>\n",
       "    wind_speed                         (time, prediction_timedelta, level, latitude, longitude) float32 dask.array<chunksize=(1, 60, 2, 721, 1440), meta=np.ndarray>
" ], "text/plain": [ "\n", "Dimensions: (time: 702, prediction_timedelta: 60,\n", " latitude: 721, longitude: 1440, level: 2)\n", "Coordinates:\n", " * latitude (latitude) float64 -90.0 -89.75 ... 90.0\n", " * level (level) int32 500 850\n", " * longitude (longitude) float64 0.0 0.25 ... 359.8\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] ...\n", " * time (time) datetime64[ns] 2020-01-01 ... 2...\n", "Data variables:\n", " 10m_u_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_v_component_of_wind (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 10m_wind_speed (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " 2m_temperature (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " mean_sea_level_pressure (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " temperature (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " total_precipitation_24hr_from_6hr (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " total_precipitation_6hr (time, prediction_timedelta, latitude, longitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, latitude, longitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, latitude, longitude) float32 dask.array" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/fuxi/2020-1440x721.zarr')" ] }, { "cell_type": "markdown", "id": "50d8b2dd-a853-43b9-a760-ca38a954c6aa", "metadata": { "id": "50d8b2dd-a853-43b9-a760-ca38a954c6aa" }, "source": [ "\n", "### NeuralGCM\n", "\n", "Forecasts made with the [Neural General Circulation Model](https://arxiv.org/abs/2311.07222) are available for 2020. The deterministic model has a raw resolution of 0.7 degrees\n", "\n", "Location: [`gs://weatherbench2/datasets/neuralgcm_deterministic/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/neuralgcm_deterministic/)\n", "\n", "Files:\n", "* `2020-512x256.zarr`\n", "* `2020-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2020-64x32_equiangular_conservative.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "549f55a7-2987-44a8-a58f-7661e5ef2ac2", "metadata": { "id": "549f55a7-2987-44a8-a58f-7661e5ef2ac2", "outputId": "edf29d57-e86d-461a-d8fd-0b78356a9c3f" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                              (time: 732, prediction_timedelta: 31,\n",
       "                                          longitude: 240, latitude: 121,\n",
       "                                          level: 37)\n",
       "Coordinates:\n",
       "  * latitude                             (latitude) float64 -90.0 -88.5 ... 90.0\n",
       "  * level                                (level) int64 1 2 3 5 ... 950 975 1000\n",
       "  * longitude                            (longitude) float64 0.0 1.5 ... 358.5\n",
       "  * prediction_timedelta                 (prediction_timedelta) timedelta64[ns] ...\n",
       "  * time                                 (time) datetime64[ns] 2020-01-01 ......\n",
       "Data variables:\n",
       "    P_minus_E_cumulative                 (time, prediction_timedelta, longitude, latitude) float32 dask.array<chunksize=(1, 8, 240, 121), meta=np.ndarray>\n",
       "    geopotential                         (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    specific_cloud_ice_water_content     (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    specific_cloud_liquid_water_content  (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    specific_humidity                    (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    temperature                          (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    u_component_of_wind                  (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    v_component_of_wind                  (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "    wind_speed                           (time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(1, 8, 37, 240, 121), meta=np.ndarray>\n",
       "Attributes:\n",
       "    experiment_id:  67001173\n",
       "    worker_id:      1
" ], "text/plain": [ "\n", "Dimensions: (time: 732, prediction_timedelta: 31,\n", " longitude: 240, latitude: 121,\n", " level: 37)\n", "Coordinates:\n", " * latitude (latitude) float64 -90.0 -88.5 ... 90.0\n", " * level (level) int64 1 2 3 5 ... 950 975 1000\n", " * longitude (longitude) float64 0.0 1.5 ... 358.5\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] ...\n", " * time (time) datetime64[ns] 2020-01-01 ......\n", "Data variables:\n", " P_minus_E_cumulative (time, prediction_timedelta, longitude, latitude) float32 dask.array\n", " geopotential (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_cloud_ice_water_content (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_cloud_liquid_water_content (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_humidity (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " temperature (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " u_component_of_wind (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " v_component_of_wind (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " wind_speed (time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", "Attributes:\n", " experiment_id: 67001173\n", " worker_id: 1" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/neuralgcm_deterministic/2020-240x121_equiangular_with_poles_conservative.zarr')" ] }, { "cell_type": "markdown", "id": "5c38645e-9999-4b81-85e0-2007d97ea448", "metadata": { "id": "5c38645e-9999-4b81-85e0-2007d97ea448" }, "source": [ "The ensemble version has a resolution of 1.4 degrees and has been run to produce 50 members. We also computed ensemble means\n", "\n", "Location: [`gs://weatherbench2/datasets/neuralgcm_ens/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/neuralgcm_ens/)\n", "\n", "Files:\n", "* `2020-256x128.zarr`\n", "* `2020-240x121_equiangular_with_poles_conservative.zarr`\n", "* `2020-240x121_equiangular_with_poles_conservative_mean.zarr`\n", "* `2020-64x32_equiangular_conservative.zarr`\n", "* `2020-64x32_equiangular_conservative_mean.zarr`" ] }, { "cell_type": "code", "execution_count": null, "id": "6e7c3ce1-c4ab-4274-9958-fbdd765c7cd6", "metadata": { "id": "6e7c3ce1-c4ab-4274-9958-fbdd765c7cd6", "outputId": "0d4354b4-4f9c-4bba-f957-1d75dc30b97f" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                              (realization: 50, time: 732,\n",
       "                                          prediction_timedelta: 32, level: 37,\n",
       "                                          longitude: 240, latitude: 121)\n",
       "Coordinates:\n",
       "  * latitude                             (latitude) float64 -90.0 -88.5 ... 90.0\n",
       "  * level                                (level) int64 1 2 3 5 ... 950 975 1000\n",
       "  * longitude                            (longitude) float64 0.0 1.5 ... 358.5\n",
       "  * prediction_timedelta                 (prediction_timedelta) timedelta64[ns] ...\n",
       "  * realization                          (realization) int64 0 1 2 ... 47 48 49\n",
       "  * time                                 (time) datetime64[ns] 2020-01-01 ......\n",
       "Data variables:\n",
       "    geopotential                         (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    specific_cloud_ice_water_content     (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    specific_cloud_liquid_water_content  (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    specific_humidity                    (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    temperature                          (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    u_component_of_wind                  (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    v_component_of_wind                  (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "    wind_speed                           (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array<chunksize=(50, 1, 2, 1, 240, 121), meta=np.ndarray>\n",
       "Attributes:\n",
       "    experiment_id:  73974210\n",
       "    worker_id:      3
" ], "text/plain": [ "\n", "Dimensions: (realization: 50, time: 732,\n", " prediction_timedelta: 32, level: 37,\n", " longitude: 240, latitude: 121)\n", "Coordinates:\n", " * latitude (latitude) float64 -90.0 -88.5 ... 90.0\n", " * level (level) int64 1 2 3 5 ... 950 975 1000\n", " * longitude (longitude) float64 0.0 1.5 ... 358.5\n", " * prediction_timedelta (prediction_timedelta) timedelta64[ns] ...\n", " * realization (realization) int64 0 1 2 ... 47 48 49\n", " * time (time) datetime64[ns] 2020-01-01 ......\n", "Data variables:\n", " geopotential (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_cloud_ice_water_content (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_cloud_liquid_water_content (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " specific_humidity (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " temperature (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " u_component_of_wind (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " v_component_of_wind (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", " wind_speed (realization, time, prediction_timedelta, level, longitude, latitude) float32 dask.array\n", "Attributes:\n", " experiment_id: 73974210\n", " worker_id: 3" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xr.open_zarr('gs://weatherbench2/datasets/neuralgcm_ens/2020-240x121_equiangular_with_poles_conservative.zarr')" ] }, { "cell_type": "markdown", "id": "be3c961b-f04d-4cc7-b9ab-398ad50394ec", "metadata": { "id": "f2d902f1-be42-4869-bff1-75fd34d67b05" }, "source": [ "## Observation data\n", "\n", "### MADIS METAR\n", "\n", "[MADIS](https://madis.ncep.noaa.gov/index.shtml) is a NOAA operated database that provides access to several meteorological datasets. [METAR](https://madis.ncep.noaa.gov/madis_metar.shtml) is a dataset of weather station observations. The raw data comes in NetCDF format and has been downloaded from the [MADIS ftp server](https://madis.ncep.noaa.gov/mesonet_providers.shtml). The data has been converted to [Parquet](https://parquet.apache.org/docs/) format in different chunking formats.\n", "\n", "Location: [`gs://weatherbench2/datasets/metar/`](https://console.cloud.google.com/storage/browser/weatherbench2/datasets/metar/)\n", "\n", "Files:\n", "- metar-timeNominal-by-month\n", "- metar-timeNominal-by-day\n", "- metar-timeNominal-by-hour\n", "- metar-timeObs-by-hour_2001-07-01T00_2024-01-01T00" ] }, { "cell_type": "code", "execution_count": 2, "id": "284788dc-05d3-4ddb-a016-0aa1de823eb2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stationNamelocationNamelatitudelongitudeelevationtimeObstimeNominaltimeReceivedreportTypeautoStationType...precip3Hourprecip3HourDDprecip6Hourprecip6HourDDprecip24Hourprecip24HourDDpressChange3HourpressChange3HourDDsnowCoversnowCoverDD
recNum
0NZCMMCMURDO/WILLIAMS_FD-77.870003166.9700018.02019-12-31 23:55:002020-01-012019-12-31 23:58:00METAR...NaNZNaNZNaNZNaNZNaNZ
1YMER-36.900002149.8999942.02019-12-31 23:54:002020-01-012019-12-31 23:56:00SPECIAUTO...NaNZNaNZNaNZNaNZNaNZ
2SUMUCARRASCO_(CIV/MIL)-34.830002-56.00000032.02019-12-31 23:45:002020-01-012019-12-31 23:48:00SPECI...NaNZNaNZNaNZNaNZNaNZ
3SCELSANTIAGO_PUDAHUEL-33.380001-70.779999476.02020-01-01 00:00:002020-01-012019-12-31 23:56:00METAR...NaNZNaNZNaNZNaNZNaNZ
4SCVM-32.950001-71.480003141.02020-01-01 00:00:002020-01-012019-12-31 23:58:00METAR...NaNZNaNZNaNZNaNZNaNZ
\n", "

5 rows × 43 columns

\n", "
" ], "text/plain": [ " stationName locationName latitude longitude elevation \\\n", "recNum \n", "0 NZCM MCMURDO/WILLIAMS_FD -77.870003 166.970001 8.0 \n", "1 YMER -36.900002 149.899994 2.0 \n", "2 SUMU CARRASCO_(CIV/MIL) -34.830002 -56.000000 32.0 \n", "3 SCEL SANTIAGO_PUDAHUEL -33.380001 -70.779999 476.0 \n", "4 SCVM -32.950001 -71.480003 141.0 \n", "\n", " timeObs timeNominal timeReceived reportType \\\n", "recNum \n", "0 2019-12-31 23:55:00 2020-01-01 2019-12-31 23:58:00 METAR \n", "1 2019-12-31 23:54:00 2020-01-01 2019-12-31 23:56:00 SPECI \n", "2 2019-12-31 23:45:00 2020-01-01 2019-12-31 23:48:00 SPECI \n", "3 2020-01-01 00:00:00 2020-01-01 2019-12-31 23:56:00 METAR \n", "4 2020-01-01 00:00:00 2020-01-01 2019-12-31 23:58:00 METAR \n", "\n", " autoStationType ... precip3Hour precip3HourDD precip6Hour \\\n", "recNum ... \n", "0 ... NaN Z NaN \n", "1 AUTO ... NaN Z NaN \n", "2 ... NaN Z NaN \n", "3 ... NaN Z NaN \n", "4 ... NaN Z NaN \n", "\n", " precip6HourDD precip24Hour precip24HourDD pressChange3Hour \\\n", "recNum \n", "0 Z NaN Z NaN \n", "1 Z NaN Z NaN \n", "2 Z NaN Z NaN \n", "3 Z NaN Z NaN \n", "4 Z NaN Z NaN \n", "\n", " pressChange3HourDD snowCover snowCoverDD \n", "recNum \n", "0 Z NaN Z \n", "1 Z NaN Z \n", "2 Z NaN Z \n", "3 Z NaN Z \n", "4 Z NaN Z \n", "\n", "[5 rows x 43 columns]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "# Opening a single partition\n", "pd.read_parquet('gs://weatherbench2/datasets/metar/metar-timeNominal-by-month/year=2020/month=1/2020-01.parquet').head()" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 5 }