How to forecast

1 Overview: How to submit a forecast

We provide an overview of the steps for submitting with the details below:

  1. Register at https://forms.gle/kg2Vkpho9BoMXSy57. You are not required to register if your forecast submission uses the word “example” in your model_id (i.e myforecast_example). Any forecasts with “example” in the model_id will not be used in forecast evaluation analyses.
  2. Generate a forecast!
  3. Write the forecast output to a file that follows our standardized format (described below).
  4. Submit your forecast using an R function (provided below).
  5. Watch your forecast be evaluated as new data are collected.

2 Generating a forecast

2.1 All forecasting approaches are welcome

We encourage you to use any modeling approach to make a prediction about the future conditions at the two LTREB reservoirs or the stream that connects them.

2.2 Must include uncertainty

Forecasts require you to make an assessment of the confidence in your prediction of the future. You can represent your confidence (i.e., uncertainty in the forecast) using a distribution or numerically using an ensemble (or sample) of predictions.

For new forecasters, as a first step, two possible approaches for quantifying forecast uncertainty are: 1) using the residuals from the model fitting/calibration period to generate a model process uncertainty distribution that can be applied to the forecast; or 2) for forecasts that use meteorological variables as drivers, using weather forecasts provided by the VERA challenge organizers to account for driver uncertainty in a forecast.

2.3 Any model drivers/covariates/features are welcome

You can use any data as model inputs (including all of the forecast target data available to date). All sensor-based target data are available in near-real time (1-day latency). You may want to use the updated target data to re-train a model or for data assimilation.

As VERA is a genuine forecasting challenge, you will need forecasted drivers if your model uses drivers as inputs. If you are interested in using forecasted meteorology for the VERA sites, we provide to two approaches:

  • We provide weather forecasts from the NOAA Ensemble Forecasting System using custom functions in the vera4castHelpers R package. You can learn more about accessing the weather forecasts using this method here. This approach provides access to weather forecasts back to September 2020.

  • We have developed an R package (https://github.com/FLARE-forecast/RopenMeteo) that can be used to access ensemble weather forecasts or seasonal forecasts for the through the open-meteo.com API. The README in the R package repo provides instructions for use. This approach only provides the current forecast.

2.4 Forecasts can be for a range of horizons

Forecasts can be submitted at the daily time step for 1 day to 10 years ahead into the future.

2.5 Forecasts can be submitted every day

Since forecasts can be submitted every day, automation is important. We provide an example GitHub repository that can be used to automate your forecast with GitHub Actions. It also includes the use of a custom Docker Container rqthomas/vera4cast:latest that has many of the R packages and functions needed to generate and submit forecasts.

3 Online resources

We have prepared a video playlist to guide you through the submission process

4 Forecast file format

The forecast file should be in a csv format with the following columns:

  • project_id: use vera4cast

  • model_id: the short name of the model defined as the model_id in the file name (see below) and in your registration. The model_id should have no spaces. model_id should reflect a method to forecast one or a set of target variables and must be unique to the vera4cast challenge.

  • datetime: forecast timestamp. Format %Y-%m-%d %H:%M:%S with UTC as the time zone. Forecasts submitted with a %Y-%m-%d format will be converted to a full datetime assuming UTC midnight.

  • reference_datetime: The start of the forecast; this should be 0 time steps into the future. There should only be one value of reference_datetime in the file. Format is %Y-%m-%d %H:%M:%S with UTC as the time zone. Forecasts submitted with a %Y-%m-%d format will be converted to a full datetime assuming UTC mid-night.

  • duration: the time-step of the forecast. Use the value of P1D for a daily forecast and PT1H for an hourly forecast. This value should match the duration of the target variable that you are forecasting. Formatted as ISO 8601 duration

  • site_id: code for site (bvre, fcre, or tubr) bvre = Beaverdam Reservoir, fcre = Falling Creek Reservoir, and tubr = Tunnel Branch, the stream that connects the two reservoirs

  • depth_m: the depth (meters) for the forecasted variable. Use NA for variables without a specified depth (i.e Secchi).

  • family name of the probability distribution that is described by the parameter values in the parameter column (see list below for possible distributions). An ensemble forecast has a family of ensemble (see note below about family possibilities).

  • parameter the parameters for the distribution (see note below about the parameter column) or the number of the ensemble members. For example, the parameters for a normal distribution are called mu and sigma.

  • variable: standardized variable name (see “What to forecast” tab for more information)

  • prediction: forecasted value for the parameter in the parameter column

5 Representing uncertainty

Uncertainty is represented through the family and parameter columns.

5.0.1 Parameteric forecast

Parametric forecasts describe uncertainty by providing the values of parameters for an uncertainty distribution representing the forecast. The name of the distribution is provided in the ‘family’ column and the names of the parameters for the distribution are provided in the ‘parameter’ column. The values of the parameters are provided in the ‘prediction’ column. For a parametric forecast with a normal distribution, the family column would have the word normal to designate a normal distribution and the parameter column must have values of mu and sigma for each forecasted variable, site_id, depth, and time combination.

Parametric forecasts for binary variables should use bernoulli as the distribution.

The following names and parameterization of the distribution are currently supported by VERA (family: parameters):

  • lognormal: mu, sigma
  • normal: mu,sigma
  • bernoulli: prob
  • beta: shape1, shape2
  • uniform: min, max
  • gamma: shape, rate
  • logistic: location, scale
  • exponential: rate
  • poisson: lambda

If you are submitting a forecast that is not in the supported list, we recommend using the ensemble format and sampling from your distribution to generate a set of ensemble members that represents your forecast distribution.

5.0.2 Ensemble (or sample) forecast

Ensemble, or sample, forecasts describe uncertainty by providing many predictions from multiple ensemble members, which altogether represent the uncertainty in the forecast. When submitting an ensemble forecast, the value of the ‘family’ column will be ‘ensemble’ and the ‘parameter’ column is used to index the ensemble members (e.g., 1-100 for a 100-member ensemble).

When forecasts using the ensemble family are scored, we assume that the ensemble is a set of equally likely realizations that are sampled from a distribution that is comparable to that of the observations (i.e., no broad adjustments are required to make the ensemble more consistent with observations). This is referred to as a “perfect ensemble” by Bröcker and Smith (2007). Ensemble (or sample) forecasts are transformed to a probability distribution function (e.g., dressed) using the default methods in the scoringRules R package (empirical version of the quantile decomposition for the crps and kernel density estimation using a Gaussian kernel for the logs). Kernel density estimation uses the default bandwidth produced by the bw.nrd function in the R stats package.

5.1 Example forecasts

Here is an example of a forecast that has a normal distribution:

df <- readr::read_csv("https://renc.osn.xsede.org/bio230121-bucket01/vera4cast/forecasts/raw/T20231001231345_daily-2023-10-01-climatology.csv.gz", show_col_types = FALSE)
dplyr::glimpse(df)
Rows: 288
Columns: 11
$ model_id           <chr> "climatology", "climatology", "climatology", "clima…
$ datetime           <date> 2023-10-02, 2023-10-02, 2023-10-02, 2023-10-02, 20…
$ reference_datetime <date> 2023-10-01, 2023-10-01, 2023-10-01, 2023-10-01, 20…
$ site_id            <chr> "bvre", "bvre", "bvre", "bvre", "bvre", "bvre", "bv…
$ variable           <chr> "Chla_ugL_mean", "Chla_ugL_mean", "Temp_C_mean", "T…
$ family             <chr> "normal", "normal", "normal", "normal", "normal", "…
$ parameter          <chr> "mu", "sigma", "mu", "sigma", "mu", "sigma", "mu", …
$ prediction         <dbl> 10.041987, 2.587292, 18.331126, 2.531732, 9.541139,…
$ depth_m            <dbl> 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1…
$ project_id         <chr> "vera4cast", "vera4cast", "vera4cast", "vera4cast",…
$ duration           <chr> "P1D", "P1D", "P1D", "P1D", "P1D", "P1D", "P1D", "P…

For an ensemble (or sample) forecast, the family column uses the word ensemble to designate that it is an ensemble forecast and the parameter column is the ensemble member number (1, 2, 3 …)

df <- readr::read_csv("https://renc.osn.xsede.org/bio230121-bucket01/vera4cast/forecasts/raw/T20231001231348_daily-2023-10-01-persistenceRW.csv.gz", show_col_types = FALSE)
dplyr::glimpse(df)
Rows: 28,800
Columns: 11
$ model_id           <chr> "persistenceRW", "persistenceRW", "persistenceRW", …
$ datetime           <dttm> 2023-10-02, 2023-10-03, 2023-10-04, 2023-10-05, 20…
$ reference_datetime <dttm> 2023-10-01, 2023-10-01, 2023-10-01, 2023-10-01, 20…
$ site_id            <chr> "bvre", "bvre", "bvre", "bvre", "bvre", "bvre", "bv…
$ family             <chr> "ensemble", "ensemble", "ensemble", "ensemble", "en…
$ parameter          <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
$ variable           <chr> "Chla_ugL_mean", "Chla_ugL_mean", "Chla_ugL_mean", …
$ prediction         <dbl> 10.59539, 13.04463, 13.86989, 13.81281, 14.09787, 1…
$ depth_m            <dbl> 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1…
$ project_id         <chr> "vera4cast", "vera4cast", "vera4cast", "vera4cast",…
$ duration           <chr> "P1D", "P1D", "P1D", "P1D", "P1D", "P1D", "P1D", "P…

6 Submission process

Individual forecast files can be uploaded any time.

Teams will submit their forecast csv output files through an R function.

The function is available using the following code:

remotes::install_github("LTREB-reservoirs/vera4castHelpers")

The submit function is:

library(vera4castHelpers)
submit(forecast_file = "your_file.csv")

If you will be submitting multiple forecasts using the same model_id, use the following code:

vera4castHelpers::submit(forecast_file = "your_file.csv",
                         first_submission = FALSE)

7 Post-submission

7.1 Processing

After submission, our servers will process uploaded files by converting them to a parquet format on our public s3 storage. A pub_datetime column will be added that denotes when a forecast was submitted.

7.2 Evaluation

All forecasts are scored daily using new data until the full horizon of the forecast has been scored. Forecasts are scored using the crps and logs functions in the scoringRules R package. These scores evaluate both the accuracy and precision of the forecast distribution (i.e., how close was the forecast mean to the observation and how certain was the forecast).

7.3 Comparison

Forecast performance can be compared to the performance of baseline models. We are automatically submitting the following baseline models:

  • climatology: the normal distribution (mean and standard deviation) of that day-of-year in the historical observations
  • persistenceRW: a random walk model that assumes no change in the mean behavior. The random walk is initialized using the most resent observation.

Our forecast performance page includes evaluations of all submitted models.

7.4 Catalog

Information for accessing the forecasts and scores can be found on our forecast catalog page.

8 Questions?

Thanks for reading this document! If you still have questions about how to submit your forecast to the VERA Forecasting Challenge, we encourage you to email Dr. Quinn Thomas (rqthomas{at}vt.edu).