The Model Diagnostics Task Force is constituted to develop, coordinate, and implement process-based model evaluation metrics and a metrics framework in National modeling center metrics packages, leveraging ongoing efforts at the modeling centers toward advancing model evaluation and development capabilities.

This Task Force is constituted of researchers whose projects were successfully evaluated as part of the FY15 competition held by NOAA’s Modeling, Analysis, Predictions, and Projections program on the topic of Process Oriented Metrics. This competition was motivated by a number of factors, including community interest in moving beyond performanceoriented metrics toward processoriented metrics, ongoing efforts to develop next-generation climate and Earth system models, evolving plans for CMIP, and a need to link model development and evaluation efforts across modeling centers. Nine projects were selected as a result of the competitive process, including eight focused on developing metrics and one that will develop metrics as well as define a direction and framework for the overall effort.

Task Force Function and Implementation

This Task Force will initiate its activities in October 2015 and will have a duration of three years. It is expected that researchers who were selected through the NOAA competitive process will participate actively in this Task Force helping to build an integrative processoriented metrics framework serving NOAA and other modeling centers. Task Force membership may also include affiliates, elected by Task Force leadership, who are not supported by MAPP funding but whose expertise and work are relevant to achieving the goals of this task force. The Task Force will coordinate with other relevant ongoing diagnostic efforts including emerging community processoriented metrics efforts such as the WCRP/WGCM Metrics Panel, PCMDI’s UVCDAT effort, and the EMBRACE ESMValTool project.

The Task Force will connect all of the funded investigators and other invitees. Beyond enhancing communication between investigators, expectations of this group include contribution by the individual projects to the development and implementation of the collective effort with a focus on implementing a cohesive metrics framework, open documentation of the efforts, and a focus on maximizing the community utility of metrics and the metrics framework. It is expected the main group will have monthly teleconferences and consider inperson meetings, as appropriate.


As of Spring 2017, the Task Force has developed a functional Application Program Interface (API) for the metrics package, which is documented here. The linked document also describes the overall status of the effort as of April 2017. The API is python based and designed to be flexible such that metrics coded in different non-proprietary software packages can be integrated into the API.

MAPP Task Force Concept and Terms of Reference


Lead: Eric Maloney, Colorado State University
Co-Lead: Yi Ming, NOAA Geophysical Fluid Dynamics Laboratory
Co-Lead: Andrew Gettelman, National Center for Atmospheric Research
Co-Lead: Aiguo Dai, University at Albany


Relevant MAPP Program PIs and selected additional invitees.
To view the full Participants list, please visit the Participants page.


To view the full list of Projects, please click here.

News & Events

NOAA and partners release database for research to bridge weather to climate forecast gap

  • 17 October 2017
  • Number of views: 1028

Wouldn’t it be nice to know now what the weather is going to be like for the vacation you have planned next month? Or, if you’re a farmer, whether you’re going to get enough rainfall during a crucial planting time coming up in a few weeks? Weather forecasts help us make decisions about the next few days to a week, and seasonal climate forecasts give us information on the time scale of three months to a year or more. But a significant gap in scientists’ understanding has limited the ability to forecast what will happen two weeks to two months from now, also called the subseasonal scale. Scientists are starting to produce experimental subseasonal climate forecasts, and they are exploring predictions, for example, of conditions that led to the recent hurricanes to strike the United States. Despite growing demand, public access to those forecasts in support of research and experimental applications has been limited.

Two new datasets, funded in part by NOAA Research’s Modeling, Analysis, Predictions, and Projections (MAPP) Program, now provide easy access to 60 terabytes of climate forecasts containing predictions of rainfall, temperature, winds and other variables at the subseasonal level. Available online through the Columbia University International Research Institute for Climate and Society (IRI) Data Library, the data will allow scientists to conduct extensive research to improve subseasonal forecasts and advance understanding of this “predictability desert”. 

One of the datasets makes subseasonal forecasts available from 11 out of 13 of the World Meteorological Organization's (WMO) official long-range forecasting centers. Known as the S2S Database, this dataset underpins the international Subseasonal to Seasonal (S2S) Prediction Project, said Andrew Robertson, who heads IRI's climate team and is the co-chair of the S2S Project. The WMO’s World Weather and World Climate Research Programs created the project in 2013 to improve forecasts and understanding on timescales of two weeks to a season, filling the gap between daily weather and seasonal climate forecasts. WMO also wants to promote use of the forecasts for better early warning of high impact weather events such as floods and droughts, and heat and cold waves.

The data is also foundational to the NOAA MAPP Program’s S2S Prediction Task Force, whose members consist of MAPP-funded scientists from universities, research laboratories, and NOAA labs and centers. The Task Force collaborates and coordinates with the international S2S Prediction Project to expedite advances in NOAA’s and the Nation’s ability to bridge the forecast gap.

While the S2S Database is also available at two official S2S archiving centers (ECMWF and CMA), the IRI Data Library version brings a large subset of the S2S database online so that it is accessible “in the cloud” using any web browser, even on a tablet or smart phone. Researchers involved in both the international and national efforts can also more easily assess S2S forecast accuracy using the Data Library’s extensive datasets of observed climate conditions. Additionally, the IRI Data Library allows users to visualize the S2S data before downloading it. For some, said Robertson, viewing the data – such as weekly averages of forecasted daily rainfall during a past flood event – may be all that's needed. Others are likely to find it useful for exploratory analysis before downloading.

“The functionality that the Data Library adds will be particularly relevant for researchers who are working on how subseasonal forecasting data might be used in applications for public health, water resources, agriculture, disaster risk reduction and more,” said Robertson.

Because of restrictions in place at many operational forecasting centers, the S2S Database forecasts are only available three weeks after they are issued, which limits their use in experimental applications.

However, the NOAA MAPP Program’s/Climate Test Bed’s Subseasonal Experiment (SubX) research project data, also newly available from the IRI Data Library, does offer real-time experimental sub-seasonal forecasts, in addition to a set of forecasts for past dates (also called reforecasts). The SubX dataset combines North American global models from NOAA, NASA, Environment Canada, the Navy, and National Center for Atmospheric Research to produce once-a-week real-time experimental forecasts 3-4 weeks ahead. Previously using only 1-3 models, the real-time forecasts provide NOAA’s operational Climate Prediction Center with additional models to experimentally guide its week 3-4 outlooks.

“The S2S Database and SubX datasets are very complementary,” said Ben Kirtman, lead of the SubX project team. “The focus with the S2S data is forecasts from operational centers, whereas the SubX data includes forecasts from research models.”

Unlike the S2S Database, all of the models in the SubX dataset also apply a common framework and produce reforecasts identical to the real-time predictions, which supports equal model comparison and robust skill assessment, foundational to the research effort intended with the SubX project.

“I think that vetting the models in a real-time context is going to show us a lot and give us a lot of opportunity to understand subseasonal prediction challenges and where they are coming from and how we can work on them,” said  Kathy Pegion, an Assistant Professor at George Mason University who is a lead researcher with the SubX project.

The new datasets are already making it more efficient for researchers based in the U.S. to get their work done. Pegion said that her work involves evaluating models to improve forecasting, especially at the three to four week timescale.

“It’s not always easy to get access to the global models from around the world,” said Pegion. With both the S2S and SubX datasets in the IRI Data Library, she said, it makes it easier to compare more models, and lead to a better understanding of large scale climate drivers and sources of climate predictability.

Housing the S2S and SubX data at IRI was made possible through grants from the NOAA Office of Atmospheric Research’s Modeling, Analysis, Predictions, and Projections Program, and National Weather Service Next Generation Global Prediction System program. Collaboration with Suzana Camargo, Shuguang Wang and Haibo Liu at Columbia University through their NOAA MAPP (NA16OAR4310079; NA16OAR4310076) and NSF (#1543932) projects is gratefully acknowledged.


About MAPP

The Modeling, Analysis, Predictions, and Projections (MAPP) Program is a competitive research program in NOAA Research's Climate Program Office. MAPP's mission is to enhance the Nation's and NOAA's capability to understand, predict, and project variability and long-term changes in Earth's system and mitigate human and economic impacts. To achieve its mission, MAPP supports foundational research, transition of research to applications, and engagement across other parts of NOAA, among partner agencies, and with the external research community. MAPP plays a crucial role in enabling national preparedness for extreme events like drought and longer-term climate changes. For more information, please visit www.cpo.noaa.gov/MAPP.

View all posts by MAPP>>




Climate Program Office
1315 East-West Hwy, Suite 1100
Silver Spring, MD 20910



Americans’ health, security and economic wellbeing are tied to climate and weather. Every day, we see communities grappling with environmental challenges due to unusual or extreme events related to climate and weather. In 2011, the United States experienced a record high number (14) of climate- and weather-related disasters where overall costs reached or exceeded $1 billion. Combined, these events claimed 670 lives, caused more than 6,000 injuries, and cost $55 billion in damages. Businesses, policy leaders, resource managers and citizens are increasingly asking for information to help them address such challenges.