By: Brad King
The Need for Web-Based Weather Maps
Mosaic has been working with weather data for decades through partnerships with the U.S. government, commercial airlines, and a diverse set of commercial customers. Mosaic’s consultants are skilled at transforming weather data for different use cases and for efficiently storing this information in scalable cloud-based systems.
In addition, Mosaic archives historical weather forecast data from multiple sources to enable efficient development and testing of custom data science solutions that feed into enterprise-scale systems. Mosaic technical consultants work closely with in-house meteorologists to ensure that the right data is deployed for each use case and to help interpret and combine weather information from different models and data streams.
WMS (Web Mapping Service) has long been Mosaic’s workhorse server-side method of delivering weather imagery to web clients. This is a widely used, OGC standard HTTP service specification for serving raster data as images (tiles) to display on a map. WMS relies on GeoServer to store converted files and formats. GeoServer responds to requests by dynamically generating images for tiles the client needs to satisfy the viewport bounding boxes.
In our new cloud-based world, Mosaic has been moving away from using GeoServer whenever possible due to its limitations. The platform requires at least one host running 24×7, a dedicated public URL, and multiple instances to scale. GeoServer also has to be configured for each NetCDF or GRIB file processed in order to create a new data source. In addition, while GIS-enabled databases like PostGIS can be used as a data source, weather data is large and time-consuming to process into a database – not to mention costly to maintain in large quantities.
GeoServer generates image tiles “on the fly” for requests. If the client application must dynamically change the color scale, or filter different levels, an “override” XML style document must be provided in each request to the server. This generates an unwieldy, massive URL, and inhibits the ability to cache image tiles, which relies on repeatable urls as cache keys.
Lastly, there is no reasonable way to keep historical weather data readily available for display with GeoServer. The data source approach requires keeping a NetCDF file “mounted” to GeoServer using configuration, and the creation of layers from that data source. This explodes into thousands of layers quickly, especially for weather products containing multiple future forecast horizons.
COG – The Evolution of Raster data
GeoTIFF has been around for a while and is a self-describing file format that also has imagery at multiple zoom levels. COG (Cloud Optimized Geotiff) is a new twist on GeoTIFF which takes the ability to create tilesets that live internally in a single GeoTIFF file and combines it to request overviews over HTTP, or tilesets at a specific zoom level. GeoTIFF can be served without COG, but would generally require loading the entire file to get only a subset of the data – a process that can be slow.
Using COG, the client plugin is able to request only the parts of the file it needs using a new feature introduced in HTTP 1.1 called GET Range Requests. This along with the compression provided in GeoTIFFs makes for a highly efficient method to deliver raster tiles to the client, with no server-side processing required. This is the magic that makes COG work directly in S3 and serve partial data based on zoom level and bounds.
As mentioned before, the COG file has no styling by default, only data. Data are colored based on a client-side function, using GeoTIFF data (such as altitudes), and a set of color ranges. Client-side styling gives us the maximum flexibility to allow users to filter unwanted levels, change to winter coloring modes, etc. This is a huge advantage over GeoServer Styled Layer Descriptors (SLD).
How Do I Create a COG File?
If you already have a standard GeoTIFF, use the rio-cogeo rasterio command line interface to generate a COG version of the file. The easiest way to get the tools is using conda. Note: you must have gdal version 3.1 or higher.
If you have NetCDF or another data input format, convert it to standard GeoTIFF with GDAL first, then use rio-cogeo.
Conclusion
Using the COG generation process and a naming convention for files stored in S3, an indefinite number of weather products and files can be readily available for web client display using only S3 infrastructure.
COG generation is fast and efficient compared to processing and serving NetCDF using GeoServer, and client-side styling functions provide total flexibility for developers generating color ranges, or handling the raw data however is necessary.
Weather has a high impact on operations in many industries and therefore is of great value to integrate into strategic decision making. Mosaic’s roots in aviation research & development, give us deep expertise in combining weather data streams with planning applications to facilitate efficient resource allocation. Web-based weather maps make the collection and storage of this data more efficient & effective.
Interested to learn how we can help you develop web-based weather maps?
