The Eclipse: Tracking where the Moon's shadow GOES
Here is one of the coolest examples that I have created using IDL in a while. For this blog post, I'm going to walk through how I created an animation of the Moon's shadow during the Great American Total Solar Eclipse using several different technologies for accessing, downloading, and visualizing the data. Before getting into the details, here is some eye candy that shows a single frame of the generated animation. Note that this image was processed using IDL's hist_equal() function and it also has a color table applied to accentuate the shadow of the moon.
Now for the details! First, I want to add a small disclaimer on the state of the GOES data that is available on AWS. On the web page that I will mention below, the following is stated: While the GOES-16 ABI L1b and CMI data have reached provisional validation, please keep in mind that since GOES-16 satellite has not been declared operational, its data are still considered preliminary and undergoing testing. Luckily for the purposes of this example, we are just looking for a pretty picture of the eclipse. In order to do that, the workflow consisted of:
- Discovering that GOES data is available on Earth on AWS.
- Using node.js to create a script that uses open-source JavaScript modules from NPM to query and download the data locally.
- Wrote a script using IDL to extract the data and correctly display the data on a map.
- Extended the script to process a stack of images and produce a video animation of the data
Now for the details about how I accomplished each step. For the first step (maybe not really a step), I thought it would be good to talk about Earth on AWS. If you haven't yet, you should check out Amazon's great page which lists all of the free data that Amazon has available. In addition to GOES data you can also find Landsat, MODIS, Sentinel 2 and more. Such easy access to data is a great way to develop interesting applications without the hassle of needing to purchase the data that you want to use. The most important part of developing any application, including this example, is doing your research and understanding how to pull the data down from AWS and use it in an application.
In order to access the data, which is located in Amazon's S3 storage, I used JavaScript to take advantage of the free AWS SDK that contains easy-to-use tools for querying the contents of S3 "buckets." I used this to search and identify the files that I needed to download based upon different input parameters. In order to restrict the search, I stuck with the ABI-L1b-RadF product which you can learn more about at https://www.goes-r.gov/.
For the L1b radiance product over CONUS (CONtinental United States), the specific times that I searched for data were from Colorado time at 10 AM to 2PM on August 21st. This yielded a result of 60 different scenes (one approximately every 5 minutes for the datasets I was using). It's worth mentioning that in order to download and search for data using the AWS S3 SDK, you do need to have an account and an associated key and secret key. If you don't access keys for AWS, here is a link to Amazon's documentation which should provide the information that you need to get one or sign up:
https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/
Once I had my keys I was able to download the data for my desired times which, altogether, was about 900 MB total. Once I had the data I then needed to write some IDL code that would search for files, extract the radiance values, and place the data correctly on a map. Chris Torrence, a long time IDL developer, added support for the GOES map projection in IDL and the projection is a part of the most recent release (IDL 8.6.1). In order to use this, you can simply specify the 'GOES-R' map projection when creating an image or projection. To demonstrate this, here is an IDL example that shows how to accomplish this for a CONUS GOES file:
data = NCDF_Parse(file, /READ_DATA)
center_lon = data['geospatial_lat_lon_extent', $
'geospatial_lon_nadir', '_DATA']
radiance = data['Rad','_DATA']
xscale = data['x', 'scale_factor', '_DATA']
xoffset = data['x', 'add_offset', '_DATA']
x_radians = data['x', '_DATA']*double(xscale) + xoffset
yscale = data['y', 'scale_factor', '_DATA']
yoffset = data['y', 'add_offset', '_DATA']
y_radians = data['y', '_DATA']*double(yscale) + yoffset
i = image(hist_equal(radiance), x_radians, y_radians, $
RGB_TABLE=15, $
LIMIT=[15,-120, 50, -60], MARGIN=[0.1,0.02,0.08,0.02], $
MAP_PROJECTION='GOES-R', GRID_UNITS='meters', $
CENTER_LONGITUDE=center_lon, $
DIMENSIONS=[1000,600], $
TITLE='GOES-16 Level 1b Radiance CONUS')
mc = mapContinents(/COUNTRIES)
mc = mapContinents(/US)
From here, the code was just wrapped in a loop to generate the map for each file, capture what the map looked like, and then add each frame to an MP4 file using IDL. Here is a link to the official animation that was generated where you can see how the moon's shadow GOES (pun intended) across the US:
https://www.facebook.com/NV5GeospatialSoftware/videos/10155456330801006/
If you want to take a crack at this yourself, you can find all of the necessary files on GitHub (including an MP4 that you can download and view locally) at:
https://github.com/znorman17/goes-tools
Just know that the JavaScript code for downloading the data does not have the most error catching, so it is not very lenient if you make mistakes in the code. Note that, while each band of the GOES datasets might be small (bands from CONUS imagery range from 3 MB to 15 MB, with one about 80 MB) many files downloaded over a larger search period will take some time to download if you don't have a fast internet connection.
Enjoy!