This page describes how to retrieve proton-detector data from satellite archives provided by the US National Oceanic and Atmospheric Administration (NOAA). By applying some Python code to this data, we will plot measurements of the radiation environment captured by satellites in geosynchronous orbit.
Table of Contents
First, a little back story. In this material, I lean heavily on the Wikipedia pages on the GOES satellites — they are a treasure trove and I thoroughly recommend you go wandering in Wikipedia-land.
For over 40 years, the US National Oceanic and Atmospheric Administration (NOAA) has operated a series of spacecraft called the Geostationary Operational Environmental Satellites (GOES). Starting with GOES-1 in 1975 (manufactured by Ford Aerospace — yes, that Ford), most of these satellites were successfully launched and served for many years, and have since been decomissioned.
The GOES family of satellites is still going strong: GOES-15 through GOES-17 are in orbit and either functioning, or standing by in case a repalacement is needed. Imagery from GOES-17 — launched just a few months ago — is already available on NASA's web site. Additional GOES launches are scheduled in 2020 and 2024.
Satellite technology has changed a lot in since 1975, and consequently, the instruments on these satellites vary quite a bit. Different GOES satellites have included:
- Proton, alpha-particle, X-ray, and electron flux detectors,
- Earth-facing optical, infrared, and near-infrared cameras,
- Solar imagers,
- Lightning detectors, and
- Radio equipment for communicating with the Earth.
Since I am specifically interested in the proton detectors — data that's readily available in a standard format from 1986 onwards — this is the data we will investigate. First, though, why protons?
Radiation is bad for electronics. Energetic protons are a big component of the radiation mixture in geosynchronous orbits.
Electronics in a radioactive environment generally exhibit two kinds of effects caused by radiation:
- Long-term effects (Jargon: "Total Ionizing Dose" or "TID"), in which radiation damage accumulates over time and leads to degradation and the eventual and permanent failure of spacecraft electronics, and
- Short-term effects (Jargon: Single-Event Effects, Single-Event Upsets, Single-Event Errors) caused by individual interactions between radiation and spacecraft systems. These effects may flip a bit from its correct value or generate a small transient signal that shouldn't be present.
These radiation effects occur on the ground, too, but the Earth's atmosphere absorbs most of the radiation that would otherwise scramble our electronics. On the ground, we mostly (but not always!) ignore the remaining errors. If your computer has ever inexplicably rebooted, the problem may have been cosmic radiation.
Back to space. These effects are not only caused by protons, but especially inside the spacecraft's shell where there is some shielding present, protons are a good enough proxy for radiation levels that we can focus on just one type of particle for now. (This is an useful simplification and a great way to get experts — I am not one of them — to start arguing.)
During solar storms, the arrival rates (fluxes) of protons are orders of magnitude higher than background conditions. The result? For long-term damage (TID), the quiet conditions between solar storms don't accumulate enough damage to matter at all.
For short-term damage (SEE), errors do occur during quiet conditions, and engineers do need to worry about their effects. However, the design of spacecraft electronics must accommodate (and is typically driven by) the worst conditions. For critical spacecraft systems (such as propulsion), survival is not enough: these systems are sometimes required to operate without error during during radiation events in order to maintain spacecraft function.
The frequency, source, intensity, and particle mixtures associated with solar radiation events are another good opportunity to fall down the Wikipedia hole. However, it's time to come back to the premise of this article and add a little Python code.
OK, let's get our hands dirty. The following Python code does not have terribly esoteric requirements, but we do require the following:
- Python 3.5 or newer (I use Debian's stock 3.6.6rc1 build),
- Asynchronous packages (asyncio, aiohttp)
- Scientific packages (numpy, scipy, matplotlib)
You can download the source code from Bitbucket as follows:
$ git clone firstname.lastname@example.org:gsmecher/goes-proton-python.git $ cd goes-proton-python
Because we make use of a large dataset, the processing pipeline is split into two stages:
- A script (fetch.py) that fetches GOES data from NOAA's archive into a local on-disk cache, and
- A script (analyze.py) that processes cached data (in bulk) and produces monthly radiation plots.
The first script is an attempt to be polite to the NOAA servers: we can (and do) still download a large dataset fairly quickly, but at least we can avoid downloading it more than once.
The second script does a batch analysis (and is aggressively parallel — it's your PC, so we have no need to be polite here.)
It has been a few years since I have developed Python code in earnest, and back then, the codebase I maintained was firmly in Python 2.6 territory. We made use of the wonderful asynchronous event loop provided by the Tornado project, and while watching the development of Python's batteries-included asynchronous framework I have often wished I could pull some of my older code into the more modern style. Bulk data fetch over HTTP was a great excuse to enter the brave world of Python 3.5+ asynchronous coding.
When you execute this script, it will create and populate a
directory. The default options fetch only a single year's worth of data for a
single satellite — if you want to download a larger dataset, you will have to
say so explicitly.
There are a number of ways of analyzing GOES data, including using the open-source SPEDAS <http://spedas.org>_ software (which I have not tried). If you are doing serious work, you are almost certainly better off not reinventing the wheel. Now, let's re-invent the wheel.
The GOES dataset is large, and because we're talking about decades of history and over a dozen satellites, it is natural to expect tension between the ideal data format and the compatible data format. The data we use here is formatted idiosyncraticically but not more than one would expect, given its age and scope.
GOES CSV is not strict CSV: it is CSV with embedded sub-tables, annotations, and metadata. I have not attempted to write a general-purpose parser for the overarching data format; rather, I have picked through the pieces of the file I need and created a degenerate parser that only understands what it needs to.
This code extends Python's DictReader with some rudimentary variable type conversion and teaches it to extract (and skip) the metadata blocks at the top of the CSV file.
Now that we can retrieve and parse GOES CSV data, we are ready to produce graphics from it. The analysis script leans heavily on matplotlib, which does not like threaded code. Hence, in order to generate a large volume of independent plots, I used the multiprocessing module instead.
By default, this code will search the entire
_cache produced by the
fetch.py script (above) and process everything in it, as quickly as it
can. For a large GOES cache, this can still take quite a while. The resulting
images (one per month per satellite) are placed in a
named per the original CSV data.
Let's take a look. Simply run:
This command produces a bunch of plots in the
This is an ordinary quiet day during an active sunspot cycle (1989 was near the peak of solar cycle 22). Before we proceed to a more spectacular example, let's take a look at this plot and figure out what information it contains.
This is a plot of proton intensity versus time, over the entire month of January 1989. Using different colours, we show superimposed plots for a variety of different sensors, each of which is sensitive to a range of proton kinetic energies. The vertical axis indicates proton flux, i.e. the number of particles seen by the detector at a given time. Apart from a normalization against proton kinetic energy, the units are logical. As you might expect, protons with low kinetic energies (e.g. the blue curve) are present at higher amounts than protons with higher energy levels (e.g. the magenta curve.) Although the detectors are sensitive to different ranges of proton energies, and hence the underlying measurements are probably incommensurate between sensors, the data has been normalized against these ranges to make comparisons between different sensors "fair". (Hence the "MeV" in the denominator in the y-axis units.)
In the blue trace, the satellite's orbital cycle in and out of the earth's shadow is clearly visible. The GOES-6 satellite orbits relatively close to the equator (14.7 degrees inclination) so it flies in and out of the Earth's shadow once per day. (By definition, geosynchronous satellites have a day that's exactly as long as ours. This is decidedly different from low-earth orbits, where a satellite might orbit over a dozen times per day!)
There are a few events visible on this day, particularly visible in the orange band. However, the really high-energy proton detectors do not really see anything except noise.
March 1989 was a different story, as we will now see.
In March 1989, a coronal mass ejection launched an intense volley of radiation at the earth. The resulting effects were substantial, even on the ground: Wikipedia claims a large power outage in Quebec, and a panic among Cold War spooks fearing a nuclear first strike.
Since the Y axis is in logarithmic units, and since these plots are normalized to wash away the effects of protons' kinetic energies, this plot understates just how bad the radiation event in March 1989 was. (In fact, data is missing for what is likely the worst portion of the event!)
...and with those two plots, we have successfully downloaded, analyzed, and plotted proton data from a geosynchronous satellite using Python — just as promised. I will leave off with a couple of interesting links:
- The Carrington Event was a severe geomagnetic storm occurring in 1859 — a long time ago, but recent enough history to have a lively written record of the event's effects.
- While decomissioning a low-earth orbit generally involves dropping it into the ocean, geosynchronous satellites are typically moved into a Graveyard orbit — far enough beyond the 30,000 km geostationary orbit to stay out of trouble.