Three-Speed Logic

Running Vivado in an LXC Container

2020-06-25T00:00:00-07:00

Table of Contents

Summary
- The Problem
- The Solution
Questions and Answers

Summary

Don't run Vivado on your host OS. Run it in an LXC container instead. Here's why and how.

Each Vivado release is supported on a limited set of Linux distributions and releases. While Xilinx documents this well in Vivado's release notes, and attempts to maintain compatibility, newer Vivado releases require newer (but not quite the newest) OSes.

When you need to run multiple Vivado versions, you can end up gridlocked, unable or unwilling to upgrade or maintain your PC. For example:

You cannot, with a single officially-supported distribution and release, run both new and old versions of Vivado. For example, Vivado 2020.1 and Vivado 2016.1 are not both supported on any single version of any Linux distribution.
"Rolling" changes to many Linux distributions, even within a single version, can also break Vivado. This is a disincentive to regular OS maintainance and can have security implications.
While you can often run Vivado outside the officially-supported distribution list, it does not always work. Sometimes the problems are easily fixed (or google-able). Sometimes you will be the first to stumble across an obscure segfault. Xilinx support will often (and understandably) refuse to offer support in this scenario.
Many other software packages (TI's Code Composer Studio; Microsemi's Libero; MATLAB) also impose restrictions on OS releases. Managing to find a single OS that supports all your software becomes more unlikely the more packages impose these limitations.
If upgrading Vivado requires you to upgrade your OS, you will defer upgrades until you absolutely need them.

This is no way to live. Besides, Debian isn't on Xilinx's approved list, and who wants to run a host OS that isn't Debian?

The Solution

By running Vivado within an LXC container, we can separate the OS we run (the "host" OS) from the OS Vivado perceives (the "guest"). By adding an extra OS layer, we are free to run multiple different guests for different versions of Vivado, and upgrade the host without worrying about breaking something obscure.

The following instructions assume the following:

Host: Debian Testing, with full (sudo) access.
Guest: Ubuntu 18.04.4 (bionic)
Vivado: 2020.1

I have used this recipe for several Vivado and Ubuntu releases, on the same Debian Testing host.

These LXC directions are cribbed from the Debian wiki (https://wiki.debian.org/LXC). Please go there for fuller, and possibly better maintained, guidance.

Installing and Configuring LXC on the Host

Install the necessary packages:

$ sudo apt-get install lxc libvirt-clients libvirt0 libpam-cgfs bridge-utils uidmap

We will follow the "simple bridge" (lxc-net) version of the instructions. Create the file /etc/default/lxc-net with the following contents:

USE_LXC_BRIDGE="true"
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
LXC_DHCP_CONFILE=""
LXC_DOMAIN=""

Now enable and start lxc-net:

host$ sudo systemctl enable lxc-net
host$ sudo systemctl start lxc-net

At this point I had to reboot my system to see the network device lxcbr0 appear:

host$ sudo reboot
[...]
host$ sudo ifconfig lxcbr0
lxcbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
        inet 10.0.3.1 netmask 255.255.255.0 broadcast 0.0.0.0
[...]

Creating the Container

Create and start a container named vivado2020_1 as follows:

host$ sudo lxc-create -n vivado2020_1 -t ubuntu -- -r bionic

Do not start the container yet (we'll do that below).

Configuring the Container

Now, in the host, edit the container configuration (/var/lib/lxc/vivado2020_1/config) and add the following lines:

# Network configuration
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxcbr0
lxc.net.0.hwaddr = aa:bb:cc:dd:ee:ff

# Xilinx
lxc.mount.entry=/home/your_username /var/lib/lxc/vivado2020_1/rootfs/home/your_username none bind 0 0
lxc.mount.entry=/opt/xilinx /var/lib/lxc/vivado2020_1/rootfs/opt/xilinx none bind 0 0

# Automatically start this container
lxc.start.auto = 1

You will have to modify three things here:

The "hwaddr" should correspond to an arbitrary MAC address with a valid Xilinx license.
your_username should be changed to match your setup
/opt/xilinx should be altered to match your host's Xilinx installation (/opt/Xilinx is default, I think.)

You will have to create the bind-mount directories on your host, matching your alterations above:

host$ sudo mkdir -p /var/lib/lxc/vivado2020_1/rootfs/home/your_username
host$ sudo mkdir -p /var/lib/lxc/vivado2020_1/rootfs/opt/xilinx

Now, start the container, attach to it, and configure it as follows:

host$ sudo lxc-start -n vivado2020_1
host$ sudo lxc-attach -n vivado2020_1

This last command will bring up a root prompt inside the container. Install a few basic packages:

guest# apt-get update
guest# apt-get install avahi-daemon openssh-server xutils x11-apps xauth

You will also want to modify the ubuntu user to match your UID and GID on the host, and set its password:

guest# usermod -u 1000 -g 1000 -d /home/your_username -l your_username ubuntu
guest# passwd your_username

...where the UID 1000 and GID 1000, here, should be altered to match your UID and GID on the host. You can find these by running id -u and id -g on the host.

You can now exit out of the root session and restart the container:

host$ sudo lxc-stop -n vivado2020_1
host$ sudo lxc-start -n vivado2020_1

You now have a user in the container that matches your user on the host. To avoid entering your password each time, use the ssh-copy-id script:

host$ ssh-copy-id vivado2020_1.local

If these steps worked, you can now "ssh" into the machine and run the all-important xeyes test:

host$ ssh -XC vivado2020_1.local xeyes

If the eyes show up, congratulations! You are able to connect to the container and forward X11 apps, which is how you'll be using Vivado.

Installation and Licensing

If your host already has Vivado installed, it is also installed on your client. (That's what the lxc.mount.entry line above accomplished.)

If not, you can now install it inside your guest (but note that you will have to grant write permission on the host's /opt/xilinx to the user within your guest!) Your environment inside the LXC environment should be similar to the environment outside.

Questions and Answers

How do Vivado licenses work with LXC?

This depends on you.

Network licenses likely "just work" once you set up the server.

My Vivado installations are licensed by Ethernet MAC address. In this regime, the simplest (and most legitimate) way is to issue a separate license file for each guest. You will need to assign randomly generated MAC addresses to your guests via the config file you modified above. (See the hwaddr line.)

If you are interested in re-using the same MAC-based license as your host uses (I am not advocating this!), you should be able to create a dummy Ethernet interface in your guest with the same MAC address as your host's licensed Ethernet address. Provided no traffic flows over this interface, the fact that we have two Ethernet adapters on the same PC with the same MAC address is not problematic. While this is probably against Xilinx's terms of service, it's hard to imagine they would really have a problem with you running the software on the PC it's licensed with (albeit with an extra OS layer between the licensed MAC and Vivado.)

Why Not a Virtual Machine (VM)?

Most FPGA designers will be familiar with Virtual Machines (VMs) like VirtualBox or VMWare. These programs, with the help of OS- and processor-level features, emulate an entire PC and provide a well-isolated sandbox for you to install a "guest" OS on your PC. This guest OS is completely separate from your host OS, which can run several guests at a time.

You can install Vivado in a VM running an approved Linux distribution. However, you shouldn't. VMs generally require a fixed allocation of resources (RAM, disk, CPUs). When you run Vivado, you typically want it to consume as much CPU and RAM as it needs, up to the limits imposed by your hardware. Whenever your VM requires a static allocation, you are artificially hobbling the performance of both the host and guest. Even if you don't run into artificial resource limits imposed by static allocations, a VM will perform worse due to the extra virtualization overhead over a container.

What's a Container?

A container is like a VM, only less virtualized. Where a VM host virtualizes the CPU and allows a guest to run its own OS, a container runs only a single kernel and virtualizes only the userspace environment. The host and guest share kernel, and hence hardware, drivers, CPUs, and memory.

What's LXC?

LXC is one of several container management packages commonly provided in Linux distributions. Alternative container services include LXD (a user-friendlier layer on LXC, but which is not packed for Debian) and Docker, which solves a slightly different problem.

Why Not Docker?

Docker, like LXC, is a container management service. However, Docker is intended to build short-lived, reproducible containers for non-interactive use. Vivado is a pretty poor match for these environments because it is so big, and because reproducing an automated Xilinx installation is not easy. Although you can find Docker instructions (Dockerfiles) for Vivado on-line, they are more of a hassle than the above procedure.

What about multiple users / non-sudo LXC?

Sorry, it's possible but it's not what I describe here. I use this on my laptop and workstation, so I'm the only one driving. (I am aware that the manual uid remapping is slightly cheesy!)

Surely there's a Catch?

It is possible for the host kernel (which, remember, is shared with your guest OS) to be incompatible with something in the guest's userspace or in Vivado. For example, kernel-level security features might expose a latent bug or assumption in the guest OS that Vivado doesn't tickle when used in a non-containerized environment. I have not surveyed past versions of Vivado carefully to check for bugs. As with all free internet advice, you are welcome to apply for a refund.

C/RTL Cosimulation with Vivado and Python

2020-06-01T00:00:00-07:00

Table of Contents

Summary
Motivation
Demonstration
Conclusions

Summary

Xilinx's xsim simulator (UG900) includes the Xilinx Simulator Interface (XSI), a way to embed a RTL simulation kernel inside a C/C++ program. This page further embeds XSI kernels within a Python session, creating a layered simulation environment where Python, C/C++, and RTL coexist. Testbenches can be coded in a mixture of the three languages, removing the hard boundaries that surround an RTL-only testbench environment.

Warning: As of Vivado 2020.2, fatal exceptions in RTL (VHDL: assert False severity failure) currently leave the simulator kernel in a deeply inconsistent state. Xilinx is aware and has promised a fix in an upcoming Vivado release. You will have to demote your exceptions to a non-fatal severity in the meantime.

Motivation

RTL testbenches are a good way to verify low-level design elements, but RTL testbenches alone are not good enough.

Most importantly, I want to test RTL using best-of-breed tools from the software world. I want to automate regression testing, launch tests quickly and in parallel, and automatically discover tests. I want tests to be parameterized, randomized, and able to execute remotely without struggling to manage licensing limitations.

Next, I want to verify a combination of C and RTL code, where the interface between C and RTL is complex and malleable. Driving a wedge between C code and RTL code so I can extract test vectors via file I/O feels like a monumental distraction.

Also, I need to generate or post-process test data in Python. For signal-processing analysis, my go-to environment is Python/Scipy/Matplotlib. For data analysis, there is nothing in C or RTL that offer the same broad reach and high productivity. You can do this by running pre- and post-processing scripts, but it is brittle and frustrating for no justifiable reason.

Yes, there's more: I want my test environment to resemble my deployment environment. My design is an amalgam of C and RTL when it's deployed; why should the testing environment be different? (Here, ASIC designers will be tearing out their hair. It's OK, I understand.) Combining C and RTL reduces the amount of hassle associated with testing, which allows me to focus on solving problems and not fighting tools.

Finally: I need all of this to work with mixed languages (SystemVerilog/VHDL) and encrypted IP. This, unfortunately, shuts out all of the open-source simulators (Verilator, GHDL, CXXRTL) I would love to be supporting and using. While the open-source tools are creeping closer to mixed-language simulation, it is difficult to see a (legal) path towards supporting encrypted IP in an open simulator. Practically speaking, I'm stuck with either Xilinx's simulator (xsim), or very expensive commercial options.

Compounding the issue further: many excellent test environments (cocotb, VUnit, UVVM) do not work with Xilinx's simulator. As a result, my options for higher-level testing constructs focused on an RTL world dwindle from "frustrating" to "squalid". This is largely Xilinx's fault: although it is slowly improving, xsim's support for SystemVerilog and VHDL-2008 constructs lags other simulators [1].

Demonstration

The following sections show a start-to-finish build, from checking out the source code to completing a test.

Check Out the Source Code

Source code is available at https://github.com/gsmecher/pyxsi.

$ git clone https://github.com/gsmecher/pyxsi.git
Cloning into 'pyxsi'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 14 (delta 0), reused 14 (delta 0), pack-reused 0
Receiving objects: 100% (14/14), 6.72 KiB | 6.72 MiB/s, done.

Build the RTL

Now set up Xilinx's environment variables and build the RTL into a simulation library:

$ cd pyxsi
pyxsi$ . /opt/xilinx/Vivado/2019.2/settings64.sh
pyxsi$ make rtl
xelab work.widget -prj rtl/widget.prj -debug all -dll -s widget
Vivado Simulator 2019.2
Copyright 1986-1999, 2001-2019 Xilinx, Inc. All Rights Reserved.
Running: /opt/xilinx/Vivado/2019.2/bin/unwrapped/lnx64.o/xelab work.widget -prj rtl/widget.prj -debug all -dll -s widget
Multi-threading is on. Using 2 slave threads.
Determining compilation order of HDL files.
INFO: [VRFC 10-163] Analyzing VHDL file "/home/foo/bar/baz/pyxsi/rtl/widget.vhd" into library work
INFO: [VRFC 10-3107] analyzing entity 'widget'
Starting static elaboration
Completed static elaboration
Starting simulation data flow analysis
Completed simulation data flow analysis
Time Resolution for simulation is 1ps
Compiling package std.standard
Compiling package std.textio
Compiling package ieee.std_logic_1164
Compiling package ieee.numeric_std
Compiling architecture behav of entity work.widget
Built XSI simulation shared library xsim.dir/widget/xsimk.so

Build the C++ Code

Now build the C++ code:

pyxsi$ make
g++ -fPIC -std=c++17 -I/usr/include/python3.8 -I/opt/xilinx/Vivado/2019.2/data/xsim/include -Isrc -c -o pybind.o src/pybind.cpp
g++ -fPIC -std=c++17 -I/usr/include/python3.8 -I/opt/xilinx/Vivado/2019.2/data/xsim/include -Isrc -c -o xsi_loader.o src/xsi_loader.cpp
g++ -fPIC -std=c++17 -I/usr/include/python3.8 -I/opt/xilinx/Vivado/2019.2/data/xsim/include -Isrc -shared -o pyxsi.so pybind.o xsi_loader.o -ldl

Execute Tests

Finally, tests are discovered and executed using Python's pytest environment.

pyxsi$ make test
============================= test session starts ==============================
platform linux -- Python 3.8.3rc1, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
rootdir: /home/foo/bar/baz/pyxsi
plugins: xdist-1.32.0, forked-1.1.3
collected 2 items
[...]
=========================================== 2 passed in 7.16 seconds ===========================================

Conclusions

This is only a skeletal example, with just enough scaffolding to build on.

Since this article was first published, I've extended my own experiments into a fully-fledged regression-testing environment. It's perfect for signal-path environments. If you're curious, please ask for details.

[1]	I use xsim because none of the open-source simulators can combine VHDL and Verilog, or simulate encrypted IP. Commercial simulators don't make sense for a small instrumentation consultancy (mine, anyway).

XSim's New AXI Analyzer

2019-12-05T00:00:00-08:00

Table of Contents

Motivation
AXI Analyzer in Block Designs
Using the AXI Analyzer in RTL-Only Designs
Fully Expanded AXI Analyzer

Motivation

If you've developed AXI machinery in Vivado, you're likely familiar with this:

This screenshot from Xilinx's built-in simulator (xsim) shows two AXI4-Lite writes followed by a single read. It's functional but, for designs of any complexity, it's tough to keep track of both the forest (the high-level behaviour you're presumably debugging) and the trees (the nitty-gritty associated with each AXI transaction).

Sometimes I use markers and signal colors to make navigation easier, but these are throwaway techniques that aren't useful for more than a single simulator session at a time.

I often wish for some easily available visual cues that support moving around the simulation without losing focus on the problem I'm trying to solve.

AXI Analyzer in Block Designs

You may be surprised to hear that Vivado 2019.1 added something better. From the 2019.1 release notes:

Vivado Simulator

"Mark Simulation" feature in block diagram to add AXI interfaces in waveform viewer directly

That's a little oblique. Since a picture speaks a thousand words:

That thing at the bottom is a "protocol analyzer", and if you're doing your development in a Block Design, you may have already stumbled across it. Lucky you.

Using the AXI Analyzer in RTL-Only Designs

For RTL designs, unfortunately, it's not so easy. It does not appear that the Vivado machinery that infers AXI buses from port names and a few judicious X_INTERFACE-style attributes (VHDL) or comments (Verilog) is hooked up here.

Unofficially, here's how you can get this working in Vivado 2019.2.

Design Heirarchy

Because we need to name design units within the heirarchy, I've included a screenshot showing the heirarchy itself.

This is a perverse mixture of VHDL, SystemVerilog, and Xilinx cores.

The simulation top-level (top) is SystemVerilog (because I'm using the AXI VIP, which relies on SystemVerilog mojo).
My custom AXI4-Lite peripheral (axi_hw) is written in VHDL, since that's what the majority of my designs use.
For AXI stimulus, I'm using Xilinx's AXI VIP. (As an aside, the project settings matter here: Vivado 2019.2 segfaults when generating the simulator unless the VIP's generated RTL is in Verilog, and the project settings are what determines this.)

Since the top-level SystemVerilog is relatively short, I've pasted it in entirety here.

`timescale 1ns / 1ps

import axi_vip_pkg::*;
import axi_vip_0_pkg::*;

module top #(
    parameter integer C_AXI_ADDR_WIDTH=12,
    parameter integer C_AXI_DATA_WIDTH=32) ();

logic m_axi_aclk=0, m_axi_aresetn=0;

logic [3:0] led;

logic [C_AXI_ADDR_WIDTH-1:0] m_axi_awaddr, m_axi_araddr;
logic m_axi_awvalid, m_axi_awready, m_axi_wvalid, m_axi_wready;
logic m_axi_arvalid, m_axi_arready, m_axi_rvalid, m_axi_rready;
logic m_axi_bvalid, m_axi_bready;
logic [C_AXI_DATA_WIDTH-1:0] m_axi_wdata, m_axi_rdata;
logic [C_AXI_DATA_WIDTH/8-1:0] m_axi_wstrb;
logic [1:0] m_axi_bresp, m_axi_rresp;

always #5ns m_axi_aclk = ~m_axi_aclk;

axi_hw #(C_AXI_ADDR_WIDTH, C_AXI_DATA_WIDTH) dut (
    .s_axi_aclk(m_axi_aclk),
    .s_axi_aresetn(m_axi_aresetn),
    .s_axi_awaddr(m_axi_awaddr),
    .s_axi_awvalid(m_axi_awvalid),
    .s_axi_awready(m_axi_awready),
    .s_axi_wdata(m_axi_wdata),
    .s_axi_wstrb(m_axi_wstrb),
    .s_axi_wvalid(m_axi_wvalid),
    .s_axi_wready(m_axi_wready),
    .s_axi_bresp(m_axi_bresp),
    .s_axi_bvalid(m_axi_bvalid),
    .s_axi_bready(m_axi_bready),
    .s_axi_araddr(m_axi_araddr),
    .s_axi_arvalid(m_axi_arvalid),
    .s_axi_arready(m_axi_arready),
    .s_axi_rdata(m_axi_rdata),
    .s_axi_rresp(m_axi_rresp),
    .s_axi_rvalid(m_axi_rvalid),
    .s_axi_rready(m_axi_rready),
    .led(led));

axi_vip_0 vip(
    .aclk(m_axi_aclk),
    .aresetn(m_axi_aresetn),
    .m_axi_awaddr(m_axi_awaddr),
    .m_axi_awvalid(m_axi_awvalid),
    .m_axi_awready(m_axi_awready),
    .m_axi_wdata(m_axi_wdata),
    .m_axi_wstrb(m_axi_wstrb),
    .m_axi_wvalid(m_axi_wvalid),
    .m_axi_wready(m_axi_wready),
    .m_axi_bresp(m_axi_bresp),
    .m_axi_bvalid(m_axi_bvalid),
    .m_axi_bready(m_axi_bready),
    .m_axi_araddr(m_axi_araddr),
    .m_axi_arvalid(m_axi_arvalid),
    .m_axi_arready(m_axi_arready),
    .m_axi_rdata(m_axi_rdata),
    .m_axi_rresp(m_axi_rresp),
    .m_axi_rvalid(m_axi_rvalid),
    .m_axi_rready(m_axi_rready));

/* AXI VIP stimulus */
axi_vip_0_mst_t master_agent;
logic [31:0] x=0;
xil_axi_resp_t resp;
initial begin
    /* Create an agent */
    master_agent = new("axi_vip_0_mst",vip.inst.IF);
    master_agent.set_agent_tag("Master VIP");
    master_agent.start_master();

    /* Remove reset */
    #50ns m_axi_aresetn  = 1;

    /* Poke LED bit 0 */
    #20ns master_agent.AXI4LITE_WRITE_BURST(
             12'h000,0,32'h1,resp);

    /* Poke LED bit 1 */
    #20ns master_agent.AXI4LITE_WRITE_BURST(
             12'h004,0,32'h1,resp);

    // Peek LED bit 0 */
    #20ns master_agent.AXI4LITE_READ_BURST(
             12'h000,0,x,resp);

    #100ns
    $finish;
end
endmodule

Creating the Protoinst File

When using block designs, Vivado creates .protoinst files to annotate AXI buses. These files are passed to xsim in order to create protocol analyzers.

Xilinx is not yet ready to generate protoinst files for RTL-only designs, so we do it by hand:

{
   "version": "1.0",
   "modules": {
      "top": {
         "proto_instances": {
            "/M_AXI": {
               "interface": "xilinx.com:interface:aximm:1.0",
               "ports": {
                  "ACLK": { "actual": "m_axi_aclk"},
                  "ARADDR": { "actual": "m_axi_araddr"},
                  "ARESETN": { "actual": "m_axi_aresetn"},
                  "ARREADY": { "actual": "m_axi_arready"},
                  "ARVALID": { "actual": "m_axi_arvalid"},
                  "AWADDR": { "actual": "m_axi_awaddr"},
                  "AWREADY": { "actual": "m_axi_awready"},
                  "AWVALID": { "actual": "m_axi_awvalid"},
                  "BREADY": { "actual": "m_axi_bready"},
                  "BRESP": { "actual": "m_axi_bresp"},
                  "BVALID": { "actual": "m_axi_bvalid"},
                  "RDATA": { "actual": "m_axi_rdata"},
                  "RREADY": { "actual": "m_axi_rready"},
                  "RRESP": { "actual": "m_axi_rresp"},
                  "RVALID": { "actual": "m_axi_rvalid"},
                  "WDATA": { "actual": "m_axi_wdata"},
                  "WREADY": { "actual": "m_axi_wready"},
                  "WSTRB": { "actual": "m_axi_wstrb"},
                  "WVALID": { "actual": "m_axi_wvalid"}
               }
            }
         }
      }
   }
}

You will likely need to modify the instance name ("top") and AXI signal names ("m_axi_awaddr" etc.) to match your design.

Launching the Simulator

Once your .protoinst file matches your design, you need to pass it to xsim. I find this most convenient to do on the command line, outside of Vivado. The first step is to find your simulation directory:

# from somewhere in the Vivado project directory:
$ find . -name simulate.sh
./my_design.sim/sim_1/behav/xsim/simulate.sh
$ cd my_design.sim/sim_1/behav/xsim

This directory contains a simulate.sh script generated by Vivado that invokes xsim:

$ tail -2 simulate.sh
xsim top_behav -key {Behavioral:sim_1:Functional:top} -tclbatch top.tcl -log simulate.log

All we need to do is add the "-protoinst" option (pointing to our JSON file):

$ xsim top_behav -key {Behavioral:sim_1:Functional:top} \
        -tclbatch top.tcl -log simulate.log \
        -protoinst my_design.protoinst

Fully Expanded AXI Analyzer

That's it! You should now be able to annotate AXI transactions in an RTL-only design. I'll leave you with a fully expanded image of the AXI analyzer showing what signals and derived information are visible.

Designing CIC Compensation Filters

2019-11-18T00:00:00-08:00

This page describes how to design CIC compensation filters using open-source tools. Due to the use of newer algorithms, compensators designed using this method should outperform MATLAB's built-in "ciccomp" filters.

Table of Contents

A Little Motivation
Introducing the CIC Filter
A Magic Trick: Small-Angle Approximation
Compensating CIC Filters
The Solution
References

A Little Motivation

Occasionally, DSP designers need to build decimators.

Here, the goal is to remove high-frequency content from a sampled data source, producing a new sequence of data samples at a lower sampling rate.

From a theoretical perspective, the canonical approach to decimation is to use a Finite Impulse Response (FIR) filter. FIR decimators are well-understood, well-behaved, and approachable. FIR design is mature, and there exists a canonical FIR design tool (the [Parks-McClellan] or Remez algorithm) that makes it straightforward to turn a filter specification into a set of FIR coefficients. Implementations of this algorithm are available in Python/Scipy and MATLAB. More recent work has produced variations on the algorithm with better convergence [Filip].

From a practical perspective, a naive implementation of a FIR decimator (that is, a FIR followed by a decimation stage) is computationally inefficient. Here, too, there's an obvious path to follow. Polyphase techniques [Vaidyanathan] can be used to decompose a single, large FIR into a set of parallel FIRs operating at a reduced data rate.

As flexible and useful as FIR decimators are, there are a few corner cases where they don't produce adequate designs:

When the decimation ratios involved are very high, the number of FIR coefficients needed to meet performance requirements grows. Even if the Parks-McClellan algorithm does converge to an usable FIR, the resulting coefficient vector may be impractical to use even after polyphase decomposition.
When memory (coefficient and signal buffers) is very highly constrained (i.e. in an FPGA, or when processing many signals in a design.) The polyphase decomposition reduces the computational burden associated with the FIR, but does nothing for the amount of memory required.
When an FIR does not offer adequate flexibility (see, for example, the variable-sampling-rate trick described below.)

In these situations, a CIC is an interesting alternative.

Introducing the CIC Filter

A CIC, or Cascaded Integrator-Comb, is a filter that's much more restrictive than an FIR. It has a very wide transition region and is only really effective as an anti-aliasing filter for very narrowband signals. However, it's computationally extremely efficient (it requires only adders and registers, and relatively few of each) and its shortcomings can be corrected. When paired with a conventional FIR, the resulting compound decimator plays to the strengths of each.

The original CIC paper [Hogenauer] is among my favourite DSP references. It is well written and approachable, and revolves around a single mathematical trick so filthy it's genuinely worth celebrating. Please stop here and go find read the original paper.

Assuming you have skimmed the original CIC paper, I will now pull out several tidbits. (Note that I'm focused on decimators, here, but there is always a dual development that applies equally to interpolators.)

In the following, we assume the compound decimation structure shown above, in which large decimation factors are dealt with by the CIC. The FIR exists only to compensate or "clean up" passband droop imposed by the CIC, and decimates by a final factor of 2 because it is convenient and efficient to do so.

A Magic Trick: Small-Angle Approximation

The amplitude response for a CIC decimator is (using the notation from [Hogenauer]):

$$A(f) = \left[ \frac{\sin (\pi M f)}{\sin \left( \frac{\pi f}{R}\right)} \right]^N$$

The terms in this equation are defined as follows:

$N$ is the number of stages in the CIC filter,
$M$ is a differential delay (a tunable parameter, though I only ever use 1),
$R$ is the CIC's rate-change parameter, and
$f$, here, is a normalized frequency in which 0.5 corresponds to the Nyquist rate.

Hogenauer notes the small-angle approximation for the denominator term:

$$\newcommand{\sinc}{\mathop{\rm sinc}\nolimits} A(f) \approx \left[ RM \right]^N \sinc^N(\pi M f)$$

where $\sinc(x) = \sin (\pi x)/(\pi x)$.

Let's assume our compensation filter will only correct the frequency-dependent portions of this expression. (When $[RM]^N$ is a power of 2, as they are in my case, these gains are typically corrected using bit shifts, which are free in an FPGA.) We are left with the frequency-dependent portion of the expression

$$\sinc^N \left( \pi M f \right)$$

To correct for this amplitude droop, the FIR must impose the inverse gain on its passband:

$$C(f) = \sinc^{-N}(\pi Mf)$$

for $f$ over a passband extending as high as the normalized frequency 1/4 (that is, the Nyquist rate at the FIR's output, remembering that it is also has a decimation-by-2 rate change function.) In reality, the filter's passband edge must be relaxed below the Nyquist rate to introduce a transition region, allowing the filter's design to be tractable.

The decimation rate R does not appear in this expression! That is, provided the small-angle approximation is appropriate, a single single, fixed compensation FIR will suffice for any CIC decimation rate. This provides an easy way to build a signal path that's adaptable to multiple operating scenarios without being redesigned each time.

We have motivated the need for a compensating filter, and proposed a structure for it. We have also calculated the required passband amplitude to correct for droop in the CIC's passband. We are finally ready to calculate it.

Compensating CIC Filters

Of the dominant signal-processing packages, unfortunately, only MATLAB provides the necessary machinery to create a CIC compensator FIR "out-of-the-box". In the past, I have managed to scrounge up a MATLAB installation in the rare case I need it. However, my lab shifted wholesale to Python a decade ago, and I'm otherwise completely MATLAB-free. Sorry, Mathworks, it's time to move on.

Here are a few open-source Parks-McClellan or Remez implementations:

Unfortunately, none of these implementations is suitable on its own. The Liquid SDR implementation allows a callback to inject the desired amplitude responses $C(f)$; however, this implementation did not converge well enough for me to use it. Where other implementations of the Parks-McClellan algorithm admitted a stepwise or piecewise-linear approximation to $C(f)$, experiments with this approach were far too brittle to produce usable results.

I was, however, able to add a callback-enabled version of Silviu-Ioan Filip's code.

The Solution

I began with the source code here:

https://github.com/sfilip/firpm [Filip]

This code contains three implementations of an extended Parks-McClellan algorithm: two using underlying C types (double and long double), and one using an arbitrary-precision library (Pavel Holoborodko's excellent C++ wrappers for the also excellent GNU MPFR library.) I opted for the third, since I am more interested in limiting developer commitment than maximizing computational efficiency.

(As an aside, this code would benefit greatly from some refactoring by an experienced C++ developer.)

My modifications are contained in a new repository here:

https://bitbucket.org/gsmecher/ciccomp

Enjoy!

References

The following links are often behind IEEE's paywall. However, the papers referenced here are "classics" and are easily available with a little creative searching. Both the [Vaidyanathan] and [Hogenauer] references are among my all-time favourites for both content and presentation.

[Vaidyanathan]

(1, 2) P.P. Vaidyanathan, "Multirate Digital filters, filter banks, polyphase networks, and applications: a tutorial". In Proc. IEEE, vol. 78, Jan. 1990. https://ieeexplore.ieee.org/document/52200)

[Hogenauer]

(1, 2, 3) E. Hogenauer, "An Economical Class of Digital Filters for Decimation and Interpolation". In IEEE Trans. Acoustics, Speech, and Signal Processing, vol. 29, Apr. 1981. https://ieeexplore.ieee.org/document/1163535

[Parks-McClellan]

J.H.McClellan and T.W.Parks, "A Personal History of the Parks-McClellan Algorithm". In IEEE Signal Processing Magazine, vol. 22, Mar. 2005. https://ieeexplore.ieee.org/document/1406492

[Filip]

(1, 2) Silviu-Ioan Filip, "A Robust and Scalable Implementation of the Parks-McClellan Algorithm for Designing FIR Filters". In ACM Trans. Mathematical Software, Vol. 43, Aug. 2016. https://dl.acm.org/citation.cfm?id=2904902

Tworoutines in Python

2018-11-19T00:00:00-08:00

This article describes a style of coding in Python that permits easy mixing of synchronous and asynchronous code. As part of the control software for large microwave telescopes (including the South Pole Telescope), we have been using this style of code under a Tornado / Python 2.x stack with success.

Unfortunately, architectural changes in Python 3.7 conspire against the @tworoutine. In the hopes of contributing to a lively discussion about Python's asynchronous ecosystem, we describe why they have been so useful to us.

Table of Contents

Introduction
Coding for Telescopes
Enter the @tworoutine
Lament: Of Course There's A Catch
Disclaimer

Introduction

Asynchronous coding in Python was pioneered by third-party libraries like Twisted, Tornado, and gevent. An "official" event-loop implementation landed in Python 3.4, and was expanded significantly in Python 3.7. A new breed of asynchronous libraries like curio and trio continue to push the boundaries beyond what's "normal" in the space.

There are also some excellent (and opinionated) articles about Python's asynchronous ecosystem. I don't always agree with them and I don't intend to recapitulate them. To allow me to get the point, though, I will provide a few links that set the stage for what follows.

Of these, the last one is probably the most interesting because it identifies and attempts to address the same problem we run into when designing telescope tuning software: asynchronous and synchronous coding styles occupy different universes in Python, but it is exteremly useful to mix them freely.

To motivate mixing synchronous and asynchronous code, here is a short description of the kind of code we write for tuning telescopes.

Coding for Telescopes

My day job includes work on CMB telescopes including the South Pole Telescope in Antarctica and the Simons Array on Chile's Atacama Plateau.

The readout electronics in these telescopes is a large array of software defined radios, with many thousands of transmitters and receivers used to bias and measure the leftover signature of the Big Bang. These radios are implemented in hundreds of custom boards hosting FPGAs installed in crates near the telescope, and controlled by a PC. This PC gets the system up and running, controls cryogenic refrigerators, aims the telescope, and captures the torrent of data it produces.

The entire tuning, control, and analysis stack makes very heavy use of Python, along with C, C++, and VHDL. (I am inexpressibly grateful to the many open-source communities we rely on, and it is a great privilege when I can give back in some capacity.)

As you can imagine, we don't just deploy code straight onto the telescope. Along with the telescopes themselves are small-scale installations ranging from a circuit board or two on a benchtop, to crates of cryogenic equipment at university labs around the world. During development, code might be running in a Jupyter notebook or an IPython shell, perhaps with a small crate of electronics or nothing at all. Here, interactive REPL sessions are used to prototype algorithms, explore data, and try out new tuning and analysis techniques.

For an algorithm to be useful in deployment, however, it needs to run at scale. Here's where we use asynchronous code heavily: command interactions with many hundreds of circuit boards are a natural fit for asynchronous coding styles. This leads to the following workflow:

Design flow, with separate asynchronous/synchronous implementations:

Prototype code, probably synchronous and focused on proofing out an algorithm or technique;
Test for function on a small-scale deployment, likely in an interactive (ipython) environment;
Re-code the algorithm using an asynchronous style; and
Integration testing, optimization, and deployment.

This approach has advantages:

When developing a proof-of-concept, developers are able to ignore performance and focus on the problem (physics, instrumentation, cryogenics, electronics) that they are attempting to address.
During prototyping, when interactive exploration is most useful, synchronous code promotes use of environments such as IPython or Jupyter.

However, this workflow has three major disadvantages:

It's clumsy: it requires writing and testing a synchronous version, then shifting it wholesale to an asynchronous environment. It is easy to imagine this workflow looping back on itself as bugs are discovered or introduced along the way.
The synchronous version never stops being useful, despite not scaling to telescope-level performance. We would often much rather have the simpler semantics, more predictable control flow, and shorter error traces associated with a synchronous call when debugging or experimenting. In addition, it can be conveniently invoked in a REPL environment -- invaluable if the telescope is operating and we need to do some quick hand-tuning.
It's not composable. Over the years, we have build up libraries of useful tuning and control algorithms, and as long as synchronous and asynchronous code is kept distinct, we cannot meaningfully compose algorithms out of smaller pieces without two implementations of everything.

Asking developers to maintaining two versions under different coding idioms (and expecting to keep the versions synchronized) is resolving a technical flaw by requiring skilled labourers to do menial work; this is often an expensive mistake. (Interactive use of asynchronous code is getting easier in IPython 7.0 due to the autoawait functionality. This extension addresses the second but not the third point.)

Instead, we are looking for a way to freely mix asynchronous and synchronous coding styles.

Enter the @tworoutine

What's a @tworoutine? It is a synchronous wrapper around an asynchronous function, allowing a single piece of code to be called in either idiom.

(If you are following along at home, you will need the source code. You will also need nest_asyncio.)

import tworoutine
import asyncio

@tworoutine.tworoutine
async def double_slowly(x):
    await asyncio.sleep(0.1)
    return 2*x

How can we call this function synchronously? Just call it!

>>> double_slowly(1)
2

How did this work? The @tworoutine decorator returns a class whose __call__ method is a synchronous wrapper that obtains an event loop and invokes the asynchronous code, blocking until it's complete. Because we want synchronous calling to be convenient and carpal-tunnel-friendly, that's the default.

If there's already an event loop running, this code is reasonably efficient (aside from being a blocking call, of course!) Any asynchronous events already queued in the event loop are allowed to proceed alongside this one. Only the current execution context is blocked until the coroutine completes.

So much for synchronous calls. How can we call this function asynchronously? We first have to undo or "invert" the wrapper and obtain a reference back to the coroutine.

>>> (~double_slowly)(2)
<coroutine object double_slowly at 0x7f5d494fd348>

With the exception of the invert operator around the function name, this is ordinary asynchronous code; there is no additional overhead except for the operator itself. Here is a complete example showing mixed coding styles within an event loop:

async def main():
    # Run asynchronously
    r = await (~double_slowly)(2)
    print(r)

    # Run synchronously within an event loop
    r2 = double_slowly(3)
    print(r2)

# try asynchronous entry
asyncio.run(main())

The obvious benefit, here, is the ability to call asynchronous code synchronously when we're too lazy to carry around an event loop or deal with the turtles-all-the-way-down nature of Python's asynchronous coding idiom.

Design flow with @tworoutine. The synchronous and asynchronous implementations are replaced with a single implementation that can mix idioms.

Lament: Of Course There's A Catch

@tworoutine's days are probably numbered. This style of coding has been implicitly but firmly rejected by Python developers:

Issue 22239: asyncio: nested event loop

We have been using this approach (implemented on Python 2.7 and Tornado <4.5) for several years now at the South Pole and elsewhere, and we will have to adapt.

To complete a synchronous @tworoutine call, we need to obtain an event loop, schedule the asynchronous (decorated) call, and block until it is complete. Currently there is no way to do that in Python 3.7 asyncio without patching it. Asynchronous code at any point in the call stack must be linked to the event loop via asynchronous calls only, all the way up.

To work around this problem in the Python 3.7 code shown here, I have used the nest_asyncio monkey patch. It is a short and effective piece of code, but it runs against Python orthodoxy and adopting this kind of patch in production risks being stranded by changes to Python's core libraries.

Without this patch, we are able to upgrade as far as Tornado 4.5 on Python 3.x, but Tornado 5.0 moves to an asyncio event loop and we are suddenly unable to upgrade.

Disclaimer

The code examples here have been forward-ported from Python 2.7 and Tornado 4.5 to Python 3.7 and "pure" asyncio. It's an experiment -- this is not production code!

Exploring the Orbital Radiation Environment with Python

2018-06-24T00:00:00-07:00

This page describes how to retrieve proton-detector data from satellite archives provided by the US National Oceanic and Atmospheric Administration (NOAA). By applying some Python code to this data, we will plot measurements of the radiation environment captured by satellites in geosynchronous orbit.

Table of Contents

The GOES Satellites
The Problem with Protons
Getting Started
Quiet Conditions
Solar Storms
Conclusions

The GOES Satellites

First, a little back story. In this material, I lean heavily on the Wikipedia pages on the GOES satellites — they are a treasure trove and I thoroughly recommend you go wandering in Wikipedia-land.

For over 40 years, the US National Oceanic and Atmospheric Administration (NOAA) has operated a series of spacecraft called the Geostationary Operational Environmental Satellites (GOES). Starting with GOES-1 in 1975 (manufactured by Ford Aerospace — yes, that Ford), most of these satellites were successfully launched and served for many years, and have since been decomissioned.

The GOES family of satellites is still going strong: GOES-15 through GOES-17 are in orbit and either functioning, or standing by in case a repalacement is needed. Imagery from GOES-17 — launched just a few months ago — is already available on NASA's web site. Additional GOES launches are scheduled in 2020 and 2024.

Satellite technology has changed a lot in since 1975, and consequently, the instruments on these satellites vary quite a bit. Different GOES satellites have included:

Proton, alpha-particle, X-ray, and electron flux detectors,
Earth-facing optical, infrared, and near-infrared cameras,
Solar imagers,
Magnetometers
Lightning detectors, and
Radio equipment for communicating with the Earth.

Since I am specifically interested in the proton detectors — data that's readily available in a standard format from 1986 onwards — this is the data we will investigate. First, though, why protons?

The Problem with Protons

Radiation is bad for electronics. Energetic protons are a big component of the radiation mixture in geosynchronous orbits.

Electronics in a radioactive environment generally exhibit two kinds of effects caused by radiation:

Long-term effects (Jargon: "Total Ionizing Dose" or "TID"), in which radiation damage accumulates over time and leads to degradation and the eventual and permanent failure of spacecraft electronics, and
Short-term effects (Jargon: Single-Event Effects, Single-Event Upsets, Single-Event Errors) caused by individual interactions between radiation and spacecraft systems. These effects may flip a bit from its correct value or generate a small transient signal that shouldn't be present.

These radiation effects occur on the ground, too, but the Earth's atmosphere absorbs most of the radiation that would otherwise scramble our electronics. On the ground, we mostly (but not always!) ignore the remaining errors. If your computer has ever inexplicably rebooted, the problem may have been cosmic radiation.

Back to space. These effects are not only caused by protons, but especially inside the spacecraft's shell where there is some shielding present, protons are a good enough proxy for radiation levels that we can focus on just one type of particle for now. (This is an useful simplification and a great way to get experts — I am not one of them — to start arguing.)

During solar storms, the arrival rates (fluxes) of protons are orders of magnitude higher than background conditions. The result? For long-term damage (TID), the quiet conditions between solar storms don't accumulate enough damage to matter at all.

For short-term damage (SEE), errors do occur during quiet conditions, and engineers do need to worry about their effects. However, the design of spacecraft electronics must accommodate (and is typically driven by) the worst conditions. For critical spacecraft systems (such as propulsion), survival is not enough: these systems are sometimes required to operate without error during during radiation events in order to maintain spacecraft function.

The frequency, source, intensity, and particle mixtures associated with solar radiation events are another good opportunity to fall down the Wikipedia hole. However, it's time to come back to the premise of this article and add a little Python code.

Getting Started

OK, let's get our hands dirty. The following Python code does not have terribly esoteric requirements, but we do require the following:

Python 3.5 or newer (I use Debian's stock 3.6.6rc1 build),
Asynchronous packages (asyncio, aiohttp)
Scientific packages (numpy, scipy, matplotlib)

You can download the source code from Bitbucket as follows:

$ git clone git@bitbucket.org:gsmecher/goes-proton-python.git
$ cd goes-proton-python

Because we make use of a large dataset, the processing pipeline is split into two stages:

A script (fetch.py) that fetches GOES data from NOAA's archive into a local on-disk cache, and
A script (analyze.py) that processes cached data (in bulk) and produces monthly radiation plots.

The first script is an attempt to be polite to the NOAA servers: we can (and do) still download a large dataset fairly quickly, but at least we can avoid downloading it more than once.

The second script does a batch analysis (and is aggressively parallel — it's your PC, so we have no need to be polite here.)

The Fetch Script

(Source: fetch.py.)

Simply run

$ ./fetch.py

It has been a few years since I have developed Python code in earnest, and back then, the codebase I maintained was firmly in Python 2.6 territory. We made use of the wonderful asynchronous event loop provided by the Tornado project, and while watching the development of Python's batteries-included asynchronous framework I have often wished I could pull some of my older code into the more modern style. Bulk data fetch over HTTP was a great excuse to enter the brave world of Python 3.5+ asynchronous coding.

When you execute this script, it will create and populate a _cache directory. The default options fetch only a single year's worth of data for a single satellite — if you want to download a larger dataset, you will have to say so explicitly.

Parsing GOES CSV

(Source: goes.py.)

There are a number of ways of analyzing GOES data, including using the open-source SPEDAS <http://spedas.org>_ software (which I have not tried). If you are doing serious work, you are almost certainly better off not reinventing the wheel. Now, let's re-invent the wheel.

The GOES dataset is large, and because we're talking about decades of history and over a dozen satellites, it is natural to expect tension between the ideal data format and the compatible data format. The data we use here is formatted idiosyncraticically but not more than one would expect, given its age and scope.

GOES CSV is not strict CSV: it is CSV with embedded sub-tables, annotations, and metadata. I have not attempted to write a general-purpose parser for the overarching data format; rather, I have picked through the pieces of the file I need and created a degenerate parser that only understands what it needs to.

This code extends Python's DictReader with some rudimentary variable type conversion and teaches it to extract (and skip) the metadata blocks at the top of the CSV file.

The Analysis Script

(Source: analyze.py.)

Now that we can retrieve and parse GOES CSV data, we are ready to produce graphics from it. The analysis script leans heavily on matplotlib, which does not like threaded code. Hence, in order to generate a large volume of independent plots, I used the multiprocessing module instead.

By default, this code will search the entire _cache produced by the fetch.py script (above) and process everything in it, as quickly as it can. For a large GOES cache, this can still take quite a while. The resulting images (one per month per satellite) are placed in a png/ directory, named per the original CSV data.

Let's take a look. Simply run:

$ ./analyze.py

This command produces a bunch of plots in the png/ directory.

Quiet Conditions

January 1989 (corrected proton channels)

This is an ordinary quiet day during an active sunspot cycle (1989 was near the peak of solar cycle 22). Before we proceed to a more spectacular example, let's take a look at this plot and figure out what information it contains.

This is a plot of proton intensity versus time, over the entire month of January 1989. Using different colours, we show superimposed plots for a variety of different sensors, each of which is sensitive to a range of proton kinetic energies. The vertical axis indicates proton flux, i.e. the number of particles seen by the detector at a given time. Apart from a normalization against proton kinetic energy, the units are logical. As you might expect, protons with low kinetic energies (e.g. the blue curve) are present at higher amounts than protons with higher energy levels (e.g. the magenta curve.) Although the detectors are sensitive to different ranges of proton energies, and hence the underlying measurements are probably incommensurate between sensors, the data has been normalized against these ranges to make comparisons between different sensors "fair". (Hence the "MeV" in the denominator in the y-axis units.)

In the blue trace, the satellite's orbital cycle in and out of the earth's shadow is clearly visible. The GOES-6 satellite orbits relatively close to the equator (14.7 degrees inclination) so it flies in and out of the Earth's shadow once per day. (By definition, geosynchronous satellites have a day that's exactly as long as ours. This is decidedly different from low-earth orbits, where a satellite might orbit over a dozen times per day!)

There are a few events visible on this day, particularly visible in the orange band. However, the really high-energy proton detectors do not really see anything except noise.

March 1989 was a different story, as we will now see.

Solar Storms

In March 1989, a coronal mass ejection launched an intense volley of radiation at the earth. The resulting effects were substantial, even on the ground: Wikipedia claims a large power outage in Quebec, and a panic among Cold War spooks fearing a nuclear first strike.

March 1989 (corrected proton channels)

Since the Y axis is in logarithmic units, and since these plots are normalized to wash away the effects of protons' kinetic energies, this plot understates just how bad the radiation event in March 1989 was. (In fact, data is missing for what is likely the worst portion of the event!)

Conclusions

...and with those two plots, we have successfully downloaded, analyzed, and plotted proton data from a geosynchronous satellite using Python — just as promised. I will leave off with a couple of interesting links:

The Carrington Event was a severe geomagnetic storm occurring in 1859 — a long time ago, but recent enough history to have a lively written record of the event's effects.
While decomissioning a low-earth orbit generally involves dropping it into the ocean, geosynchronous satellites are typically moved into a Graveyard orbit — far enough beyond the 30,000 km geostationary orbit to stay out of trouble.

ICEboard: an FPGA / ARM Motherboard

2018-06-06T13:36:55-07:00

The ICEboard is a fully custom FPGA / ARM motherboard hosting two high-pin-count (HPC) FPGA Mezzanine (FMC) slots.

The ICEboard, revision 2. The two red mezzanines host high-speed (gigasample per second) ADCs for the CHIME experiment. The motherboard hosts an FPGA and ARM for processing elements, and a variety of high-speed interfaces.

Three-Speed Logic collaborated with McGill's Winterland cosmology lab to design the ICEboard. This board includes:

A Kintex-7 '420T FPGA (socket-compatible with '325T and '480T variants)
A TI AM3874 ARM SoC
A variety of high-speed serial interfaces:
- Two 10/100/100 Ethernet interfaces
- 28 10-Gigabit GTX links
On-chip storage resources
- SPI flash for bitstream and software storage
- DDR3 SDRAM

The ICEboard occupies a 9u-height VME slot (340mm high), although cards are not as deep as a full 9u rack (233mm deep, compared to standard 360mm deep). Most VME crates are adjustable to accomodate this variation; crate vendors are also able to ship correctly sized crates.

Another view of the ICEboard, without FMC daughtercards. The RF shield surrounding the ARM CPU has been removed, exposing two DDR3 ICs and two Ethernet PHYs. The attached programming header is used to configure the board's two PLLs, which are used to generate system clocks for the CPU and FPGA.