Mind Chasers Inc.
Mind Chasers Inc.

Use FPGA Open Source Toolchain with Private Island and Lattice ECP5UM

Working towards a port of our open source networking project for the Lattice ECP5UM with Yosys on Ubuntu

Overview

We're just getting started with open source tools for FPGAs with an eye towards porting our Private Island™ secure networking project to the Yosys open source toolchain. We're going to document the high-level progress & pitfalls on this page as we proceed with Yosys and related tools. For the time being, please be patient with rough edges and missing information. However, if you can help clarify issues we have or point out mistakes, please share your insight by posting at the bottom of this page.

There is no doubt that the Yosys and related tools have generated a lot of excitement within the open source FPGA community. It's our perception that the drivers behind the excitement are:

  • FPGAs provide developers with a blank canvas to work with in developing amazing systems, so it's only natural that developers would want a toolchain that holds the promise of unlimited flexibility and customization.
  • Some FPGAs require a paid license, such as the Lattice ECP5UM that we use on our Darsena board. If you're reading this article, then you're also probably using tools like GCC and Python. It's only natural for open source minded people to want an open source toolchain for their FPGA.
  • And for us, we hope to see at some point an open FPGA that fully supports an open source toolchain flow. We want a sea of programmable gates with generic high-speed DDR I/O and lots of memory from which we can build custom solutions for networking that are impractical / near impossible to inject backdoors within the silicon or tools.

Some of our core questions:

  • Is it practical to think that an open source toolchain could produce more compact or higher frequency implementations?
  • Is it practical to customize an existing open source toolchain to solve issues that a commercial toolchain can't or won't?
  • Can open source tools make a network design more secure?
  • Does it make sense to use open source tools with a design that will go to production when the manufacturer doesn't support the toolchain?

FPGA Open Source Toolchain Flow

The figure below is our preliminary view of the Yosys flow compared to the Dimaond flow that we use everyday. For the record, we think Diamond is excellent. It's easy to use / setup, and the outputs / reports are clear. It's also free for most devices. Unfortunately, for the ECP5UM on Darsena, a paid license is required. However, the work flow and features are identical between the paid and free versions of Diamond. For Intel Quartus Prime, the paid versions offer more features than the free (lite) version, and they appear to be a generation ahead (19 vs 18).

tool flow
Figure 1. FPGA Toolchain Flow

Define a Simple Test Project

Our first set of goals includes building the toolchain from source and working through creating a bitstream for a trivial project.

Our trivial Lattice Diamond ECP5UM project that we wish to port is shown below and consists of 1) a lightweight top module that toggles an LED about once a second and 2) a preference file that defines two I/O.

led_tst.v
module led_tst(
    input rstn,
    output led
);

    wire clk;
    reg [20:0] cnt;
    
    GSR GSR_INST(.GSR(rstn));
    PUR PUR_INST(.PUR(1'b1));
    OSCG oscg_inst(.OSC(clk));		// internal oscillator 
    defparam oscg_inst.DIV = 32;	// ~10 MHz
    
    always @(posedge clk, negedge rstn)begin
        if (!rstn)
            cnt <= 0;
        else
            cnt <= cnt+1;
    end
    
    assign led = ~cnt[20];
            
endmodule
led_tst.lpf
LOCATE COMP "led" SITE "P20" ;
IOBUF PORT "led" IO_TYPE=LVCMOS33;
LOCATE COMP "rstn" SITE "F1";
IOBUF PORT "rstn" IO_TYPE=LVCMOS33;           

Yosys

The Yosys Open SYnthesis Suite (YOSYS) is a framework for Verilog RTL synthesis. The source can be found on Github, and there is a fairly active subreddit for it.

From the Yosys Manual: " A Hardware Description Language (HDL) is a computer language used to describe circuits. A HDL synthesis tool is a computer program that takes a formal description of a circuit written in an HDL as input and generates a netlist that implements the given circuit as output."

Building Yosys from source:

$ cd build
$ git clone https://github.com/YosysHQ/yosys.git
$ cd yosys

$ ls
backends  CHANGELOG      CodingReadme  Dockerfile  frontends  libs      manual  passes     techlibs
Brewfile  CodeOfConduct  COPYING       examples    kernel     Makefile  misc    README.md  tests

$ $ git log
commit 3414ee1e3fe37d4bf383621542828d4fc8fc987f (HEAD -> master, origin/master, origin/HEAD)
Merge: 5545cd3c 58e512ab
Author: Eddie Hung 
Date:   Wed Aug 7 12:25:26 2019 -0700

    Merge pull request #1248 from YosysHQ/eddie/abc9_speedup
    
    abc9: speedup by using using "clean" more efficiently
    
$ mkdir build; cd build
    
$ make -f ../Makefile config-gcc
...
echo 'CONFIG := gcc' > Makefile.conf

$ ls
Makefile.conf

$ make -f ../Makefile 
[Makefile.conf] CONFIG := gcc
[  0%] Building kernel/version_ac2fc3a1.cc
[  0%] Building kernel/version_ac2fc3a1.o
...
[100%] Building share/gowin/brams_init3.vh
[100%] Building share/sf2/arith_map.v
[100%] Building share/sf2/cells_map.v
[100%] Building share/sf2/cells_sim.v

  Build successful.
  
 $ ls
abc       frontends  libs           passes  techlibs  yosys-abc     yosys-filterlib
backends  kernel     Makefile.conf  share   yosys     yosys-config  yosys-smtbmc

$ make -f ../Makefile install PREFIX=/opt/yosys/

$ cd /opt/yosys

$ tree -d -L 4
.
├── bin
├── share
│   └── yosys
│       ├── achronix
│       │   └── speedster22i
│       ├── anlogic
│       ├── coolrunner2
│       ├── ecp5
│       ├── gowin
│       ├── greenpak4
│       ├── ice40
│       ├── include
│       │   ├── backends
│       │   ├── frontends
│       │   ├── kernel
│       │   ├── libs
│       │   └── passes
│       ├── intel
│       │   ├── a10gx
│       │   ├── common
│       │   ├── cyclone10
│       │   ├── cycloneiv
│       │   ├── cycloneive
│       │   ├── cyclonev
│       │   └── max10
│       ├── python3
│       │   └── __pycache__
│       ├── sf2
│       └── xilinx
└── tsts


        
$ export PATH=/opt/yosys/bin:$PATH


$ yosys
 ...
 Yosys 0.8+634 (git sha1 ac2fc3a1, gcc 7.4.0-1ubuntu1~18.04.1 -fPIC -Os)
 
yosys> read -vlog2k /opt/yosys/tsts/led_tst.v 
1. Executing Verilog-2005 frontend: /opt/yosys/tsts/led_tst.v
...

yosys> hierarchy -top led_tst
2. Executing HIERARCHY pass (managing design hierarchy).
...

yosys> proc; opt; techmap; opt
4. Executing PROC pass (convert processes to netlists).
...

yosys> write_verilog /opt/yosys/tsts/synth.v
8. Executing Verilog backend.
Dumping module `\led_tst'.
synth.v
/* Generated by Yosys 0.8+634 (git sha1 ac2fc3a1, gcc 7.4.0-1ubuntu1~18.04.1 -fPIC -Os) */

(* top =  1  *)
(* src = "/opt/yosys/tsts/led_tst.v:1" *)
module led_tst(rstn, led);
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  wire [20:0] _00_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:203" *)
  (* unused_bits = "20" *)
  wire [31:0] _01_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _02_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _03_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _04_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _05_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _06_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _07_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _08_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _09_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _10_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _11_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _12_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _13_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _14_;
  (* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)
  wire _15_;
  (* src = "/opt/yosys/tsts/led_tst.v:6" *)
  wire clk;
  (* src = "/opt/yosys/tsts/led_tst.v:7" *)
  reg [20:0] cnt;
  (* src = "/opt/yosys/tsts/led_tst.v:3" *)
  output led;
  (* src = "/opt/yosys/tsts/led_tst.v:2" *)
  input rstn;
  assign led = ~(* src = "/opt/yosys/tsts/led_tst.v:21" *) cnt[20];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[0] <= 0;
    else
      cnt[0] <= _00_[0];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[1] <= 0;
    else
      cnt[1] <= _00_[1];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[2] <= 0;
    else
      cnt[2] <= _00_[2];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[3] <= 0;
    else
      cnt[3] <= _00_[3];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[4] <= 0;
    else
      cnt[4] <= _00_[4];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[5] <= 0;
    else
      cnt[5] <= _00_[5];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[6] <= 0;
    else
      cnt[6] <= _00_[6];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[7] <= 0;
    else
      cnt[7] <= _00_[7];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[8] <= 0;
    else
      cnt[8] <= _00_[8];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[9] <= 0;
    else
      cnt[9] <= _00_[9];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[10] <= 0;
    else
      cnt[10] <= _00_[10];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[11] <= 0;
    else
      cnt[11] <= _00_[11];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[12] <= 0;
    else
      cnt[12] <= _00_[12];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[13] <= 0;
    else
      cnt[13] <= _00_[13];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[14] <= 0;
    else
      cnt[14] <= _00_[14];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[15] <= 0;
    else
      cnt[15] <= _00_[15];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[16] <= 0;
    else
      cnt[16] <= _00_[16];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[17] <= 0;
    else
      cnt[17] <= _00_[17];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[18] <= 0;
    else
      cnt[18] <= _00_[18];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[19] <= 0;
    else
      cnt[19] <= _00_[19];
  (* src = "/opt/yosys/tsts/led_tst.v:14" *)
  always @(posedge clk or negedge rstn)
    if (!rstn)
      cnt[20] <= 0;
    else
      cnt[20] <= _00_[20];
  assign _00_[1] = cnt[1] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  cnt[0];
  assign _00_[2] = cnt[2] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[1];
  assign _00_[3] = cnt[3] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[2];
  assign _00_[4] = cnt[4] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[3];
  assign _00_[5] = cnt[5] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[4];
  assign _00_[6] = cnt[6] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[5];
  assign _00_[7] = cnt[7] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[6];
  assign _00_[8] = cnt[8] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[7];
  assign _00_[9] = cnt[9] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[8];
  assign _00_[10] = cnt[10] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[9];
  assign _00_[11] = cnt[11] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[10];
  assign _00_[12] = cnt[12] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[11];
  assign _00_[13] = cnt[13] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[12];
  assign _00_[14] = cnt[14] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[13];
  assign _00_[15] = cnt[15] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[14];
  assign _00_[16] = cnt[16] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[15];
  assign _00_[17] = cnt[17] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[16];
  assign _00_[18] = cnt[18] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[17];
  assign _00_[19] = cnt[19] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[18];
  assign _00_[20] = cnt[20] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:263" *)  _01_[19];
  assign _00_[0] = cnt[0] ^(* src = "/opt/yosys/tsts/led_tst.v:18|:262" *)  1'h1;
  assign _01_[1] = cnt[1] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:221" *)  cnt[0];
  assign _01_[3] = _02_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:221" *)  _01_[1];
  assign _01_[7] = _11_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:221" *)  _01_[3];
  assign _01_[15] = _15_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:221" *)  _01_[7];
  assign _02_ = cnt[3] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[2];
  assign _03_ = cnt[5] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[4];
  assign _04_ = cnt[7] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[6];
  assign _05_ = cnt[9] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[8];
  assign _06_ = cnt[11] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[10];
  assign _07_ = cnt[13] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[12];
  assign _08_ = cnt[15] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[14];
  assign _09_ = cnt[17] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[16];
  assign _10_ = cnt[19] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  cnt[18];
  assign _11_ = _04_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  _03_;
  assign _12_ = _06_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  _05_;
  assign _13_ = _08_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  _07_;
  assign _14_ = _10_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  _09_;
  assign _15_ = _13_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:222" *)  _12_;
  assign _01_[11] = _12_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[7];
  assign _01_[19] = _14_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[15];
  assign _01_[5] = _03_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[3];
  assign _01_[9] = _05_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[7];
  assign _01_[13] = _07_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[11];
  assign _01_[17] = _09_ &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[15];
  assign _01_[2] = cnt[2] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[1];
  assign _01_[4] = cnt[4] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[3];
  assign _01_[6] = cnt[6] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[5];
  assign _01_[8] = cnt[8] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[7];
  assign _01_[10] = cnt[10] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[9];
  assign _01_[12] = cnt[12] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[11];
  assign _01_[14] = cnt[14] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[13];
  assign _01_[16] = cnt[16] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[15];
  assign _01_[18] = cnt[18] &(* src = "/opt/yosys/tsts/led_tst.v:18|:260|:229" *)  _01_[17];
  (* module_not_derived = 32'd1 *)
  (* src = "/opt/yosys/tsts/led_tst.v:9" *)
  GSR GSR_INST (
    .GSR(rstn)
  );
  (* module_not_derived = 32'd1 *)
  (* src = "/opt/yosys/tsts/led_tst.v:11" *)
  OSCG #(
    .DIV(32'sd32)
  ) oscg_inst (
    .OSC(clk)
  );
  assign { _01_[31:21], _01_[0] } = { 11'h000, cnt[0] };
endmodule

For reference, we show the output of Synplify for the same led_tst module. It's immediately obvious that Synplify has performed a technology mapping while the YoSys output still needs to be mapped.

led_tst1_impl1.vm
//
// Written by Synplify Pro 
// Product Version "N-2018.03L-SP1-1"
// Program "Synplify Pro", Mapper "maplat2018q2p1, Build 055R"
// Sat Aug 10 11:50:44 2019
//
// Source file index table:
// Object locations will have the form :
// file 0 "\c:\lscc\diamond\3.11_x64\synpbase\lib\lucent\ecp5um.v "
// file 1 "\c:\lscc\diamond\3.11_x64\synpbase\lib\lucent\pmi_def.v "
// file 2 "\c:\lscc\diamond\3.11_x64\synpbase\lib\vlog\hypermods.v "
// file 3 "\c:\lscc\diamond\3.11_x64\synpbase\lib\vlog\umr_capim.v "
// file 4 "\c:\lscc\diamond\3.11_x64\synpbase\lib\vlog\scemi_objects.v "
// file 5 "\c:\lscc\diamond\3.11_x64\synpbase\lib\vlog\scemi_pipes.svh "
// file 6 "\c:\projects\lattice\yosys\led_tst1\led_tst.v "
// file 7 "\c:\lscc\diamond\3.11_x64\synpbase\lib\nlconst.dat "

`timescale 100 ps/100 ps
module led_tst (
  rstn,
  led
)
;
input rstn ;
output led ;
wire rstn ;
wire led ;
wire [20:0] cnt;
wire [18:0] cnt_cry;
wire [20:0] cnt_s;
wire [0:0] cnt_cry_0_S0;
wire [19:19] cnt_cry_0_COUT;
wire [20:0] cnt_QN;
wire clk ;
wire GND ;
wire VCC ;
wire rstn_i ;
wire N_1 ;
  VLO GND_0 (
	.Z(GND)
);
  VHI VCC_0 (
	.Z(VCC)
);
  INV rstn_RNI7UU (
	.A(rstn),
	.Z(rstn_i)
);
  INV \cnt_RNIVVJ5[20]  (
	.A(cnt[20]),
	.Z(led)
);
// @6:15
  FD1S3DX \cnt_reg[0]  (
	.D(cnt_s[0]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[0])
);
// @6:15
  FD1S3DX \cnt_reg[1]  (
	.D(cnt_s[1]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[1])
);
// @6:15
  FD1S3DX \cnt_reg[2]  (
	.D(cnt_s[2]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[2])
);
// @6:15
  FD1S3DX \cnt_reg[3]  (
	.D(cnt_s[3]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[3])
);
// @6:15
  FD1S3DX \cnt_reg[4]  (
	.D(cnt_s[4]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[4])
);
// @6:15
  FD1S3DX \cnt_reg[5]  (
	.D(cnt_s[5]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[5])
);
// @6:15
  FD1S3DX \cnt_reg[6]  (
	.D(cnt_s[6]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[6])
);
// @6:15
  FD1S3DX \cnt_reg[7]  (
	.D(cnt_s[7]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[7])
);
// @6:15
  FD1S3DX \cnt_reg[8]  (
	.D(cnt_s[8]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[8])
);
// @6:15
  FD1S3DX \cnt_reg[9]  (
	.D(cnt_s[9]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[9])
);
// @6:15
  FD1S3DX \cnt_reg[10]  (
	.D(cnt_s[10]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[10])
);
// @6:15
  FD1S3DX \cnt_reg[11]  (
	.D(cnt_s[11]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[11])
);
// @6:15
  FD1S3DX \cnt_reg[12]  (
	.D(cnt_s[12]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[12])
);
// @6:15
  FD1S3DX \cnt_reg[13]  (
	.D(cnt_s[13]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[13])
);
// @6:15
  FD1S3DX \cnt_reg[14]  (
	.D(cnt_s[14]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[14])
);
// @6:15
  FD1S3DX \cnt_reg[15]  (
	.D(cnt_s[15]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[15])
);
// @6:15
  FD1S3DX \cnt_reg[16]  (
	.D(cnt_s[16]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[16])
);
// @6:15
  FD1S3DX \cnt_reg[17]  (
	.D(cnt_s[17]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[17])
);
// @6:15
  FD1S3DX \cnt_reg[18]  (
	.D(cnt_s[18]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[18])
);
// @6:15
  FD1S3DX \cnt_reg[19]  (
	.D(cnt_s[19]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[19])
);
// @6:15
  FD1S3DX \cnt_reg[20]  (
	.D(cnt_s[20]),
	.CK(clk),
	.CD(rstn_i),
	.Q(cnt[20])
);
  CCU2C \cnt_cry_0[0]  (
	.A0(VCC),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[0]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(N_1),
	.COUT(cnt_cry[0]),
	.S0(cnt_cry_0_S0[0]),
	.S1(cnt_s[0])
);
defparam \cnt_cry_0[0] .INIT0=16'h500c;
defparam \cnt_cry_0[0] .INIT1=16'ha003;
defparam \cnt_cry_0[0] .INJECT1_0="NO";
defparam \cnt_cry_0[0] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[1]  (
	.A0(cnt[1]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[2]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[0]),
	.COUT(cnt_cry[2]),
	.S0(cnt_s[1]),
	.S1(cnt_s[2])
);
defparam \cnt_cry_0[1] .INIT0=16'ha003;
defparam \cnt_cry_0[1] .INIT1=16'ha003;
defparam \cnt_cry_0[1] .INJECT1_0="NO";
defparam \cnt_cry_0[1] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[3]  (
	.A0(cnt[3]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[4]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[2]),
	.COUT(cnt_cry[4]),
	.S0(cnt_s[3]),
	.S1(cnt_s[4])
);
defparam \cnt_cry_0[3] .INIT0=16'ha003;
defparam \cnt_cry_0[3] .INIT1=16'ha003;
defparam \cnt_cry_0[3] .INJECT1_0="NO";
defparam \cnt_cry_0[3] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[5]  (
	.A0(cnt[5]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[6]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[4]),
	.COUT(cnt_cry[6]),
	.S0(cnt_s[5]),
	.S1(cnt_s[6])
);
defparam \cnt_cry_0[5] .INIT0=16'ha003;
defparam \cnt_cry_0[5] .INIT1=16'ha003;
defparam \cnt_cry_0[5] .INJECT1_0="NO";
defparam \cnt_cry_0[5] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[7]  (
	.A0(cnt[7]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[8]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[6]),
	.COUT(cnt_cry[8]),
	.S0(cnt_s[7]),
	.S1(cnt_s[8])
);
defparam \cnt_cry_0[7] .INIT0=16'ha003;
defparam \cnt_cry_0[7] .INIT1=16'ha003;
defparam \cnt_cry_0[7] .INJECT1_0="NO";
defparam \cnt_cry_0[7] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[9]  (
	.A0(cnt[9]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[10]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[8]),
	.COUT(cnt_cry[10]),
	.S0(cnt_s[9]),
	.S1(cnt_s[10])
);
defparam \cnt_cry_0[9] .INIT0=16'ha003;
defparam \cnt_cry_0[9] .INIT1=16'ha003;
defparam \cnt_cry_0[9] .INJECT1_0="NO";
defparam \cnt_cry_0[9] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[11]  (
	.A0(cnt[11]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[12]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[10]),
	.COUT(cnt_cry[12]),
	.S0(cnt_s[11]),
	.S1(cnt_s[12])
);
defparam \cnt_cry_0[11] .INIT0=16'ha003;
defparam \cnt_cry_0[11] .INIT1=16'ha003;
defparam \cnt_cry_0[11] .INJECT1_0="NO";
defparam \cnt_cry_0[11] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[13]  (
	.A0(cnt[13]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[14]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[12]),
	.COUT(cnt_cry[14]),
	.S0(cnt_s[13]),
	.S1(cnt_s[14])
);
defparam \cnt_cry_0[13] .INIT0=16'ha003;
defparam \cnt_cry_0[13] .INIT1=16'ha003;
defparam \cnt_cry_0[13] .INJECT1_0="NO";
defparam \cnt_cry_0[13] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[15]  (
	.A0(cnt[15]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[16]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[14]),
	.COUT(cnt_cry[16]),
	.S0(cnt_s[15]),
	.S1(cnt_s[16])
);
defparam \cnt_cry_0[15] .INIT0=16'ha003;
defparam \cnt_cry_0[15] .INIT1=16'ha003;
defparam \cnt_cry_0[15] .INJECT1_0="NO";
defparam \cnt_cry_0[15] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[17]  (
	.A0(cnt[17]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[18]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[16]),
	.COUT(cnt_cry[18]),
	.S0(cnt_s[17]),
	.S1(cnt_s[18])
);
defparam \cnt_cry_0[17] .INIT0=16'ha003;
defparam \cnt_cry_0[17] .INIT1=16'ha003;
defparam \cnt_cry_0[17] .INJECT1_0="NO";
defparam \cnt_cry_0[17] .INJECT1_1="NO";
// @6:15
  CCU2C \cnt_cry_0[19]  (
	.A0(cnt[19]),
	.B0(VCC),
	.C0(VCC),
	.D0(VCC),
	.A1(cnt[20]),
	.B1(VCC),
	.C1(VCC),
	.D1(VCC),
	.CIN(cnt_cry[18]),
	.COUT(cnt_cry_0_COUT[19]),
	.S0(cnt_s[19]),
	.S1(cnt_s[20])
);
defparam \cnt_cry_0[19] .INIT0=16'ha003;
defparam \cnt_cry_0[19] .INIT1=16'ha00a;
defparam \cnt_cry_0[19] .INJECT1_0="NO";
defparam \cnt_cry_0[19] .INJECT1_1="NO";
// @6:10
  GSR GSR_INST (
	.GSR(rstn)
);
// @6:11
  PUR PUR_INST (
	.PUR(VCC)
);
// @6:12
(* DIV=32 *)  OSCG oscg_inst (
	.OSC(clk)
);
defparam oscg_inst.DIV=32;
endmodule /* led_tst */


It appears that Berkeley's ABC tool is used for technology mapping??? We also find that during the make of Yosys, ABC is cloned, built, and copied to /opt/yosys/bin as yosys-abc. It's not clear if we need this for FPGA synthesis.

From Berkeley's web site: "ABC is a growing software system for synthesis and verification of binary sequential logic circuits appearing in synchronous hardware designs. ABC combines scalable logic optimization based on And-Inverter Graphs (AIGs), optimal-delay DAG-based technology mapping for look-up tables and standard cells, and innovative algorithms for sequential synthesis and verification." We'll also review the ABC tutorial.

We also want to understand the use of the cell files in techlibs/ecp5. We can see our clock primitive OSCG in cells_bb.v, and we see the FD1S3DX flop that Synopsys used above in both cells_map.v and cells_sim.v. And at some point, we're going to need the DCUA primitive to connect our Ethernet PHYs, and we see this in cells_bb.v. The bb file suffix indicates a black box. The map suffix indicates that the Lattice primitive is being mapped. In the case of the FD1S3DX, it's being mapped to a TRELLIS_FF

Project Trellis

Project Trellis provides the device database and tools for ECP5 bitstream creation. We're very curious to discover more about how they handle the ECP5UM with it's PCS/SERDES primitive.

Note that we're following the directions on Github that state to build Trellis before NextPnR. Also note that it took us several tries to get trellis to build since we kept getting errors dealing with old python libraries being picked up. We see some of these issues on Github (e.g., not finding the Python interpreter). In general we resolved our build errors by removing old python libraries and binaries manually.

$ git clone --recursive https://github.com/SymbiFlow/prjtrellis
...
Submodule path 'database': checked out 'd0b219af41ae3da6150645fbc5cc5613b530603f'

$ cd prjtrellis

$ git log
commit 2dd0e0ab3841228b48f7c907cb4ed90dc135fe23 (HEAD -> master, origin/master, origin/HEAD)
Author: David Shah 
Date:   Thu Aug 8 21:38:28 2019 +0100

    examples: Preparing from removal of default --package in nextpnr
...

$ ls
CODE_OF_CONDUCT.md  create-empty-db.sh  diamond.sh      environment.sh  fuzzers     minitests  third_party  util
CONTRIBUTING.md     database            diamond_tcl.sh  examples        libtrellis  misc       timing
COPYING             devices.json        docs            experiments     metadata    README.md  tools

$ cd libtrellis

$ cmake -DCMAKE_INSTALL_PREFIX=/opt/trellis  .  # We're not installing to /usr
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.8", minimum required is "3.5") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable version "3.6.8", minimum required is "3.5") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   filesystem
--   thread
--   system
--   chrono
--   date_time
--   atomic
CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:1626 (message):
  No header defined for python-py368; skipping header check
Call Stack (most recent call first):
  CMakeLists.txt:36 (find_package)


-- Could NOT find Boost
CMake Warning at /usr/share/cmake-3.10/Modules/FindBoost.cmake:1626 (message):
  No header defined for python-py36; skipping header check
Call Stack (most recent call first):
  CMakeLists.txt:43 (find_package)


-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   python-py36
--   filesystem
--   thread
--   system
--   chrono
--   date_time
--   atomic
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   program_options
-- Configuring done
-- Generating done
-- Build files have been written to: /build/prjtrellis/libtrellis


$ make
...
[100%] Linking CXX executable ecpmulti
[100%] Built target ecpmulti

$ make install
$ cd /opt/trellis
$ tree -d -L 4
.
├── bin
├── lib
│   └── trellis
└── share
    └── trellis
        ├── database
        │   └── ECP5
        ├── misc
        │   ├── basecfgs
        │   └── openocd
        ├── timing
        │   └── util
        └── util
            └── common

NextPnR

Below we basically follow the directions on the nextpnr Github page. Note that we found we needed to install libeigen3-dev first

$ cd /build
$ git clone https://github.com/YosysHQ/nextpnr.git
$ cd nextpnr
$ git log
commit ccd9ca2a30374341207a975cfd1de172b9a59480 (HEAD -> master, origin/master, origin/HEAD)
Merge: c192ba2 a26c9bb
Author: David Shah 
Date:   Sat Aug 24 19:43:50 2019 +0100

    Merge pull request #317 from DurandA/feature/ecp5-unpromote-clock
    
    Restrict clock promotion to global on ECP5
...

$ ls
3rdparty  CMakeLists.txt      common   docs  generic  ice40  python     tests
bba       CodeCoverage.cmake  COPYING  ecp5  gui      json   README.md

$ cmake -DCMAKE_INSTALL_PREFIX=/opt/nextpnr -DARCH=ecp5 \
-DTRELLIS_ROOT=/opt/trellis/share/trellis -DPYTRELLIS_LIBDIR=/opt/trellis/lib/trellis

...
-- Configuring architecture : ecp5
-- Configuring done
-- Generating done
-- Build files have been written to: /build/nextpnr

$ make 
...
[100%] Linking CXX executable nextpnr-ecp5
[100%] Built target nextpnr-ecp5


$ make install
...
Install the project...
-- Install configuration: "Release"
-- Installing: /opt/nextpnr/bin/nextpnr-ecp5

$ cd /opt/nextpnr/

$ tree
.
└── bin
    └── nextpnr-ecp5

Complete toolchain flow to generate our first bitstream file:

Bringing it all together for our test example, we will be using Yosys, nextpnr, and Project Trellis to generate a bitstream file for our led_tst project. For now, we will to comment out the lines in our Verilog source containing PUR and GSR, as these primitives are not currently supported by Yosys. To generate the bitstream, we will need led_tst.v and led_tst.ldf, along with the tools we just installed. We also make use of the simple script "led_tst.ys" shown below:

led_tst.ys
read_verilog led_tst.v
synth_ecp5 -json led_tst.json

As shown above, we installed all Yosys tools and libraries to /opt in their own directories. This was done in order to be able to clearly identify, remove, and overwrite each of these tools.

For now, we'll create an environment file that we'll source before using the tools:

set-yosys-env
export PATH=/opt/yosys/bin:/opt/trellis/bin:/opt/nextpnr/bin:$PATH

Below, we work through each step of the build process:

Yosys

$ cd ~/led_tst
$ ls
led_tst.lpf  led_tst.v  led_tst.ys

$ source set-yosys-env

$ yosys led_tst.ys
 Yosys 0.8+643 (git sha1 78b30bbb, gcc 7.4.0-1ubuntu1~18.04.1 -fPIC -Os)


-- Executing script file `led_tst.ys' --

...
2.29. Printing statistics.

=== led_tst ===

   Number of wires:                 29
   Number of wire bits:            132
   Number of public wires:           4
   Number of public wire bits:      24
   Number of memories:               0
   Number of memory bits:            0
   Number of processes:              0
   Number of cells:                 55
     CCU2C                          11
     LUT4                           22
     OSCG                            1
     TRELLIS_FF                     21

2.30. Executing CHECK pass (checking for obvious problems).
checking module led_tst..
found and reported 0 problems.

2.31. Executing JSON backend.

End of script. Logfile hash: 969712a261
CPU: user 0.53s system 0.04s, MEM: 230.32 MB total, 202.82 MB resident
Yosys 0.8+643 (git sha1 78b30bbb, gcc 7.4.0-1ubuntu1~18.04.1 -fPIC -Os)
Time spent: 65% 13x read_verilog (0 sec), 10% 1x share (0 sec), ...

NextPnR

$ nextpnr-ecp5 --json led_tst.json --lpf led_tst.lpf --textcfg led_tst.config --um-45k --package CABGA381
Info: Importing module led_tst
Info: Rule checker, verifying imported design
Info: Checksum: 0x6c3b2cec
...
Info: Device utilisation:
Info: 	       TRELLIS_SLICE:    55/21924     0%
Info: 	          TRELLIS_IO:     2/  244     0%
Info: 	                DCCA:     1/   56     1%
Info: 	              DP16KD:     0/  108     0%
Info: 	          MULT18X18D:     0/   72     0%
Info: 	              ALU54B:     0/   36     0%
Info: 	             EHXPLLL:     0/    4     0%
Info: 	             EXTREFB:     0/    2     0%
Info: 	                DCUA:     0/    2     0%
Info: 	           PCSCLKDIV:     0/    2     0%
Info: 	             IOLOGIC:     0/  160     0%
Info: 	            SIOLOGIC:     0/   84     0%
Info: 	                 GSR:     0/    1     0%
Info: 	               JTAGG:     0/    1     0%
Info: 	                OSCG:     1/    1   100%
Info: 	               SEDGA:     0/    1     0%
Info: 	                 DTR:     0/    1     0%
Info: 	             USRMCLK:     0/    1     0%
Info: 	             CLKDIVF:     0/    4     0%
Info: 	           ECLKSYNCB:     0/    8     0%
Info: 	             DLLDELD:     0/    8     0%
Info: 	              DDRDLL:     0/    4     0%
Info: 	             DQSBUFM:     0/   10     0%
Info: 	     TRELLIS_ECLKBUF:     0/    8     0%
...
Info: Max frequency for clock '$glbnet$clk': 271.00 MHz (PASS at 12.00 MHz)

Info: Max delay              -> posedge $glbnet$clk: 17.46 ns
Info: Max delay posedge $glbnet$clk ->             : 2.92 ns
...
Info: Max frequency for clock '$glbnet$clk': 326.37 MHz (PASS at 12.00 MHz)

Info: Max delay              -> posedge $glbnet$clk: 8.07 ns
Info: Max delay posedge $glbnet$clk ->             : 1.81 ns
...

Trellis

$ ecppack led_tst.config led_tst.bit
$ ls
led_tst.bit  led_tst.config  led_tst.json  led_tst.lpf  led_tst.v  led_tst.ys

We tried to load the generated bit file in the Lattice Pogrammer and observed the following warning:

WARNING - 
Cannot get the device name from the file. 
File ...led_tst_yosys.bit: is invalid for expected LFE5UM-45F device.

After some investigation, we determined that our Diamond generated bit file contained the following ASCII header: "Lattice Semiconductor Corporation Bitstream Version: Diamond (64-bit) 3.11.0.396.4 Bitstream Status: Final Version 10.27 Design name: ECP5_basic_test_impl1.ncd Architecture: sa5p00m Part: LFE5UM-45F-8CABGA381 Date: Tue Aug 20 15:04:15 2019 Rows: 9470 Cols: 846 Bits: 8011620 Readback: Off Security: Off Bitstream CRC: 0x66BA".

This header contains the target device, and it appears that Diamond examines this header for the device ID, rather than in the actual bitstream itself. Trellis doesn't generate this Diamond-specific text, so Diamond generates the aforementioned warning.

The good news is that after manually inserting the header, we verified that the bit file can be programmed on Darsena using the Yosys flow!

Simulation, Debug, and Verification

We know developers love Verilator, but we like using free versions of Modelsim and Aldec. Moving to an open source Verilog simulator seems secodary to us at this point, but maybe this will change as we get into it.

We use Lattice's Reveal virtual logic analyzer on a regular basis to see inside our FPGA design. It's unclear whether we need to give this up when moving to open source. The Reveal inserter encapsulates a user's top with debug logic and trace memory. Is it possible to still use Reveal? Will Reveal choke on a bistream produced by Project Trellis?

And we should probably point out that we have our own ideas about next generation debugging / introspection of networked FPGA designs. We're determined to design out FTDI and JTAG bit banging from our boards.

The core members of Yosys appear to be deeply rooted in verification, and we look forward to learning more about the state of the art as we dig into Yosys.

Next Up...

We'll continue updating this article as we come up to speed on this open source toolchain. We'll also continue to compare the toolchain to the work flow and outputs when using both Lattice Diamond and Intel Quartus Prime Lite (free version).

Addtional Terms and Acronyms

  • AIG: And-Inverter Graph, a Boolean network composed of two-input ANDs and inverters
  • AST: Abstract Syntax Tree
  • BLIF: Berkeley Logic Interchange Format
  • LUT: Look up table (building block of ECP5, Cyclone, and other FPGAs)
  • SAT: Boolean satisfiability. From Wikipedia: "asks whether the variables of a given Boolean formula can be consistently replaced by the values TRUE or FALSE in such a way that the formula evaluates to TRUE"

Additional References

Didn't find an answer to your question? Post your issue below or in our new FORUM, and we'll try our best to help you find a solution.

And please note that we update our site daily with new content related to our open source approach to network security and system design. If you would like to be notified about these changes, then please follow us on Twitter and join our mailing list.

Related articles on this site:

share
subscribe to mailing list:

Please help us improve this article by adding your comment or question:

your email address will be kept private
authenticate with a 3rd party for enhanced features, such as image upload
previous month
next month
Su
Mo
Tu
Wd
Th
Fr
Sa
loading