Provided below is draft documentation for the FPGA architecture and Verilog modules that comprise the Private Island project. Documentation is being updated on a regular basis, and the source is hosted on Github.
The system is highly configurable, and the user can specify at build time many parameters including the number of PHYs to instantiate, packet filter purpose, trigger points, & size, and buffer depths.
Today, the project supports the Lattice Semiconductor ECP5UM. The Gigabit Ethernet PHY supported is the Texas Instruments DP83867 with also some support provided for the Marvell Alaska series. An external on-board micro controller is assumed to be present for FPGA configuration and control. Any micro controller can be used, but our development system utilizes an NXP Kinetis K2x (ARM Cortex M4) series with an Ethernet MAC integrated within the FPGA.
The figure below depicts a system with three PHYs instantiated for the purpose of using PHY2 as a dedicated system controller port connected to an external Linux processor performing high-level packet analysis and control. In this scenario, the Private Island system acts as an Ethernet bridge / filter / firewall between PHY0 and PHY1.
Verilog Coding Conventions
- data buses follow a naming convention: dir_driver_function_srcSlot_destSlot (e.g, rx_df_fifo_d_01)
- delayed / registered data: _m1, _m2, _m3, _m4
- K28_5, D16_2 represents 8’hBC, 8’h50, /I2/ or IDLE 2
- FIFOs are used throughout to buffer data
- worker interface definition to start and finish work assigned by internal controller: controller.v
- one shots are common and trigger further action
The figure below depicts the packet flow and high-level switch architecture. The packet switching is centralized and is controlled by configuring the source for each TX port.
Regardless of port, the Receive (RX) direction is always into the switch, and the transmit (TX) direction is always from the switch. Data buffering and filtering / dropping is done in the RX direction.
Currently, all packet data (byte/octet wide) is clocked at 125 MHz . However, it is possible to down sample the data into a wider data path for analysis and storage functions (e.g., 32-bits @ 31.25 MHz).
The figure below depicts the packet data receive path. The PHY is external to the FPGA and connects via SGMII. The SERDES/PCS block is a hard macrocell within the ECP5UM. The remaining blocks in the figure are Verilog modules and are described further below.
A partial list of module descriptions is provided below (documentation is being added on a regular basis):
Purpose: SGMII TX/RX/AN state machines
- Controller is specific to both SGMII and a lesser extent the Lattice PCS block.
- Each frame written to the FIFO consists of the MAC destination, MAC source, Ethertype, and packet.
- Ethernet preamble and FCS are not stored
- However, the first 2 bytes of FCS are written into FIFO due to a lag issue on stopping a read during TX
- The RX packet complete bit is set on the last RX byte, so two more bytes are written after that (FCS).
- RX data written into FIFO is from rx_data_m1 register
- If code group /R/ is transmitted in an even-numbered position, append a single additional /R/ to the code-group stream to ensure that the subsequent /I/ is aligned on an even-numbered code-group boundary and EPD transmission is complete.
- Reading the FIFO starts two clocks early since data isn't available until after the first clock and then we register the data from the FIFO on the 2nd clock.
- Reading stops (tx_fifo_re is deasserted) after either tx_last_byte or tx_fifo_empty are detected on the next clock, These signals are expected to be coincident.
Purpose: packet filter / CAM wrapper with keep / drop logic
Purpose: single cycle, parameterized Content Addressable Memory
Purpose: Double buffered side-by-side FIFO with keep / drop logic
- Writer and Reader may either be active on first or second buffer simultaneously
- Delay (buffer depth) is set outside this module by writer ( hold off of keep or drop )
- A write can't begin on buffer[n] while a read is active on buffer[n].
- This is guaranteed to not happen as long as 2*IPG + tWR > tBuf[n]
- RX data is dropped after drop is asserted ( FIFO wr_ptr is reset ). Read on this buffer is inactive.
- FIFO buffer becomes available again after drop ( no need to switch to other buffer ).
- this lets writer drop packets early without regard to other buffer.
- FIFO is cleared on drop by setting wr_ptr to 0
- There is always a minimum gap between a need to switch buffers: FCS+IPG+PREAMBLE
- There is no empty flag since rd_ptr and wr_ptr may be pointing to different buffers.
Purpose: Buffer RX data until ready for TX switch
An MDIO controller is integrated within the FPGA to offload the micro controller and enable real-time, parallel communication with the Ethernet PHYs. Per 802.3-2015 clause 22, the FPGA performs Station Management (STA) via the two wire management bus, which consists of a clock (MDC) and a data signal (MDIO). The FPGA acts as the The real time nature of the communcation can be important for certain security functions as well as 1588 time stamping.
Purpose: MDIO Phy Driver
- "When the MDIO signal is sourced by the PHY, it is sampled by the STA synchronously with respect to the rising edge of MDC."
- "When the STA sources the MDIO signal, the STA shall provide a minimum of 10 ns of setup time and a minimum of 10 ns of hold time referenced to the rising edge of MDC"
- drive on the falling edge and sample on the rising edge
- Note both the Marvell and TI PHYs drive MDIO on the rising edge of the clock.
- data is msbit first on the wire.
- It is assumed that the PHY can support an MDIO burst cycle (insertion of idle bits isn't required).
- TI DP83867 requires that the FPGA waits at least one MDC clock cycle after a hard reset is negated.
State / bit definitions:
start : [0:1] opcode : [2:3] phy_addr : [4:8] 5 bits reg_addr : [9:13] 5 bits turn around : [14:15] data : [16:31]
Purpose: MDIO controller / state machine
- if rwn = 0, then next two bytes read from bus are write data
data bus / ROM format
d: end of program d: tbd d: rwn d[4:0]: reg addr