Driver
Driver Features
Requirements
The fmc-tdc device driver has been developed and tested on Linux 3.10. Other Linux versions might work as well but it is not guaranteed.
This driver depends on the zio framework and fmc library; we developed and tested against version zio 1.4 and fmc 1.1.
The FPGA address space must be visible on the host system. This requires a driver for the FPGA carrier that exports the FPGA address space to the host. As of today we support SPEC and SVEC.
Compile And Install
The compile and install the fmctdc1ns5ch device driver you need
first to export the path to its direct dependencies, and then you
execute make
. This driver depends on the zio framework and
fmc library; on a VME system it depends also on the VME bridge
driver from CERN BE-CEM. Additionally it is assumed that location of wbgen2 is
available via PATH
variable.
$ cd /path/to/fmc-tdc/software/kernel
$ export LINUX=/path/to/linux/sources
$ export ZIO=/path/to/zio
$ export FMC=/path/to/fmc-sw
$ export VMEBUS=/path/to/vmebridge
$ make
$ make install
Note
Since version v8.0.0 the fmc-tdc device driver does not depend anymore on fmc-bus subsystem, instead it uses a new fmc library
The building process generates 3 Linux modules: kernel/fmc-tdc.ko, kernel/fmc-tdc-spec.ko (for SPEC card), and kernel/fmc-tdc-svec.ko (for SVEC card).
Drivers’ Dependencies
The TDC driver requires the following drivers to function:
if the used carrier is SPEC then from spec repository: gn412x-fcl.ko, gn412x-gpio.ko, spec-gn412x-dma.ko and spec-fmc-carrier.ko
if the used carrier is SVEC then vmebus.ko
from general-cores repository: spi-ocores.ko, i2c-ocores.ko and htvic.ko (more details in the section Building General Cores drivers)
from zio repository: zio-buf-vmalloc.ko and zio.ko (more details in the section Building ZIO drivers)
from fmc repository: fmc.ko (more details in the section Building FMC driver)
drivers from the kernel tree: mtd.ko, at24.ko, m25p80.ko, i2c_mux.ko and fpga-mgr.ko (available in kernels v4.4 and newer, for older kernels see section Building FPGA manager driver)
In addition the following tools are required to build above drivers:
cheby (more details in the section Installing Cheby)
wbgen2 (more details in the section Installing Wbgen2)
Please read the following subsections for details
Building Drivers
This subsection describes the build process of Linux Device Drivers used by the TDC and tools needed during their build.
Installing Cheby
Clone cheby repository:
$ git clone https://gitlab.cern.ch/cohtdrivers/cheby.git
Install cheby:
$ cd cheby
$ python setup.py install
It may be required to install python-setuptools or python-setuptools.noarch package using your Linux distribution’s software manager.
Installing Wbgen2
Clone wbgen2 repository:
$ git clone https://ohwr.org/project/wishbone-gen.git
If needed export the location of wbgen2 (needed for fmc-tdc drivers compilation):
export WBGEN2=/path/to/wishbone-gen/wbgen2
Building FPGA Manager driver
If kernel module fpga-mgr.ko is not available in the kernel that is used, probably the backported version is needed.
Clone backported fpga-manager repository:
$ git clone https://gitlab.cern.ch/coht/fpga-manager.git
Build and install kernel module (fpga-mgr.ko):
$ cd fpga-manager
$ export LINUX=/path/to/linux/sources
$ make
$ make install
Building ZIO drivers
Clone zio repository:
$ git clone https://ohwr.org/misc/zio.git
Build and install kernel modules (zio-buf-vmalloc.ko and zio.ko):
$ cd zio
$ export LINUX=/path/to/linux/sources
$ make
$ make install
Building General cores drivers
Clone general-cores repository:
$ git clone https://ohwr.org/project/general-cores.git
Build and install kernel modules (spi-ocores.ko, i2c-ocores.ko and htvic.ko):
$ cd general-cores/software
$ export LINUX=/path/to/linux/sources
$ make
$ make install
Building FMC driver
Clone fmc repository:
$ git clone https://ohwr.org/project/fmc-sw.git
Build and install kernel module (fmc.ko):
$ cd fmc-sw/ $ export LINUX=/path/to/linux/sources $ make $ make install
Building SPEC drivers
Clone spec repository:
$ git clone https://ohwr.org/project/spec.git
Build and install kernel modules (gn412x-fcl.ko, gn412x-gpio.ko, spec-gn412x-dma.ko and spec-fmc-carrier.ko):
$ cd spec/software
$ export CHEBY=/path/to/cheby/bin/cheby
$ export I2C=/path/to/general-cores/software/i2c-ocores
$ export SPI=/path/to/general-cores/software/spi-ocores
$ export FPGA_MGR=/path/to/fpga-manager
$ export FMC=/path/to/fmc-sw
$ export LINUX=/path/to/linux/sources
$ make
$ make install
Building SVEC drivers
Building missing mainline drivers
It may happen that your system lacks of drivers that are included into the mainline Linux kernel. This section describes how to build i2c-mux.ko and m25p80.ko drivers for CENTOS 7.
The first step is to download the Linux sources that mach the version used in your system and unpack them using your favorite method. Then prepare sources for a compilation:
- ::
make prepare
Select missing drivers by adding CONFIG_I2C_MUX=m
and
CONFIG_MTD_M25P80=m
to .config manually, or with a favorite tool (like
menuconfig
. Start the build of missing drivers:
make M=drivers/i2c/
make M=drivers/mtd/devices/
Copy drivers from drivers/mtd/devices/m25p80.ko
and drivers/i2c/i2c-mux.ko
to a known place.
Top Level Driver
The fmc-tdc is a generic driver for an FPGA device that could be instanciated on a number of FMC carriers. For each carrier we write a little Linux module which acts as a top level driver (like the MFD drivers in the Linux kernel). In these modules there is the knowledge about the virtual memory range, the IRQ lines, and the DMA engine to be used.
The top level driver is a platform driver that matches a string containing the application identifier. The carrier driver builds this identification string from the device ID embedded into the FPGA (https://ohwr.org/project/fpga-dev-id).
Loading drivers for SPEC
Load drivers at24.ko and mtd.ko. They should be distributed with
your Linux distribution in package like kernel-plus
for CENTOS 7 of
linux-modules
for Ubuntu.
sudo modprobe at24
sudo modprobe mtd
Load drivers from the mainline Linux:
sudo insmod i2c-mux.ko
sudo insmod m25p80.ko
Load fmc drivers:
sudo insmod fmc.ko
Load fpga-manager drivers:
sudo insmod fpga-mgr.ko
Load drivers from general-cores:
sudo insmod htvic.ko
sudo insmod i2c-ocores.ko
sudo insmod spi-ocores.ko
Load drivers from spec-sw:
sudo insmod spec-gn412x-dma.ko
sudo insmod gn412x-gpio.ko
sudo insmod gn412x-fcl.ko
sudo insmod spec-fmc-carrier.ko
If you use the custom path to the firmware, set it at the latest at this point.
echo -n <path_to_bitstreams> | sudo tee /sys/module/firmware_class/parameters/path
Load bitstream into SPEC’s FPGA:
echo -n <bitstream.bin> | sudo tee /sys/kernel/debug/<PCIe_device>/fpga_firmware
Load the ZIO and TDC drivers:
sudo insmod zio.ko
sudo insmod zio-buf-vmalloc.ko
sudo insmod fmc-tdc.ko
sudo insmod fmc-tdc-spec.ko
Loading drivers for SVEC
For SVEC the loading procedure is very similar to SPEC. It is required to load svec-fmc-carrier.ko and fmc-tdc-svec.ko instead of spec-fmc-carrier.ko and fmc-tdc-spec.ko. Additionally, there is no need to load spec-gn412x-dma.ko, gn412x-gpio.ko and gn412x-fcl.ko, since these drivers are specific to SPEC.
sudo modprobe at24
sudo modprobe mtd
sudo insmod i2c-mux.ko
sudo insmod m25p80.ko
sudo insmod fmc.ko
sudo insmod fpga-mgr.ko
sudo insmod htvic.ko
sudo insmod i2c-ocores.ko
sudo insmod spi-ocores.ko
sudo insmod svec-fmc-carrier.ko
echo -n <path_to_bitstreams> | sudo tee /sys/module/firmware_class/parameters/path
echo -n <bitstream.bin> | sudo tee /sys/kernel/debug/svec-vme.<slot>/fpga_firmware
sudo insmod zio.ko
sudo insmod zio-buf-vmalloc.ko
sudo insmod fmc-tdc.ko
sudo insmod fmc-tdc-svec.ko
Module Parameters
The driver accepts a few load-time parameters for configuration. You can
pass them to insmod directly, or write them in /etc/modules.conf
or
the proper file in /etc/modutils/
.
The following parameters are used:
- irq_timeout_ms=NUMBER
It sets the IRQ coalesing timeout expressed in milli-seconds (ms). By default the value is set to 10ms.
- test_data_period=NUMBER
It sets how many fake timestamps to generate every seconds on the first channel, 0 to disable. By default the value is set to 0.
- dma_buf_ddr_burst_size=NUMBER
It sets DDR size coalesing timeout expressed in number of timestamps. By default the value is set to 16 timestamps.
- wr_offset_fix=NUMBER
It overwrites the White-Rabbit calibration offset for calibration value computed before 2018. By default this is set to 229460 ps.
Device Abstraction
This driver is based on the ZIO framework. It supports initial setup of the board; it allows users to manually configure the board, to start and stop acquisitions, to force trigger, and to read all the acquired time-stamps.
The driver is designed as a ZIO driver. ZIO is a framework for input/output hosted on http://www.ohwr.org/projects/zio.
ZIO devices are organized as csets (channel sets), and each of them includes channels. All channels belonging to the same cset trigger together. This device offers a channel-set for each channel.
Note
Unless specified, the units are the same as for the TDC HDL design. Therefore, this driver does not perform any data processing.
The Overall Device
As said, the device has 5 cset with 1 channel each. Channel sets from 0 to 4 represent the physical channels 1 to 5. In other words a channel set represents a single TDC channel.
The TDC registers can be accessed in the proper sysfs directory:
cd /sys/bus/zio/devices/tdc-1n5c-${ID}
The overall device (tdc-1n5c) provides the following attributes:
- calibration_data
It is a binary attribute which allows the user to change the run-time calibration data (the EEPROM will not be touched). The
fmc-tdc-calibration
tool can be used to read write calibration data. To be consistent, this binary interface expects only little endian values because this is the endianness used to store calibration data for this device.- coarse
Coarse part of the current TAI time. This value is in nanoseconds with 8 ns resolution. The
fmc-tdc-time
tool can be used to read TAI time.- command
Send the command to the driver. As today it is possible to enable/disable White Rabbit, set the board to the current time or check the source of the timing. The
fmc-tdc-time
tool can be used to send the commands related to the current time source.- seconds
Current TAI time in seconds. The
fmc-tdc-time
tool can be used to read TAI time.- temperature
It shows the current temperature. To get the temperature in C degrees use the formula
temperature/16
. Thefmc-tdc-temperature
tool can be used to read the temperature.- transfer-mode
It shows the current transfer mode. 0 for FIFO, 1 for DMA.
- wr-offset
Offset used by White Rabbit.
The Channel Set
The TDC has 5 Channel Sets named cset[0-4]
. Its attributes are
used to control and monitor each TDC channel individually. All
channel specific attributes are available at the channel set level.
The Channels
Because there is a one-to-one relation with the channel set, we have decided to put all custom attributes at the channel set level. So, at this level you will find only default ZIO attributes.
The Trigger
TODO fix this section
In ZIO, the trigger is a separate software module, that can be replaced at run time. This driver includes its own ZIO trigger type, that is selected by default when the driver is initialized. You can change trigger type (for example use the timer ZIO trigger) but this is not the typical use case for this board.
This is the list of attributes (excluding kernel-generic and ZIO-generic ones):
- enable
This is a standard zio attribute, and the code uses it to enable or disable the hardware trigger (i.e. internal and external). By default the trigger is enabled.
- post-samples, pre-samples
Number of samples to acquire. The pre-samples are acquired before the actual trigger event (plus its optional delay). The post samples start from the trigger-sample itself. The total number of samples acquired corresponds to the sum of the two numbers. For multi-shot acquisition, each shot acquires that many sample, but pre + post must be at most 2048.
The Buffer
TODO fix this section
In ZIO, buffers are separate objects. The framework offers two buffer types: kmalloc and vmalloc. The former uses the kmalloc function to allocate each block, the latter uses vmalloc to allocate the whole data area. While the kmalloc buffer is linked with the core ZIO kernel module, vmalloc is a separate module. The driver currently prefers kmalloc, but even when it preferred vmalloc (up to mid June 2013), if the respective module was not loaded, ZIO would instantiate kmalloc.
You can change the buffer type, while not acquiring, by writing its name to the proper attribute. For example:
echo vmalloc > /sys/bus/zio/devices/tdc-1n5c-0004/cset0/current_buffer
The disadvantage of kmalloc is that each block is limited in size. usually 128kB (but current kernels allows up to 4MB blocks). The bigger the block the more likely allocation fails. If you make a multi-shot acquisition you need to ensure the buffer can fit enough blocks, and the buffer size is defined for each buffer instance, i.e. for each channel. In this case we acquire only from the interleaved channel, so before making a 1000-long multishot acquisition you can do:
export DEV=/sys/bus/zio/devices/tdc-1n5c-0004
echo 1000 > $DEV/cset0/chani/buffer/max-buffer-len
The vmalloc buffer allows mmap support, so when using vmalloc you can save a copy of your data (actually, you save it automatically if you use the library calls to allocate and fill the user-space buffer). However, a vmalloc buffer allocates the whole data space at the beginning, which may be unsuitable if you have several cards and acquire from one of them at a time.
The vmalloc buffer type starts off with a size of 128kB, but you can change it (while not acquiring), by writing to the associated attribute of the interleaved channel. For example this sets it to 10MB:
export DEV=/sys/bus/zio/devices/tdc-1n5c-0004
echo 10000 > $DEV/cset0/chani/buffer/max-buffer-kb
The debugfs Interface
When the DMA mode is used, the fmctdc1ns5cha driver exports a set of debugfs
attributes which
are supposed to be used only for debugging activities. For each device
instance you will see a directory in /sys/kernel/debug/fmc-tdc.*
.
- regs
It dumps the FPGA registers
Reading Data with Char Devices
To read data from user-space, applications should use the ZIO char device interface. ZIO creates 2 char devices for each channel (as documented in ZIO documentation). The TDC can acquire data on each channel independently, so ZIO creates ten char device, as shown below:
$ ls -l /dev/zio/tdc-*
cr--r----- 1 root root 241, 0 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-0-0-ctrl
cr--r----- 1 root root 241, 1 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-0-0-data
cr--r----- 1 root root 241, 2 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-1-0-ctrl
cr--r----- 1 root root 241, 3 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-1-0-data
cr--r----- 1 root root 241, 4 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-2-0-ctrl
cr--r----- 1 root root 241, 5 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-2-0-data
cr--r----- 1 root root 241, 6 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-3-0-ctrl
cr--r----- 1 root root 241, 7 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-3-0-data
cr--r----- 1 root root 241, 8 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-4-0-ctrl
cr--r----- 1 root root 241, 9 Jan 13 13:36 /dev/zio/tdc-1n5c-000b-4-0-data
If more than one board is probed for, you’ll have more similar
pairs of devices, differing in the dev_id field, i.e. the 000b
shown
above. The dev_id field is assigned by the Linux kernel platform subsystem.
The char-device model of ZIO is documented in the ZIO manual; basically, the ctrl device returns metadata and the data device returns data. Items in there are strictly ordered, so you can read metadata and then the associated data, or read only data blocks and discard the associated metadata.
The zio-dump
tool, part of the ZIO distribution, turns metadata and data
into a meaningful grep-friendly text stream.
User Header Files
Both the kernel and the user make use of the same header file
fmc-tdc.h
. This because they need to share some data stracture and
constants use to interpret data and meta-data in the library or by an
application
Troubleshooting
This chapter lists a few errors that may happen and how to deal with them.
Installation issue with modules_install
The command sudo make modules_install
may place the modules in the wrong
directory or fail with an error like:
make: *** /lib/modules/<kernel-version>/build: No such file or directory.
This happens when you compiled by setting LINUX=
and your sudo is not
propagating the environment to its child processes. In this case, you
should run this command instead:
sudo make modules_install LINUX=$LINUX