CatiROC test firmware

This project is under source control and available as a git repository. To download, it is necessary to have access to a git client.

• Linux users may refer to their preferred package manager for instructions on installing git
• Windows users may use git for windows, mobaxterm of similar

Then, launch a terminal and cd to the location where the project will live, creating necessary folders if necessary

cd d:
mkdir FolderWhereToDownload
cd FolderWhereToDownload

Finally, clone the repository with

git clone https://gitlab.in2p3.fr/CatiROC-test/firmware.git catiROC-test-firmware

and maybe rename it to something more explicit

mv catiROC-test-firmware myGreatFirmwareProject

Just get if from

https://gitlab.in2p3.fr/CatiROC-test/firmware/repository/archive.zip

and extract its contents; then maybe rename it to something more explicit

mv catiROC-test-firmware myGreatFirmwareProject

Two flavors exist here, following the eprom programming device mounted on board: EPC04 and EPC16. To switch between them use

Project -> Cyclone III -> Device --> Device and Pin outputs --> Configuration
--> Use configuration device

Configuration

nb_asics = 1
num_of_discri_channels = 12
RegBankLength = 70

As it only deals with one ASIC and the number of available resources only provide buffering capabilities for handling up to 12 discriminator channels. The depth of the register bank is limited to 70 8-bits words.

To compile using altera’s quartus 13.1, launch the project file and do as usual

The configuration to be used is

nb_asics = 8
num_of_discri_channels = 16
RegBankLength = 70
freq_source = 40

Here, we need to manage up to 8 ASICS and all discriminator channels. The register bank length must be at least 70 8-bits words. Finally, the freq_source parameters stands for the running frequency, being 40 MHz the nominal one (another option is to run at 42 MHz).

To compile:

using xilinx ise 14.7

launch the project file and do as usual; as another possibility, you may use the provided makefile

using vivado 17.2

go to the vivado directory and regenerate the full project with

./build_project.sh

Then launch vivado

vivado -mode tcl

and open the juno_mezzanine.xpr project

open_project vivado17.2-project/vivado17.2-project.xpr

Run synthesis

launch_runs synth_1 -jobs 40
wait_on_run synth_1

and implement the design

launch_runs impl_1 -jobs 40
wait_on_run impl_1

set a property to increase spi speed

open_run impl_1
set_property STEPS.WRITE_BITSTREAM.TCL.PRE pre-bit.tcl [get_runs impl_1]

finally, generate the bitstream

launch_runs impl_1 -to_step write_bitstream -jobs 40

Another, simpler option is to run

./run_all.sh

which is equivalent to running synthesis followed by the implementation stage

./run_synthesis.sh
./run_implementation.sh

Note that in general, all shell commands are just wrappers around tcl scripts with the same name.

Use quartus programming tools with one of

/quartus13.1-project/output_files/top.sof

or

/quartus13.1-project/output_files/top.pof

to program the fpga or its eprom memory.

using xilinx ise 14.7

use Xilinx Impact as usual

using vivado 17.2

go to the vivado17.2-project directory and use the provided scripts to program the fpga or the flash using the resulting top.bit file under /vivado17.2-project/vivado17.2-project.runs/impl_1

./program_flash.sh folder/top.bit
./program_fpga.sh folder/top.bit

where folder stands for any local directory.

Finally, power cycle the card. In case you observe that the bitstream is not downloaded to the fpga and the DONE pin is not asserted (red led doesn’t lights), refer to this.

CatiROC sends information about fine and coarse time, charge, channel number and gain (scale). Additionally, in order to further monitor the acquisition, an event counter per channel is appended. All of this information sums up to 64 bits for a physical event, which is to be decoded as explained below.

Finally, every 32k physical events, 16 “special” events (one by channel) are injected into the data flow. These include additional information (trigger rate, etc.) computed online.

The data format is the following

Physical event word, 64 bits

'0' - Gain (1 bit) - Ch.Nb. (4 bits) - Coarse Time (26 bits) -
EventCounter (12 bits) - Fine Time (10 bits) - Charge (10 bits)

Special information word, 64 bits

'1' - '0' - Ch.Nb. (4 bits) - 0 (47 bits) - TriggerRate (11 bits)

From this, it is evident that the leftmost bit must be used to distinguish between physical and “special” events.

CatiROC sends information about fine and coarse time, charge, channel number and gain (scale). Additionally, in order to further monitor the acquisition, an event counter per channel is appended. All of this information sums up to 64 bits for a physical event, which is to be decoded as explained below.

Finally, every 32k physical events, 16 “special” events (one by channel) are injected into the data flow. These include additional information (trigger rate, etc.) computed online.

The data format is the following

Physical event word, 64 bits

'00' - Ch.Nb. (4 bits) - Coarse Time (26 bits) -
Gain (1 bit) - EventCounter (11 bits) - Fine Time (10 bits) - Charge (10 bits)

Special information word, 64 bits

'01' - 0 (47 bits) - Ch.Nb. (4 bits) - TriggerRate (11 bits)

Discriminator information word, 64 bits

'10' - Edge (1 bit) - Ch.Nb. (4 bits) - EventCounter (25 bits) - Time Stamp (32 bits)

From this, it is evident that the two leftmost bits must be used to distinguish between “physical”, “special” and “discriminator” events.

Special information word, 64 bits

'01' - 0 (42 bits) - Ch.Nb. (4 bits) - TriggerRate (16 bits)

Packets of 32 words (“special” events) are readout.

Motivation

The motivation for this format revision is the inclusion of additional information to take into account the ASIC number as well as the card number. This is not necessarily useful for the Omega test board, but it will be mandatory when having to handle the new Juno board with up to 8 ASICS. Then, we preview the necessity to identify the board number too. This makes necessary to expand the event word size to 10 bytes (8 bytes right here).

Details

CatiROC reads out coded information about fine and coarse times, charge, channel number and gain (scale). Additionally, in order to monitor the acquisition, an event counter per channel is appended. Further on, as up to 8 ASICS may be present in a board, the Asic number field is to be considered too. Finally, to identify the card an extra card number field appears here. All of this information sums up to 80 bits for a physical event, which is to be decoded as explained below.

Additionally, every 400 ms, 32 “special” events (one by channel) are injected into the data flow. These include additional information (trigger rate, etc.) computed online.

Finally, a discriminator word includes time stamp tagged events originating from the local trigger output of CatiROC. Here, each event includes an event number and an edge bit giving the rising or falling slope it originates from.

Note that from the coding below, it is evident that the two leftmost bits (event id) must be used to distinguish between “physical”, “special” and “discriminator” events.

Coding

The data format is the following

Physical event word, 80 bits, EventID = ’00’

'00' - Ch.Nb. (4 bits) - Coarse Time (26 bits) - Gain (1 bit) - EventCounter (11 bits) - Charge (10 bits) - Fine Time (10 bits) -
’00000' (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

Special information word, 80 bits, EventID = ’01’

'01' - ’0’ (41 bits) - Ch.Nb. (5 bits) - TriggerRate (16 bits) -
’00000' (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)'

Discriminator information word, 80 bits, EventID = ’10’

Case of Juno board:

This board includes within the FPGA embedded time to digital converters (TDC) to “fine time” sample incoming triggers out of the ASIC. Thus, a 6 bits fine time field is included. A 10 bytes word is then produced at each detected edge (falling or rising) on the 16 input trigger pins from the ASIC.

'10' - Edge (1 bit) - Ch.Nb. (4 bits) - EventCounter (25 bits) - Time Stamp (26 bits) - Fine Time (6 bits) -
’00000’ (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

Note that the time stamp is synchronous to the 40 MHz clock to be coherent with that of the ASIC. The fine time has a period P of 1/(960e6) ns., so that P*24 equals 25 ns., that’s to say, the 40 MHz clock period.

Case of Omega test board:

There is no TDC in this case, as the FPGA is not capable of embedding this feature. No fine time field in this case, then, and the time stamp is extended to 32 bits.

'10' - Edge (1 bit) - Ch.Nb. (4 bits) - EventCounter (25 bits) - Time Stamp (32 bits) -
’00000’ (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

In this case, the time stamp is synchronous to the 80 MHz clock, in order to improve timing as much as possible.

Motivation

In the case of the juno mezzanine board, we need to accommodate the supernova burst in a reduced capacity ddr3 memory. To achieve this, we have to optimize the data flow by reducing the amount of information.

Details

This 4th version aims at achieving a data reduction between a 33% and a 50% percent (following the incoming event rate) by merging the two coupled falling and rising edge data words in one, and then, instead of providing the absolute timing of both, just giving the absolute time of the first one and the time difference in between. This feature is let as an option, and the user may select it or not by software.

Coding

The data format is the following

Physical event word, 80 bits, EventID = ’00’

No difference with respecto to v3.

'00' - Ch.Nb. (4 bits) - Coarse Time (26 bits) - Gain (1 bit) - EventCounter (11 bits) - Charge (10 bits) - Fine Time (10 bits) -
’00000' (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

Special information word, 80 bits, EventID = ’01’

No difference with respecto to v3.

'01' - ’0’ (41 bits) - Ch.Nb. (5 bits) - TriggerRate (16 bits) -
’00000' (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)'

Discriminator information word, 80 bits, EventID = ’10’

Case of Juno board:

This board includes within the FPGA embedded time to digital converters (TDC) to “fine time” sample incoming triggers out of the ASIC. Thus, a 6 bits fine time field is included. A 10 bytes word is then produced at each detected edge (falling or rising) on the 16 input trigger pins from the ASIC.

'10' - Edge (1 bit) - Ch.Nb. (4 bits) - EventCounter (25 bits) - Time Stamp (26 bits) - Fine Time (6 bits) -
’00000’ (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

Note that the time stamp is synchronous to the 40 MHz clock to be coherent with that of the ASIC. The fine time has a period P of 1/(960e6) ns., so that P*24 equals 25 ns., that’s to say, the 40 MHz clock period.

In order to limit the generated amount of data, it is possible (software selectable) to only readout a reduced version of the previous. This way, instead of producing a whole 10 bytes word for each detected edge, a single word is produced for each couple of edges (falling edge followed by rising edge). Its format is the following

'10' - Edge (1 bit) - Ch.Nb. (4 bits) - EventCounter (18 bits) - Interval (7 bits) -
Time Stamp (26 bits) - Fine Time (6 bits) -
’00000’ (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

Here, a single 10 bytes word is produced for each detected falling edge, and it includes the Interval to the next rising edge. This means, the measurement of the trigger width is computed online to a maximum of a pulse width of 127 fine time periods.

Corner cases

• Most usually, a rising edge will follow a falling edge. In this case, the Edge bit is set to 0, and the difference Interval is provided up to a maximum of 125 fine time periods.
• If no rising edge happens during the 125 fine time periods following a falling edge, the Edge bit is set to 1, and the Interval field is set to its maximum, 125
• In the case a second falling edge follows the first one, the Edge bit is set to 0 and the Interval field is set to 0
• If two rising edges follow, the Edge bit is set to 1 and the Interval field is set to 0

Case of Omega test board:

There is no TDC in this case, as the FPGA is not capable of embedding this feature. No fine time field in this case, then, and the time stamp is extended to 32 bits.

'10' - Edge (1 bit) - Ch.Nb. (4 bits) - EventCounter (25 bits) - Time Stamp (32 bits) -
’00000’ (5 bits) - ASIC.Nb (3 bits) - Card.Nb (8 bits)

In this case, the time stamp is synchronous to the 80 MHz clock, in order to improve timing as much as possible.

Scan

Implement the possibility to perform a systematic scan of the coarse time window.

OK

This provides a way to measure the fine time INL. See Fine Time characterization section for details.

The same top.vhd should fit both

Remove the need for two different top level modules (top and top_test)

OK

Done, now It is only necessary to update only three generics: nb_asics (1/8), nb_discri_channels (12/16) and RegBankLength (70/540)

Reset TS in DDS

Perform a reset in the time stamp of the discriminator stream in parallel to that of the ASIC so that both time stamps start counting at the same time; additionally, use the ovf signal from the ASIC to reset it too such that the two overflow at the same time.

OK

The (fpga generated) discriminator time stamp and that of CatiROC run synchronously, start and overflow in parallel. They only run at different clock frequencies (40 / 80 MHz)

Decrease the RegBank depth

Reduce the default value of the RegBankLength generic parameter at top level in order to decrease the needed resources

OK

Fixed to 70 bytes for omega test / 540 for juno mezzanine.

Privilege sync resets

Replace async resets by its sync equivalent, best suited to fpga architectures

OK

Done

Fill in ucf

Check that the code compiles after using the proper constraints under ise14.7, then, export the project to Vivado and check again

OK

Added ucf file compiled under both ise and vivado, seems ok

Go beyond 200 KHz (11 bits)

This limitation was previously set as the space available was reduced, this is not the case anymore

OK

Extended to 16 bits. From now on, the special event has the format

’01’ - 0 (42 bits) - Ch.Nb. (4 bits) - TriggerRate (16 bits)

instead of

’01’ - 0 (47 bits) - Ch.Nb. (4 bits) - TriggerRate (11 bits)

This means that event rates up to a few MHz may be measured by the fpga.

I call this data format v2.1.

Use a watchdog in case of low data rate

Currently, special events are readout every 32k physical events. It is necessary to implement reading them out even in case of very low count rate, this will additionally remove the need to reduce the size of the packet size in the gui

OK

From now on a counter determines a 400 ms. time window. Beyond this point, special events are readout regardless of the number of physical events. This counter gets reset once the special events are sent out

Readout 32 words not only 16

Currently, only a special word is produced by physical channel, containing specific information. It is necessary to readout discriminator events information too, not only physical events, so an extension to 32 words is needed

OK

Done: first 16 special events accomodate discriminator related information; the remaining 16 special events contain information relevant to the physical events

Compensate for priority

Use a priority flag when transferring data from fifo stages, add a generic to determine the number of discri channels to implement

OK

A priority flag is decorrelated of the data transfer process, and a generic num_of_discri_channels is added

Make the module nb_asics compatible

right here, the ’if’ construct avoids any value in this generic

Priority flag

Compensate the multiplexor, so that all channels have the same probability to be readout

OK

Now the asic_empty is extended to 8 bits, regardless of nb_asics; thus, the hard coded ’if’ construct may check all the 8 bits. Additionally, a priority_flag mechanism is implemented too

Bit to disable the feature

DDR makes amount of data to be readout rise beyond limits, specially considering the usb 2. Add a bit to command 6 to disable this feature when requested by the user.

OK

Done in this commit, in parallel to the software.

Check and implement if necessary new strategy

verify that CatiROC physical data is readout before anything else, as the DDR stream by penalize it

Fix postponed warnings

When I find something weird in the code, I append a “FIXME” or “TODO” message so that I know where to look at to fix non urgent things

Meet timing by fixing the code

And not by adding constraints, so better remove them

Xilinx FIFOs have same size as their altera equivalents

as the target hardware is much more powerfull, try to optimize FIFO resources distribution to improve throughput

Show ahead is less performing

For historical reasons, the original code for omega test board included fifos in show ahead mode, so xilinx fifos inheritate this setup. It is better to remove this for performance (in two configurations)

After this, include meaningfull information in here

currently, these extra words are empty, they should contain count rate measurement for discriminator events, for example

Currently is a mess. Maybe, remove components too in packages by using word.unit syntax.

Warnings when compiling under xilinx ise

WARNING:LIT:701 - PAD symbol "usb_siwua" has an undefined IOSTANDARD.
WARNING:LIT:702 - PAD symbol "usb_siwua" is not constrained (LOC) to a specific location.

WARNING:LIT:701 - PAD symbol "raz_ext" has an undefined IOSTANDARD.
WARNING:LIT:702 - PAD symbol "raz_ext" is not constrained (LOC) to a specific location.

WARNING:LIT:701 - PAD symbol "read_ext" has an undefined IOSTANDARD.
WARNING:LIT:702 - PAD symbol "read_ext" is not constrained (LOC) to a specific ocation.

WARNING:LIT:701 - PAD symbol "hold_ext" has an undefined IOSTANDARD.
WARNING:LIT:702 - PAD symbol "hold_ext" is not constrained (LOC) to a specific

Follow guidelines in here.

• Implement communication with GCU. Provide a bit to control data readout through usb or through dedicated connector to the usb: the bit in the acquisition command enables / disables the usb if / the new gcu i/o interface, both of which access to the read side of the final fifo

  BIG –> USB If

  FIFO –> new process (running in usb if clock domain) –> NEW-SMALL-FIFO –> GCU-IF

• Implement a simplified model of the ABC, to be run coupled to the GCU: just an spi slave tanking orders, and maybe a fifo being filled up with dummy data

Implement and on line event builder to remove current random behaviour in event ordering

Recompile the firmware with the right generic value, and check operation at 42 MHz with respect to 40 MHz.

Online computing of the s-curve in the firmware, coupled to a bit of software data alignement. Calculations have been done so that all channels are processed in parallel, not just one as previously.

What has been done is implement computing as in data rate, all 16 channels in parallel, and readout the information along with four bits to take into account the channel number. Then, implement the same in a module, with generics the number of channels, and instantiate it from both omega test and juno mezzanine.

Following the implementation, it’s still necessary to check that the current implementation is right, for both omega test board and for ABC board.

Unable to be checked due to errors in the schematic. Andrea:

“The problem in the schematics is that there is a swap among bits of different bytes. In the DDR and in the FPGA memory controller the 16bit data word is organized in two byte. In the FPGA is possible to swap the bit within the same byte but not among different bytes. This limitation is due to the fact that there are two different data strobes for each byte.

I am referring to the following document at pages 193 and 194

https://www.xilinx.com/support/documentation/ip_documentation/mig_7series/v4_0/ug586_7Series_MIS.pdf”

Check

• implementation: timing at several pulses (amplitude, width and rate)
• data reduction feature