CTF

The CTF system is one of most widely used MEG systems and one of the first on which real-time MEG analysis has been done for the purpose of BCI (see Mellinger et al., NeuroImage 2007). There are 151-channel and 275-channel systems, which all provide real-time access to the MEG data through shared memory.

The acquisition software runs on a Linux computer. When prior to starting the acquisition software, shared memory with the appropriate details is initialized, the acquisition software will write a copy of the data to that shared memory. The shared memory is split over 600 packets, where each packet holding 28160 samples for older versions of the software or 40000 samples for newer versions of the software. With approximately 350 channels (MEG, EEG and status/trigger channels) in the typical MEG system, that amounts to approximately 80 (old) or 114 (new) samples per packet.

Multiple real-time interfaces have been developed over the years. The first version (ctf2ft_v1, originally known as AcqBuffer) only maintains the shared memory to allow it to be used as an ever-lasting ring buffer, but does not copy the data to the FieldTrip buffer. This version can be used in combination with the ft_realtime_ctfproxy.m function in MATLAB running on the acquisition computer. The header details must be read from the res4 file on the local filesystem. Although now deprecated, this is explained in more detail further down on this page.

The second version (ctf2ft_v2, originally known as acq2ft) combines the access to shared memory with copying to the FieldTrip buffer to make the data available elsewhere on the network. It operates by grabbing one packet (setup or data) at a time out of the shared memory, and more or less directly transferring it into a FieldTrip buffer that is started by the ctf2ft_v2 application itself, or a buffer that is running separately on the same computer or elsewhere on the network.

The ctf2ft_v2 application decodes header information from the .res4 file pointed to by the setup collection packet, and thus knows by itself which channels contain triggers instead of relying on a 3rd application or Matlab script to write that information back to shared memory. On top of that, ctf2ft_v2 overallocates the shared memory by 1000 samples and is prepared to operate successfully even if the proprietary Acq application writes too much data into any slot. Instead of preparing the shared memory segment for access by a third application, it streams the data and events (decoded from the trigger channels) to a local or remote FieldTrip buffer, as depicted by the following diagram:

You need to start ctf2ft_v2 before starting Acq so that the shared memory interface can be detected and connected to by the latter. You have the option of spawning a FieldTrip buffer server directly within ctf2ft_v2, using

ctf2ft_v2

to use the default port 1972, or using (note the minus)

ctf2ft_v2 - port

to spawn at server at the given port. You can also tell ctf2ft_v2 to stream the data to a buffer provided by another application (possibly on another machine), using

ctf2ft_v2 hostname port

Please note that for this to work, you will need to start up the remote buffer before ctf2ft_v2. Once ctf2ft_v2 is happily up and running, you can start Acq from a different command line or using existing GUI tools.

Since streaming the data to a remote FieldTrip buffer might incur a delay due to network traffic, ctf2ft_v2 employs an internal ring buffer for up to 10 data packets and a simple (socket pair) mechanism to synchronize between copying data packets from the shared memory to the internal ring buffer, and streaming the data from the ring buffer to the FieldTrip buffer. This means that small delays should not interfere with the time-critical operation of clearing up slots in the shared memory segment, as long as the average throughput is high enough.

The most recent interface, called ctf2ft_v3, does everything that version 2 does but has the additional ability to downsample the incoming CTF data, apply the correct sensor gains, and write out only selected channels. This application is started like this:

ctf2ft_v3 hostname:port:flags:decimation:channels

where flags can be any combination of R, which enables writing the .res4 file into the FieldTrip buffer header, E which enables sending events as decoded from the trigger channels, and G which enables multiplying the samples by the correct gain values, and consequently writing out single precision floating point numbers instead of the default 32bit integers. Decimation needs to be a positive integer number, and channels is a comma-separated list of channel labels, or a star (*) for sending all channels. However, it is important to note that *no* lowpass filtering is applied before decimation, that is, you have to use the hardware filters (setup in Acq) to use this option.

ctf2ft_v3 -:1972:RE:1:*

Actually you can have multiple definitions and stream different parts of the data to different buffers. For example, the following call will spawn a local FieldTrip buffer on port 1972, which will receive all channels, the .res4 header, and events (but data is kept at 32 bit integers), and in addition stream out 4x downsampled and scaled head-localization channels to a buffer on the lab-meg001 computer (also port=1972):

ctf2ft_v3 -:1972:RE:1:*
lab-meg001:1972:G:4:HLC0011,HLC0012,HLC0013,HLC0021,HLC0022,HLC0023,HLC0031,HLC0032,HLC0033

Note that the previous command should all be on a single line.

On the command line, change to the realtime/acquisition/ctf directory and type make. This will produce all versions of the interface, as well as some tools for testing and managing the shared memory. Note that you might need to compile the buffer library first.

In FieldTrip it is possible to use the fileio module to read from shared memory. Because the shared memory also has to be freed to ensure that the Acq software continues writing to it, the ctf2ft_v1 application has to be running in the background. It constantly loops over the 600 packets in shared memory, and if there are less than 20 packets free, it memcpy's the “setup” packet (containing the name of the res4 file that has the full header details in it) to the next packet, thereby freeing the packet previously containing the setup. This procedure ensures that the content of the setup packet can always be read, even while it is being copied.

The ft_realtime_ctfproxy function (part of the realtime module in FieldTrip) reads the MEG data from shared memory and writes to a FieldTrip buffer. The FieldTrip buffer is a multi-threaded and network transparent buffer that allows data to be streamed to it, while at the same time allowing another MATLAB session on the same or another computer to read data from the buffer for analysis.

Subsequently in another Matlab session you can read from the FieldTrip buffer using the ft_read_header, ft_read_data and ft_read_event functions by specifying 'buffer://hostname:port' as the filename to the reading functions.

Besides maintaining the copy of the setup packet and ensuring that there are always some packets free to receive the new data from Acq, the AcqBuffer application also performs trigger detection on indicated channels and stores these triggers in a convenient representation. This speeds up the trigger detection in the read_event function in MATLAB considerably.

In MATLAB the full header details of the MEG data set are determined by first reading the packet containing the setup information (i.e. the name of the res4 file) and subsequently by reading the details from that res4 file using the standard reading function. The number of samples available in the dataset is updated to reflect the amount of data present in the buffer.

After you start

>> ctf2ft_v1

on the linux command line, and in another terminal

>> Acq

You can access the data in MATLAB like this

hdr    = ft_read_header(filename), this returns a structure with the header information
event  = ft_read_event(filename), this returns a structure with the event information, (i.e. the triggers) 
dat    = ft_read_data(filename, ...), this returns a 2-D or 3-D array with the data

where filename should be a string containing 'ctf_shm://', i.e. similar as a Universal Resource Identifier. In case you want to use the header information from another res4 file, you can specify the filename as 'ctf_shm://<example.res4>', i.e. including the full path and filename of the res4 header file.

The (inter-)operation of the three involved software components, all running on the same machine can be summarised with the following diagram.

In 2016 CTF finalized the design for the 3000-series of the electronics (see CTF news). The new electronics does all processing on the acquisition computer, in contrast to the 2000-series electronics that does part of the processing on a 4-node compute cluster that is mounted in the electronics rack.

The new electronics comes with a new (beta) version of the acquisition software. However, that software can also be used in combination with the 2000-series electronics. This requires installing a new acquisition computer which bypasses the 4-node compute cluster.

Regardless whether you have the 3000-series electronics or not, the new version of the software (probably version 6.x and up) has the same shared-memory interface.

It seems that the default linux/redhat configuratino of the shared memory does not allow a sufficiently large memory block to be allocated. To change the setting in the operating system, you should do (as root user):

echo 1000000000 > /proc/sys/kernel/shmmax

There appears to be two ways of (re)defining the amount of shared memory in your system (tips thanks to Dave Glowacki of SSEC):

• add this line to your /etc/rc.d/rc.local file:

    echo shared_memory_size > /proc/sys/kernel/shmmax

(where shared_memory_size is the amount of shared memory you want to declare in bytes) and reboot.

A more permanent solution would be to change the value of SHMMAX in /usr/src/linux/include/asm/shmparam.h and rebuild your kernel.

There is a problem in the CTF acquisition software that sometimes causes the shared memory interface to fail. The diagnosis of the problem is that the Acq software runs and writes the data to the shared-memory buffer, where it is detected by AcqBuffer, and that after a certain random amount of time (around one minute) the AcqBuffer stops. The problem seems to be caused by a memory buffer overrun. The shared memory consists of 600 packets, each defined as

typedef struct {
 ACQ_MessageType message_type; % this is 4 bytes long
 int messageId;
 int sampleNumber;
 int numSamples;
 int numChannels;
 int data[28160];
} ACQ_MessagePacketType;

So in total each packet is 5*4+28160*4 bytes long, and there are 600 of those in shared memory. If the numChannels*numSamples of the previous block is slightly larger than 28160, it means that Acq is trying to write more data points into the “data” section of that packet than fits in, causing the data to flow over into the next packet. The first couple of integers in the next packet (indicating the Type and other details) are therefore messed up, and Aqc thinks that that packet is already filled. Then it stops writing to shared memory altogether.

I have tested this idea with a specially tweaked version of my AcqBuffer shared memory “maintenance” program and indeed see this happen for a data block that has 91*310=28210 samples in it, which is 50 more than the 28160 that would fit in. The next block is therefore corrupt.

Now understanding the problem, we can start thinking about a solution. Somehow in the CTF code there is an incorrect estimate of the number of samples that fits into a block. Probably we can play with the channel number to circumvent the problem. Given a certain channel number, Acq will have to determine how many samples fit into a single block. I suspect the bug in the Acq code to be something like sampleNumber = round(28160/numChannels) where it should be sampleNumber = floor(28160/numChannels) i.e. rounding off to the bottom. To solve this, we can look at

for chancount=1:500
success(chancount) = ((round(28160/chancount).*chancount)<=28160);
end

>> find(success)
ans =
    1     2     4     5     6     8    10    11    13    14    15    16    17    18    19    20    22    23    24    25    26    29   31    32    33    34    36    37    38    39    40    42    44    46    47    50    51    53    54    55    57    59    60    62    64    65    67    68    69    70    72    75    78    79    80    82    83    84    85    86    88    89    91    92    95    96    97    98    99   102   103   105   107   109   110   112   113   114   124   125   126   128   129   132   134   136   138   140   142   145   148   151   152   153   157   158   159   160   176   177   178   179   180   184   185   186   190   191   194   195   198   201   202   204   205   207   208   210   211   213   216   218   220   221   223   225   227   230   232   234   236   238   240   242   244   246   247   249   251   253   255   256   258   260   262   263   265   267   268   270   273   275   276   278   281   284   286   287   289   290   292   293   295   296   298   299   302   305   306   308   309   312   315   316   319   320   322   323   326   327   330   331   334   335   338   339   342   343   346   347   350   351   352   355   356   359   360   361   364   365   369   370   373   374   375   378   379   380   384   385   389   390   391   394   395   396   400   401   402   406   407   408   412   413   414   418   419   420   424   425   426   430   431   432   433   437   438   439   440   444   445   446   451   452   453   454   458   459   460   461   466   467   468   469   474   475   476   477   482   483   484   485   490   491   492   493   494   499   500

Update: This calculation seems not to be 100% correct. For example, 359 channels do NOT work. 360 seems to be okay.

This is the table (up to 500 channels) that lists the number of channels for which it will work. The test that I performed happened to be with 310 channels, which is not in the list. Since we can always add a few (unused) EEG channels to the acquisition, we can work around the bug in the CF Acq software.