Serial Data Logging

When developing and debugging firmware for a MCU, which is performing some quite extensive processing of the data from multiple sensors, it is very important to be able to look at this data as it is being collected and processed. Similarly, when tuning, for example, PID parameters it is critical to capture the RC input, attitude data, and control signals to motors. All of this points to a need to capture data from the board and subsequently transfer it to a PC for analysis.

Now that we identified the need for some form of the data transfer from MCU to a PC, we need to decide on the protocol for this communication. Article SerialProtocols Compared provides an overview of popular serial protocols.

• RS-232 (in its “modified” form adopted for MCUs) protocol is an obvious choice for implementing communication between MCU and PC the following reasons:
• Most MCUs (including my PIC) provide some form of a HW support for RS-232 communication in the form of UART modules. Actually, the PIC24EP512GP806, which I use for my board, supports four independent UART modules. 
•  Most PCs support RS-232 serial protocol either through dedicated or emulated (over USB) serial ports.
• There is a lot of third-party HW that supports RS-232 communication – FTDI cables, XBee wireless modules, Bluetooth modules, data loggers – just check Sparkfun.com.
• RS-232 considerably fast – for PIC-to-PIC communication over a short range or PIC-to-PC communication over the FTDI cable it is reasonable to boost the speed to 1 Mbps; for wireless communication, for example – over a pair of XBee modules, speed usually limited to 115.2 Kbps.
•  One MCU pin is required to implement RS-232 transmitter or two pins if you would like to implement HW Flow Control; I opt for the second choice.

Thus, my decision for this board was to use one of the available four UART modules to implement Serial Data Logger (SDLModule) interface. Project “11-SDL” tests SDL interface. However, prior to jumping into the implementation details, there are two issues that we need to discuss.

UART data stream is just a stream of bytes – serial protocol does not define any structure over this stream. From the logical perspective, we will be passing messages, so on the receiving end we need to know where one message ends and the new one begins so that we may correctly parse the stream and recreate the message even if some of the messages got distorted during transmission. In my implementation, I decided that every message would start with 0x5555 and end with 0xAAAA. For obvious reasons we could not use all 0s or all 1s (appear quite often in the data), but why this specific combination? No particular reason, except, maybe, that in both of these code words each successive bit is the inverse of the previous one (0x55 = 0b01010101 and 0xAA = 0b10101010), which results in a visually appealing and highly visible pattern in the logic analyzer.

Even at 115.2 Kbps transmitting 30-40 bytes takes between 2.5 and 3.5 msec – an eternity from the viewpoint of the processor running at 64 MIPS. During this time processor may execute about 20,000 commands – perform the whole iteration of the control loop! Thus, the implementation of data logging should be asynchronous with regard to the main execution thread with the maximum delegation of the whole transmission to the HW. Luckily, presence of the HW-based UART module together with the Interrupt system allows us easily achieve this goal on PIC.

Before we start discussing the code, a few words about the UART implementation on PICs and the timing errors. PIC supports 2 modes for UART timing – the so-called Standard mode and High-Speed mode. In Standard mode 16 timing pulses required for each bit, while in High-Speed mode only 4. The frequency of the timing pulses depends on the command frequency Fcy (which we set at 64 MHz) divided by the value in the Baud Rate Generator (BRG) register. More details on Baud rate generation and overall UART module operation provided in Microchip document UniversalAsynchronous Receiver Transmitter (UART). The important thing to remember is that with this mechanism most of the typical communication frequencies could not be defined exactly – there will be some error! However, certain amount of timing error is quite acceptable for serial communication – article TimingErrors in Serial Communication provides great background on the issue and defines the upper bound on the timing error acceptable for reliable communication.

Important thing to remember is that High-Speed mode allows for more precise setting for the communication frequency resulting in the lower error. I created an Excel file that calculates BRG values for typical communication speeds together with the respective errors for both UART modes assuming that the Fcy=64MHz.

Advantage of the Standard mode is that due to the larger number of timing pulses (16) the data line is sampled 3 times – on the 7^th, 8^th, and 9^th pulse significantly reducing potential for read error. With just the four timing pulses in the High-Speed mode multiple sampling is impractical. However, multiple sampling important only for the UART Receiver; as here we are implementing UART Transmitter I opted for High-Speed mode, which reduces the timing error.

Now we may start looking at the code of the SDL Module. Header file SDL_Profile.h provides definitions that allow to associate code with one of the four available on the MCU HW UART Modules. Most of the HW modules (UART including) of the PIC MCU are not tied permanently to some MCU pins – when needed for particular design, modules may be associated with respective pins using advanced feature – Peripheral Pin Select. _SDLInitPinMap function, defined in SDL_Profile.h implements exactly that.

Header SDL.h defines thre functions implemented by SDL module – SDLInit, SDLPostIfReady, and SDLPostWhenReady. The former one initializes the module specifying required Baud rate and the interrupt level to be used by the module. SDLPostIfReady initiates data transmission if the SDL module is Ready – that is the previous transmission has completed. SDLPostWhenReadywaits until the SDL module is Ready, and then initiates data transmission. Both functions copy the data to be transmitted into the internal buffer and then load the message header (0x5555) into the UART FIFO buffer, at which point they return to the main thread without waiting for the transmission to complete.

The rest of the data transmission is carried out in the UART interrupt routine, defined in SDL_ISR.c, which is invoked when the FIFO buffer is empty. Interrupt routine moves bytes from the internal buffer into the UART FIFO buffer and immediately returns control. If the internal buffer exhausted, interrupt routine pushes into the FIFO buffer message trailer (0xAAAA) and set the indicator that SDL Module completed transmission and is ready for the next one.

While the actual transmission of the message may take several milliseconds, the UART routine will be active just a few microseconds during this time, leaving the rest of the time for the execution of the main thread.

Project 11-SDL provides the test platform for the SDL Module and may serve as a very trivial sample of usage. In the later projects we will see more examples of the SDL usage.

Finally, a few words about the practical use of SDL module – it is not enough just to send data, we need to have some ways to receive and process it! In my development depending on specific requirements (speed vs. flexibility) I use either an FTDI 3.3 V cable to link UART from my board to the USB port on my computer or a pair of XBees – one connected to the UART on the board and the other – to again the USB port using XBee Explorer USB from Sparkfun.com. To collect detailed telemetry during the flight I use MicroSD data logger (SD-Logger-V2).

To make sense of this log data on a PC I developed two Windows-based programs – LogStreamer to read the log data directly from the USB port and LogConverter to read the log file from the MicroSD card. Both programs store the log as a tab-delimited text file ready for load into Excel. Both programs rely on a XML-based format file, which defines the format of the data in the log. SDLTest.xml is the format file to interpret the log produced by the 11-SDL project. You are welcome to use these programs under the MIT License.

The format template file can be quite elaborate and detailed discussion of these log processing routines might be a topic for a separate blog. However we will be using them quite often while working with the sensors and will be looking at more templates, so their structure and basic usage should become obvious.

This closes the logging chapter and now we may start looking at sensors. The first one will be HMC5983 magnetometer connected over the SPI bus, but that will be the topic of the next post.