Serial Data Logging

When developing and debugging firmware for a MCU, which is performing some quite extensive processing of the data from multiple sensors, it is very important to be able to look at this data as it is being collected and processed. Similarly, when tuning, for example, PID parameters it is critical to capture the RC input, attitude data, and control signals to motors. All of this points to a need to capture data from the board and subsequently transfer it to a PC for analysis.

Now that we identified the need for some form of the data transfer from MCU to a PC, we need to decide on the protocol for this communication. Article SerialProtocols Compared provides an overview of popular serial protocols.

• RS-232 (in its “modified” form adopted for MCUs) protocol is an obvious choice for implementing communication between MCU and PC the following reasons:
• Most MCUs (including my PIC) provide some form of a HW support for RS-232 communication in the form of UART modules. Actually, the PIC24EP512GP806, which I use for my board, supports four independent UART modules. 
•  Most PCs support RS-232 serial protocol either through dedicated or emulated (over USB) serial ports.
• There is a lot of third-party HW that supports RS-232 communication – FTDI cables, XBee wireless modules, Bluetooth modules, data loggers – just check Sparkfun.com.
• RS-232 considerably fast – for PIC-to-PIC communication over a short range or PIC-to-PC communication over the FTDI cable it is reasonable to boost the speed to 1 Mbps; for wireless communication, for example – over a pair of XBee modules, speed usually limited to 115.2 Kbps.
•  One MCU pin is required to implement RS-232 transmitter or two pins if you would like to implement HW Flow Control; I opt for the second choice.

Thus, my decision for this board was to use one of the available four UART modules to implement Serial Data Logger (SDLModule) interface. Project “11-SDL” tests SDL interface. However, prior to jumping into the implementation details, there are two issues that we need to discuss.

UART data stream is just a stream of bytes – serial protocol does not define any structure over this stream. From the logical perspective, we will be passing messages, so on the receiving end we need to know where one message ends and the new one begins so that we may correctly parse the stream and recreate the message even if some of the messages got distorted during transmission. In my implementation, I decided that every message would start with 0x5555 and end with 0xAAAA. For obvious reasons we could not use all 0s or all 1s (appear quite often in the data), but why this specific combination? No particular reason, except, maybe, that in both of these code words each successive bit is the inverse of the previous one (0x55 = 0b01010101 and 0xAA = 0b10101010), which results in a visually appealing and highly visible pattern in the logic analyzer.

Even at 115.2 Kbps transmitting 30-40 bytes takes between 2.5 and 3.5 msec – an eternity from the viewpoint of the processor running at 64 MIPS. During this time processor may execute about 20,000 commands – perform the whole iteration of the control loop! Thus, the implementation of data logging should be asynchronous with regard to the main execution thread with the maximum delegation of the whole transmission to the HW. Luckily, presence of the HW-based UART module together with the Interrupt system allows us easily achieve this goal on PIC.

Before we start discussing the code, a few words about the UART implementation on PICs and the timing errors. PIC supports 2 modes for UART timing – the so-called Standard mode and High-Speed mode. In Standard mode 16 timing pulses required for each bit, while in High-Speed mode only 4. The frequency of the timing pulses depends on the command frequency Fcy (which we set at 64 MHz) divided by the value in the Baud Rate Generator (BRG) register. More details on Baud rate generation and overall UART module operation provided in Microchip document UniversalAsynchronous Receiver Transmitter (UART). The important thing to remember is that with this mechanism most of the typical communication frequencies could not be defined exactly – there will be some error! However, certain amount of timing error is quite acceptable for serial communication – article TimingErrors in Serial Communication provides great background on the issue and defines the upper bound on the timing error acceptable for reliable communication.

Important thing to remember is that High-Speed mode allows for more precise setting for the communication frequency resulting in the lower error. I created an Excel file that calculates BRG values for typical communication speeds together with the respective errors for both UART modes assuming that the Fcy=64MHz.

Advantage of the Standard mode is that due to the larger number of timing pulses (16) the data line is sampled 3 times – on the 7^th, 8^th, and 9^th pulse significantly reducing potential for read error. With just the four timing pulses in the High-Speed mode multiple sampling is impractical. However, multiple sampling important only for the UART Receiver; as here we are implementing UART Transmitter I opted for High-Speed mode, which reduces the timing error.

Now we may start looking at the code of the SDL Module. Header file SDL_Profile.h provides definitions that allow to associate code with one of the four available on the MCU HW UART Modules. Most of the HW modules (UART including) of the PIC MCU are not tied permanently to some MCU pins – when needed for particular design, modules may be associated with respective pins using advanced feature – Peripheral Pin Select. _SDLInitPinMap function, defined in SDL_Profile.h implements exactly that.

Header SDL.h defines thre functions implemented by SDL module – SDLInit, SDLPostIfReady, and SDLPostWhenReady. The former one initializes the module specifying required Baud rate and the interrupt level to be used by the module. SDLPostIfReady initiates data transmission if the SDL module is Ready – that is the previous transmission has completed. SDLPostWhenReadywaits until the SDL module is Ready, and then initiates data transmission. Both functions copy the data to be transmitted into the internal buffer and then load the message header (0x5555) into the UART FIFO buffer, at which point they return to the main thread without waiting for the transmission to complete.

The rest of the data transmission is carried out in the UART interrupt routine, defined in SDL_ISR.c, which is invoked when the FIFO buffer is empty. Interrupt routine moves bytes from the internal buffer into the UART FIFO buffer and immediately returns control. If the internal buffer exhausted, interrupt routine pushes into the FIFO buffer message trailer (0xAAAA) and set the indicator that SDL Module completed transmission and is ready for the next one.

While the actual transmission of the message may take several milliseconds, the UART routine will be active just a few microseconds during this time, leaving the rest of the time for the execution of the main thread.

Project 11-SDL provides the test platform for the SDL Module and may serve as a very trivial sample of usage. In the later projects we will see more examples of the SDL usage.

Finally, a few words about the practical use of SDL module – it is not enough just to send data, we need to have some ways to receive and process it! In my development depending on specific requirements (speed vs. flexibility) I use either an FTDI 3.3 V cable to link UART from my board to the USB port on my computer or a pair of XBees – one connected to the UART on the board and the other – to again the USB port using XBee Explorer USB from Sparkfun.com. To collect detailed telemetry during the flight I use MicroSD data logger (SD-Logger-V2).

To make sense of this log data on a PC I developed two Windows-based programs – LogStreamer to read the log data directly from the USB port and LogConverter to read the log file from the MicroSD card. Both programs store the log as a tab-delimited text file ready for load into Excel. Both programs rely on a XML-based format file, which defines the format of the data in the log. SDLTest.xml is the format file to interpret the log produced by the 11-SDL project. You are welcome to use these programs under the MIT License.

The format template file can be quite elaborate and detailed discussion of these log processing routines might be a topic for a separate blog. However we will be using them quite often while working with the sensors and will be looking at more templates, so their structure and basic usage should become obvious.

This closes the logging chapter and now we may start looking at sensors. The first one will be HMC5983 magnetometer connected over the SPI bus, but that will be the topic of the next post.

The first project – Initialization and Timing

When I start with the new board, I develop firmware in successive steps with each step represented as an MPLab-X project. The first project that I do is usually the “10-Basic”. The goal of this project is to establish basic MCU initialization, configure oscillator to set the speed of MCU (INIT module), and develop (or migrate from the previous release) timing service (TMR module) and basic “information” service to control on-board LED and/or external LED/buzzer (BLI Module).

The first step is to decide at what speed we would run the MCU. PIC24EP series of MCUs could be configured to run at a speed of up to 70 MHz. This is the instruction cycle speed, which usually designated as Fcy. The first impulse would be to configure it to run at maximum allowed speed – that is at Fcy = 70 MHz. However, Fcy plays critical role in configuring other timing parameters – the frequencies of stand-alone timers as well as frequencies of multiple peripherals including UART, I2C, and SPI speed. These frequencies are defined as fractions of Fcy and these fractions are most often the powers of 2. Taking these fractions of 70 MHz will result in very odd numbers. For example, if we set a divider (usually called “prescaler”) for a timer to, let’s say, 64, the timer update interval will be 1.09375 usec – not an impossible number to work with, but also not an easy one to perform precise time calculations.

Thus, my decision was to reduce the speed of MCU by about 9% to 64 MHz for the sake of simplifying various timing calculations. This also provides some safety margin as the higher the frequency of MCU operation the more stringent are the requirements to the operation temperature. At 64 MHz (or MIPS) PIC24EP can withstand temperatures up to about 115 degrees Celsius, while at 70 MIPS it is limited to 85 degrees.

The board is equipped with the 12 MHz crystal oscillator. However, PIC MCU provides a PLL frequency multiplier and a pre-scaler and post-scaler registers so it is quite easy to convert any crystal frequency to almost any Fcy subject to some limitations for intermediate frequencies. My selections for the values of these parameters are documented in the Init.c file. If you are interested in specifics od oscillator configuration, they are pretty well documented in the Family Reference Manual for respective MCU family available at www.microchip.com.

The board has 4-position DIP switch – the initialization routine (Init.c) configures associated pins and captures their values. As this pins are read and their values set only during initialization, so flipping them over after the board already powered up does not have any effect.

Besides configuring the oscillator and reading switches, Init.c performs some other initialization functions – configures MCU global control registers (“fuses” in PIC terminology), disables all HW modules (like I2C, SPI, etc.), disable all interrupts while enabling interrupt nesting, configures all analog-capable pins as digital. Initialization interface defined in Init.h and switches values are exposed through Switches.h.

It is critical for the flight controller to have access to precise timestamps to be able to calculate derivatives and integrals of the sensors’ values, as well as rather useful to have access to other timing features, such as well-defined delays (“sleeps”) etc. This functionality is provided by the TMR module. Timer module utilizes three HW timers – 16-bit Timer1 for low-precision functions and a pair of even-odd 16-bit timers to create one high-precision timer. The high-precision timer provides timestamps with the resolution of 1 usec, which is sufficient for the flight controller. TMRProfile.h file defines which pair of the available HW timers should be combined into a 32-bit high-precision timer.

Timer1 is configured to generate interrupt every 125 usec; the interrupt routine increments 32-bit counter. This counter of 125 usec intervals, which I call “ticks”, is used to implement “delays” and “alarms”. Let’s say, you have an operation that may take from 3 to 5 msec depending on values of some parameters and other conditions, that you perform in a loop. However, you would like to repeat this loop exactly every 10 msec, so, depending on the duration of calculation in the loop, at the end of the loop you need to implement a variable delay from 5 to 7 msec. This can be easily achieved with the “alarm” – at the beginning of each iteration of the loop you set alarm for 10 msec and then perform the required calculation, at the end of the calculation before ending the iteration you wait for the pre-set alarm – bingo! Each iteration of the loop will take exactly 10 msec!

Another very important feature provided by the Timer module is the callback after a certain period of time. TMR.h provides prototype for the callback function, which should be implemented by the user. TMRCallBackAfter function allows to register user’s callback function with the Timer module and specifies when it should be called. When the required interval passes, the callback function will be invoked on the interrupt routine asynchronously to the execution of the main code. As usual with the interrupt routines, callback function should not perform any lengthy operations as it will block main execution thread for the duration of the callback. Active callback request can be cancelled by calling TMRCallBackDiscard function. My current implementation allows just for one active callback at any given time; however it would be quite easy to change the code to allow multiple active callbacks – just did not see practical use for it thus far. BLI module, which we will discuss shortly, provides an example of the need for the callback functionality.

BLI Module (“blinker”) provides functions to control on-board LED and/or external LED/buzzer. The MCU pin, associated with the LED control, is defined in BLILocal.h – did not see a need to create a separate “profile” header for this simple definitionJ. Blinker module provide functions to directly control LED (and buzzer, if connected) – BLISignalON, BLISignalOFF, and BLISignalFlip.

When the flight controller performs some lengthy operation, like initialization or calibration of sensors, or waiting for “arming”, or waiting for GPS to obtain first fix, it would be nice to inform user that an operation is taking place by blinking the signal LED at certain frequency or in certain pattern. It is possible to achieve this with the functions providing direct control over the LED, but would be extremely cumbersome. This is where Timer callbacks come very handy!

Blinker module may set the LED control and then subscribe for a Timer callback. In the callback function, which will interrupt the main routine performing the lengthy operation for a very short time, Blinker may flip the LED and re-subscribe for the next callback. BLIAsyncStart function defines blinking pattern, subscribes for callback, and immediately returns so that the main thread may initiate some lengthy operation, during which the Blinker will happily blink requested pattern on its own. When the lengthy operation completes, blinking can be stopped by a call to BLIAsyncStop.

This worked fine for me for quite some time until I exhausted my fantasy coming up with distinctive blinking patterns. To address this problem, I added to the Blinker the Morse table and BLIAsyncMorse function, which accepts as a blinking pattern some character string and then blinks it asynchronously. Morse blinking can be terminated by the call to BLIAsyncStop.

These three modules – INIT, Timer, and Blinker – are the first I implement for any new board. Project 10-Basic test the functionality of these three modules and allows to build upon them in the further development.

In the next post we will look at the asynchronous implementation of the serial UART data sender.

The board

There were several design requirements that I tried to implement while going from board Version 2 to Version 3, which we are discussing here:

• Standardize size.

When I was working on Version 2 board, I was not aware about specific size limitations – there was no standardized quad frames available, so I picked some size for the board rather randomly. With the Version 3 I decided to make the board compatible with the broad range of available frames by matching the size and mounting holes of the KK and MultiWii flight controllers.

• Add more and upgrade existing on-board sensors.

Added MPL3115A barometric altimeter and replaced HMC-5883 magnetometer with the next generation HMC-5983.

• Increase number of PWM outputs (from 4 to 8). 

With the idea to control quad-, hexa-, or octa-copter and/or provide feed for camera orientation control.

• Provide better vibration rejection.

Insulating mounting like rubber bumpers, vibration tape, etc. are traditional mechanism to eliminate vibration effect on the board. In this design, I am attempting to address vibration (the high-frequency one) by using two MPU-6050 sensors positioned at 45 degrees to each other and +/- 22.5 degrees to the axis of the board. I also expect to have more precise estimate of the vertical acceleration to be able to provide independent estimate for altitude.

• Provide GPS port.

MOLEX PicoBlade 1.25 pitch 4-pin connector provided on board to connect GPS; connector provides ground, +3.3V, TX, and RX lines.

• Provide US-100 port.

JST ZH 1.5 pitch 4-pin connector provided on board for US-100 ultrasonic range sensor; connector provides ground, +3.3V, TX, and RX lines.

• Serial port for RC receiver.

With this board I plan to use either UART-enabled receiver or a satellite receiver as my primary. I did some analysis of this configuration and posted my findings in a blog post at http://diydrones.com.

• DIP Switch.

4-channel half-pitch SMT switch to control some of the flight controller configuration parameters.

Putting all of these requirements together, I came up with the following board design:

The board is a 4-layer one with internal layers used only for Ground and power; all routing is done in the top and bottom layers. The board provides taps to connect logic analyzer to each of the 2 I2C buses for debugging and testing. Respective interrupt lines (from sensors) are also routed to these taps.

Eagle project files for this board and schematic printout are provided in my repository at Google Code.

Now we can power up the board and perform initial configuration, which will be the topic of the next post.

Control board and firmware architecture

First thing first, all my boards and all of my code are for Microchip PIC 16-bit microprocessors. Sorry, Atmel AVR and Arduino fans, you would not find much of interest here, except, maybe, something related to sensors or PID controller L

Why PIC? I simply got used to it – I started working on the flight controller idea about 10 years ago and somebody in my flight club recommended PIC, so I started with PIC MCUs, got quite comfortable with them and they are fully capable of addressing all of my needs. If you are interested in some of my previous iterations, check the Project History page.

I definitely do not imply that one architecture is better than the other – if you would like to see a great comparison of Microchip PIC and Atmel AVR MCUs, check the link http://www.eevblog.com/2010/02/22/eevblog-63-microchip-pic-vs-atmel-avr/ and if you have a strong preference and would like to get into arguments, please argue with the author of that post. I am impartial J

Now that we are over the MCU hurdle, let’s talk about the code. Code is developed using Microchip’s MPLab-X IDE and XC16 compiler – both products are freely available from www.microchip.com (the free version of XC16 compiler does not perform optimization of code beyond the basics). I work on Windows platform, so if you are a MAC or LINUX user (MPLab-X and XC16 available for all platforms) and would like to compile my code on your machine you may have to replace all “\” with “/” in all of the #include directives and in directory references in the Project Properties in MPLab-X.

Now, about the code itself – ALL of my communication libraries (UART, I2C, SPI, PWM for motor control, RC Receiver communication, etc.) are interrupt-based! There is no time lost on waiting in my control loop as all the communication with sensors and other peripheral devices happens in parallel with the computations in the Interrupt routines (ICRs). This leaves me a lot of time for calculations, so I use floating-point arithmetic for all calculations without the need to resort to time saving tricks like doing calculations using fixed-point or Q.15 or Q.31 arithmetic with subsequent normalization. I can also freely use trigonometric function and matrix operations without any concern to how much time they use! 

Time savings produced by implementing asynchronous IO are so high that without any special optimization my control loop on the Version 2 board (previous generation using 40 MHz PIC24HJ MCU), which implements self-stabilization, course lock, battery management, etc., runs at 400 Hz. I actually had to insert additional wait into the control loop to bring it down to 100 Hz because my data logger (the “black box” in aviation terminology) could not keep up with the data stream!

So, even if you are not an enthusiast of flight and do not care about the multicopters, you still may find some interesting code snippets in this blog. For example, my I2C library allows for multiple slave devices on the same bus; respective control programs may acquire bus for exclusive use in the asynchronous mode by implementing interrupt callbacks. More about this in the I2C section later.

Another interesting feature of the Version 3 design is that the board uses two MPU-6050 6-DOF (3-axis gyro and 3-axis accelerometer) sensors. Sensors are oriented at 45 degrees to each other – +/- 22.5 degree in reference to the X-axis of the board. Thus, the Roll and Pitch calculations are based upon fusion of four measurements from each sensor; Yaw and vertical acceleration are calculated based upon the average of the measurements from both sensors. Sensors are physically located roughly symmetrically around the center of the board, but quite apart from each other. My expectations are that this design will reduce sensitivity of attitude calculations to vibration, which is a serious issue for multi- and helicopters. However, whether these expectations will materialize or I just wasted a perfectly usable second sensor – we will see when we get there.

Now it is probably the right time to have a look at the board, which we will do in the next post.

Introduction

Hi! Thanks for stopping by.

Your first question probably would be “What is this blog about?” As the name implies, it is about building from scratch quadro- or multi-copter flight control board – the “brain inside the machine” that is responsible for letting it fly.

There is a multitude of flight control boards on the market today – AeroQuad, ArduCopter, DJI, KK, MultiWii - just to name a few. A rather large list of these boards provided at http://robot-kingdom.com/best-flight-controller-for-quadcopter-and-multicopter/, but if you search Google, you probably find many more. So if your interest is in just flying a multicopter or building one and then flying it – you would be much better off just buying either a kit or a RTF set.

This blog goes into the details of building the board and developing firmware for it from scratch! That is from the bare-bone MCU – no external libraries, no black-box code – everything is here and everything is explained. All the code that will be discussed here is available at https://code.google.com/p/custom-pic-quad/ under the Open Source MIT License – look at it or use it at your own discretion WITHOUT ANY WARRANTY; without even the implied warranty of   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The version of the board and corresponding firmware that will be discussed here is for the Version 3 of the board, which I am working on now. Thus, I will be publishing new articles for this blog as my development effort progresses. Is there a promise that I will conclude this effort and do not leave you, my readers, halfway there? I think so. I have been working on flight control boards since about 2004. While the first couple of revisions (starting with Version 0) were total failure, I finally succeeded with the Version 2 (code and Eagle files for the board itself are at the link above), which happily carries my quad in the air - https://www.youtube.com/watch?v=kEo5DSe7PDg.  If you are interested, more details are provided on the Project History page.

So why don’t I blog about the working Version 2? Oh, it is boring! Since then I got some new idea, improved code, designed much better board, etc. – so why bother with the old stuff when we can together delve into the new one?

Another reason for this blog is rather selfish – while working on Version 2 I implemented a lot of calibration of the sensors like calibration of magnetometer for Hard and Soft-iron interference (Application notes AN4246 and AN4248 from http://www.freescale.com/) and temperature compensation and calibration of InvenSense IMU MPU-6050. Yes, I have some Excel spreadsheets hanging around from that time, but I did not document all of those efforts properly to assist me in calibrating sensors on my new board. Now I plan to document calibrating sensors on my new board in this blog, so when I get to the next iteration (maybe, Version 4 or 5) I will have all my notes readily documented in this blog! This also applies to sensor fusion, PID tuning, etc.

If you stayed with me up to here, you probably would be interested in what is so special about my project and what is the overall architecture of my board and the corresponding firmware? That will be the topic of the next post.