Gas station without pumps

2016 March 31

PteroDAQ bug fixes

Filed under: Circuits course,Data acquisition — gasstationwithoutpumps @ 21:55
Tags: , , ,

I spent much of my lunch break today using a laptop borrowed from a student in the Applied Electronics course to debug a problem in PteroDAQ (one that I thought I had fixed on 2016 Feb 6 and again on 2016 March 30).

The problem was that on newer versions of Mac OS X, our original way for finding the serial ports and listing them in PteroDAQ (with good names) was failing, and the serial ports weren’t appearing. I fixed it for Mac 10.11.3, but that fix was breaking for most of the students in the course.  Yesterday, a student who is a friend of a student in the class suggested that the problem was just that I was only looking for USB 2 ports, and that the new USB stack used different data types for USB 3 ports.

Since I did not have any hardware with USB 3 ports (yes, I hang onto my computers for a while—the machine I’m typing this on is a MacBook Pro bought in mid-2009), I was unable to test the fix, so I released it to the class and asked them to test it.  A few students reported it working, but in the lab this morning, several students with Mac OS X computers were still unable to use PteroDAQ.  It was clearly a USB 2 vs. USB 3 problem still, since they were able to use PteroDAQ if they connected using my cable that has a USB 2 hub in it. One other student had gotten PteroDAQ to work for her by using a USB 2 hub that she had bought just for the purpose (probably a $7 or $8 one from Staples).

So instead of eating my lunch of raw vegetables, I sat with one of the laptops that was failing, uncommented some debug print statements that I had left in the code (I don’t delete debugging statements—I just comment them out, precisely for this sort of later bug fixing), and looked at the data structure for that Mac.  Apparently there are many different USB stacks for Mac OS X, with all sorts of differences. I put together ways of finding the appropriate name for the serial port using as many different methods as I could think of, based on either the child or the parent of the object in the chain that had the name I wanted.

After it was working on his computer, I handed it back to him and redid the bug fixes on my laptop and pushed them back to the bitbucket site, asking students to test the new code.  So far, everyone who has tested it has gotten it working (except for one student who seems to have a fried Teensy LC board—we were not able to get even the Arduino environment on my Mac to see his board). We don’t have any spare Teensy boards, so I suggested he order one from PJRC with the male headers already soldered on.  That makes two students (out of 48) who have had to order replacement boards (the other one soldered the male headers on the wrong side of the board, and delaminated some of the pads in attempting to correct the problem).  Perhaps next year we should order an extra 5% of Teensy LC boards, to cover for this sort of problem and have a local source for replacement boards.

I managed to finally finish my lunch after lab ended (around 5:30pm), but I’m glad to have gotten PteroDAQ working for (almost) everyone.  It is now 10pm, and time for me to prepare for tomorrow’s lecture.

2014 June 6

Wrapping up Applied Circuits and PteroDAQ bug fix

Today was the last day of the Applied Circuits class, and the students were turning in their 10th lab report.  It’s all over but the grading (though I expect a few more redone assignments to be turned in on Monday).

Today’s class started with my explaining a bug that was just found in the PteroDAQ code.

Symptoms: On the EKG assignment a number of students ran into trouble in the lab with the PteroDAQ software suddenly switching from normal recording to a sawtooth waveform, then spontaneously switching back after a while.  When plotting the saved data with gnuplot, the timestamps were all messed up in the part of the recording where the sparkline had displayed a sawtooth waveform, but returned to normal when the sparkline did.  We had never seen this behavior on  the Macs, nor had we seen it previously on the Windows machines.  The one new thing in this lab is that we were using a higher sampling rate on the Windows machines than we had previously used.

Diagnosis 1: The sawtooth waveform suggested to me that the low-order part of the timestamp was being taken as the signal, and the disordered timestamps as intrusion of the data into the timestamp.  I suspected that the old, slow Windows machines were not able to keep up with the USB serial traffic, and that they were losing a byte and getting a framing error.

I suggested this diagnosis to my son, but he showed me the framing character and checksum used to detect malformed packets, and pointed out that even if a malformed packed passed the checksum, synchronization would be re-established within a packet or two. Although the behavior certainly looks like a framing error, the packets should not go out of frame consistently.

Today, after watching the Shakespeare To Go 50-minute production of Hamlet with me, my son accompanied me to the lab to observe the behavior for himself.  We got the behavior very quickly with a 300Hz sampling rate, and the error message for a checksum failure was not printed. He added a print statement in the code where the data packets were getting their checksums checked, so that we could see what data was coming in. We looked at the transition from good behavior to bad, and noted that the packets were still the right length, but the information in the packet seemed to have been shifted. So there seemed to be a framing error, but not in the USB communication.

Diagnosis 2: This shift indicated that the problem was in the software running on the KL25Z board, rather than in the software running on the Windows host.  The packets to be sent out the USB serial port are stored in a queue in the KL25Z, and I conjectured that the queue was overflowing.  Since a circular buffer is used for the queue, writing when the queue is full would overwrite data for a different packet.  Because the queue length in bytes was not a multiple of the size of the items on the queue, this overwriting would be out of phase, and we would observe periodic switching between in-frame and out-of-frame packets, which is what we were observing.

We checked the code, and found that the test for sufficient room in the buffer for another packet had not been included in the routine for adding a new event to the queue.  The code to check the remaining space had been written, but not not used yet.  A fifteen-minute bug fix, while I went to my office to get my materials for my class, confirmed that this had indeed been the problem.  He will be pushing the fix to https://bitbucket.org/abe_k/pterodaq this evening.  The fix consists of discarding any event for which there is insufficient room in the queue.  Some loss of data is unavoidable when the queue fills up, and this delays the loss as much as possible, as well as minimizing how many events are discarded.  Since the events are timestamped, losing an event will often have little or no consequence for downstream analysis or plotting.

The first part of the class was explaining the bug to the students, which I had to do in a rather handwavy fashion, since most of the class has not had programming, much less queues and circular buffers.

We then talked about a variety of different things: deadlines for redos, when the t-shirts would be available, instructor feedback forms to fill out online, differences between bioengineering programs at the various UCs, what the job market was like for bioengineers, and other general advising sorts of things. I also reviewed some of the goals of the course in terms of engineering thinking, tinkering vs. engineering by design, the usefulness of models that aren’t perfectly accurate, trying to switch them from answer-getting to keeping in mind the meaning of what they were doing, sanity checks, methodical work and lab notebooks records, and that the real world trumps any models (“try it and see!”).

Incidentally, this year’s t-shirts have that motto on the shirts:

2014 Applied Circuits t-shirt has the design only on the front, and has the class slogan above the cyberslug.  (Click for high resolution image)

2014 Applied Circuits t-shirt has the design only on the front, and has the class slogan above the cyberslug. (Click for high resolution image)

When the students finally ran out of questions, I had about 10 minutes left, so I introduced them to the Wien-bridge oscillator, first by reminding them of the bridge-null condition (which them almost remembered from the quiz it had been on), then deriving that \omega RC =1 and R_{2} = 2 R_{1}. I even had time to tell them about how the amplitude of the original oscillators was regulated by having a light bulb as a non-linear resistor in place of R1.

After class I attempted to help one of the students get PteroDAQ working with his laptop, but we got as far as determining that the drivers were missing for the USB serial device on the KL25Z board, and that the Windows 7 driver installation did not fix the problem before he had to go to another class. My son and I will have to look for Windows 8 driver initialization that will set things up properly for the KL25Z board—this might be tricky as we have no access to a Windows 8 machine to test putative solutions on.

2013 July 24

Debugging in unfamiliar territory

Filed under: Printed Circuit Boards — gasstationwithoutpumps @ 00:21
Tags: , , , ,

Today, my son tried doing some programming on the KL25Z board, which I had been playing with yesterday (after reloading the MBED download software).  I has been using the MBED online compiler, but he wanted to use gcc, using entirely code he wrote himself.

Over the weekend he had figured out how to do the clock startup and he wanted to test that, as well as the whole gnu tool chain for programming KL25 chips, as he plans to use a KL25 chip rather than an ATMega chip in the product he is designing.

The clock startup is a rather complicated state machine on the KL25 processors, which start out using a 32kHz internal clock, then need to be switched via two intermediate states to using the external 8MHz crystal with the phase-locked loop.  The state diagram for legal transitions is given in Figure 24-16 of the KL25 Sub-Family Reference Manual:

Figure 24-16, the legal state transitions for the clock generator module.

Figure 24-16, the legal state transitions for the clock generator module.

We have to get from  FEI (Frequency-locked-loop Engaged with Internal clock) to PEE (Phase-locked-loop  Engaged with External Clock), pasing through FBE (Frequency-locked-loop Bypassed External) and PBE (Phase-locked-loop Bypassed External).  There are some tests that need to be done to make sure that the external oscillator has stabilized before making the transitions.

He figured out how to do this startup with very little help from me (I’ve never seen such a complicated startup procedure)—basically we read the manual together and I explained the idea of the state diagram, but I did not read all the details about what was needed to enable the transitions or what registers needed to be set to what values to make the transitions.

He also spent a lot of time figuring out how the gnu linker scripts work, so that he could put things (like the initial stack pointer and PC values) in specific places, without having to write assembly language.  I’ve never worked with linker scripts, so he was on his own for that!

Today when he tried downloading his first program (a simple color-change blinker that used a system tick interrupt for timing), it didn’t work.  He faced the first problem of doing embedded programming—a bug results in no output.  Figuring out what went wrong is very difficult when there is no response.  He turned to me for help debugging—despite the fact that I have no knowledge of ARM assembly language nor gnu linker scripts.

Somewhat surprisingly, I was able to help him.  Here are some things that I helped him with today:

  • Linker symbols are addresses of variables, not constant integers.  When you want the value of a linker symbol in a C program you need to use &symbol to get the address.
  • To disassemble raw binary files, you can use objdump, but you need to give lots of extra options, to make up for the lack of internal information:
    objdump -D -m arm -b binary --disassembler-options=force-thumb foo.bin
    I’ve not used objdump in a long time, and never with a raw binary—we had to read the objdump manual to figure out what options were available. He spent some time disassembling both MBED binaries and the ones he generated with the gnu gcc compiler and linker, looking for important differences. He noted that the machine code generated by gcc with the default optimization setting was difficult to read, but when he turned on optimization it got cleaner and simpler, using the standard instructions for subroutine return, for example.
  • Pointer arithmetic in C knows the size of the objects pointed to, so incrementing a uint32_t* pointer to the next word is done by ++, not by +=4. He had never used pointer arithmetic before, and had not realized that pointers and integers increment differently.
  • Linker scripts do not do link-time expressions, but require constants. He had a computed location (the start of RAM on the chip, which is at 0x20000000 – (ram_size/4) ), but the linker script was not doing the computation. When he switched to using the constant 0x1ffff000 in the linker script, the initialization code he’d written for clearing the BSS region then had the correct start location to clear.
  • I asked him if he had turned on clock gating for the GPIO module and the port module he was using.  (I think that GPIO clocks can’t be turned off, but that the port module clocks can be, disabling the port to save power. GPIO uses the platform clock, but the ports use the bus clock.)
  • In linker scripts, if a block has nothing in it, then the “.” pointer is not changed, even if the block should be in RAM rather than flash memory. His bss_start address was messed up when his data block was empty, because the “.” still pointed to the word after the end of the program. He fixed this by adding .=ram_start to the appropriate place in his linker script.

Of the things I helped him debug, about the only things I knew ahead of time were the pointer arithmetic and that linker symbols are addresses.  He was doing a lot of stuff that I’ve never done (like using gcc’s __attribute__ keyword to make symbols weak so that interrupt service routines could default to a null handler, but be overridden in the linker by defining a real interrupt service routine).

By the end of the day, he had written a blinky program using the System Tick Timer with his own startup code and interrupt vector table, written entirely in C, compiled and linked entirely with the gcc toolchain downloaded with the OpenSDA chip—and he got it all to work. This is about an order of magnitude or two harder than writing the same blinky program using the MBED software development kit and online compiler. He needed to read big chunks of the KL25 reference manual to figure everything out, and some parts we had to read together.

In fact, we’re still not sure whether there is any reason to enable the clocks to port A before doing the clock startup sequence, which the MBED code does.  As far as we can tell from the manual, the port starts out with the pins defaulting to XTAL0 and EXTAL0 (the crystal), so it shouldn’t be necessary to turn on port A, but the manual is not really very clear on this point. [Update: 2013 Jluy 24 00:45.  My son checked, you do not need to enable port A.  The external oscillator starts up fine with port A left turned off, as we thought.]

I doubt that I could have done an all-C, all-gnu program any quicker than he did—and he now knows far more about ARM processors than I ever will.  I was just pleased that I have enough general debugging experience to have been able to help him the few places he got stuck.

He plans to use the KL25Z board in building a prototype of the device he has been doing PC board layouts for all summer.  He’s going to make a through-hole-part board for prototyping the electronics.  The board will not include the processor, but will have lots of connectors .  The Freedom KL25Z board is far too big to fit in the final case, but having a debugged development board and a through-hole-part PC board for the other will let him debug his design more easily than the all-SMD board that will be needed for the final product (he’s aiming for a 5cm×5cm final board size including the processor and all peripherals).  When he does get the SMD board designed, he’ll be able to use the KL25Z board as a programmer for it, since there is a simple 10-pin connector on the KL25Z board that allows using the “OpenSDA” chip on the board to program any KL25 chip through the single-wire-debug interface.  He’ll put a similar connector on the boards he designs, so that he can load firmware onto the boards (and so that hackers can load their own programs, once the open-source design is released).

2013 July 4

Blinky EKG fixed

Filed under: Circuits course — gasstationwithoutpumps @ 18:26
Tags: , , , , , ,

In my earlier post today about the Blinky EKG, I wrote

About the only thing I can think of is that there is too large a DC offset between the EKG electrodes, as the Blinky EKG uses a large gain on the first stage and a relatively small gain on the second stage.  The DC-blocking high-pass filter is after the first stage. The EKG built on the protoboard used a smaller first-stage gain and larger second stage gain, so wouldn’t saturate the first stage as easily.  (I’d learned more about EKG electrodes by the time I’d designed that circuit.) I could fix the gains by changing a few resistors on the EKG blinky board, which may be worth the pain of unsoldering and resoldering resistors.  That may be worth trying today.

I finally got a chance to test that this afternoon.  I replaced the 470Ω Rgain resistor on the instrumentation amp with a 12kΩ resistor, reducing the first-stage gain from 175.2 to 11.667.  With this change, I could see my heart beat with no problem, if I turned the second-stage gain all the way up to 111, as high as the trimpot would let me go.  This gain was not enough to light the LED, though, so I replaced the 1kΩ resistor below the trimpot with a 100Ω resistor, which allows the second-stage gain to be adjusted from 10.9 to 1101.  The combined gain is thus adjustable from 127.2 to 12845.

UPDATE: 2013 July 5.  I realized this afternoon that I would have been better off leaving the 1kΩ resistor alone, and changing the 100kΩ resistor above the trimpot from 100kΩ to 510kΩ, giving me a second-stage gain varying from 47.36 to 521, and a total gain of 552.6 to 6078.

At high gain, the LED is always on, and at low gain the LED is always off, but when the gain is around 2300, the LED blinks nicely and the waveform recorded with the Arduino data logger is good.

I believe that the problems I’ve been having with the Blinky EKG board have been the result of DC bias in the EKG electrodes saturating the first stage, so reducing the gain of the first stage and increasing the gain of the second stage was the right fix. I’ve been thinking of redesigning the blinky EKG board to be more decorative (so that it could be worn as a pin or a pendant)—if I do that, I’ll certainly change the gain.

Blinky EKG hard to debug

Filed under: Circuits course — gasstationwithoutpumps @ 11:46
Tags: , , , , , , , ,

I’ve been having a frustrating couple of days trying to debug the Blinky EKG.  It worked when I first built it, but every time I’ve tried to demo it, the demo has failed, and I couldn’t get it working even at home on Tuesday.  I have another, very similar EKG circuit that I built on my instrumentation amp protoboard, which has worked fine every time I’ve tried to use it.

There are a few differences between the circuits (the blinky EKG is battery powered, for example, has an LED load on the final output, and has a trimpot for adjusting the gain of the second stage), but none that explain to me the difference in performance.

On Tuesday my experiments were limited to hooking up one or the other of the EKG boards and using an oscilloscope or Arduino data logger to observe the outputs or various internal signals.  Using the EKG board that worked convinced me that the EKG electrodes were providing a good 1mV signal (that had been a problem in the circuits course, as many students got EKG circuits that worked with electrodes on me, but no electrodes on themselves—we never figured out exactly why).  But I could not get anything from the blinky EKG—even the output of the instrumentation amp seemed to be constant.  I suspected that I had fried the amplifier chip, and was considering unsoldering it and putting in a new one.

Yesterday, I tried a different test, making an artificial input source, using resistors and my Elenco FG-500 function generator.

Test fixture for the EKG blinky board.  Note that with a 10V peak-to-peak oscillator input, the output would be a differential signal of about 1.8mV peak-to-peak.  The diagram was drawn with Digikey's SchemeIt.

Test fixture for the EKG blinky board. Note that with a 10V peak-to-peak oscillator input, the output would be a differential signal of about 1.8mV peak-to-peak. The diagram was drawn with Digikey’s SchemeIt.

With this test fixture, I convinced myself that the Blinky EKG board was amplifying the differential input signal correctly, over a range of about 1Hz to 40Hz, as long as the resistor for setting the DC bias was under about 300kΩ. Even with a 3.3MΩ resistor, I could see the output signal, but there was a fair amount of 60Hz noise added to it.  The gain was adjustable with the trimpot, but was high enough at all settings that I should be able to see EKG signals at the output clearly with the Arduino data logger—the gain control is mainly to get the LED to blink appropriately.

One effect I should have anticipated, but did not, was that the bias voltage showed a large change every time the LED turned on. If I redo the EKG Blinky design, I’ll probably use a voltage reference (like the TL431ILP) rather than just a voltage divider for the input to the Vbias op amp, and the LED will not be powered from the Vbias line.

In any case, the Blinky EKG board seems to be working as intended as an amplifier, and I’m still a bit mystified why it is not working when connected to the EKG electrodes.  About the only thing I can think of is that there is too large a DC offset between the EKG electrodes, as the Blinky EKG uses a large gain on the first stage and a relatively small gain on the second stage.  The DC-blocking high-pass filter is after the first stage. The EKG built on the protoboard used a smaller first-stage gain and larger second stage gain, so wouldn’t saturate the first stage as easily.  (I’d learned more about EKG electrodes by the time I’d designed that circuit.) I could fix the gains by changing a few resistors on the EKG blinky board, which may be worth the pain of unsoldering and resoldering resistors.  That may be worth trying today.

Note: I’m starting to use DigiKey’s SchemeIt for schematic capture, rather than Circuit Lab. There are a lot more symbols available in SchemeIt, and the user interface is fairly similar.  SchemeIt does not have simulation capabilities, but CircuitLab’s never worked for me anyway. SchemeIt’s  drawing is a bit cruder—they’ve not taken care to make sure that wires and components line up perfectly in the PNG output, but is better than Eagle‘s.  Best of all, I know how DigiKey monetizes their schematic capture system: you can turn the Bill of Materials (BOM) into a DigiKey order with a couple of clicks, so I have no expectation that they will start charging for SchemeIt.  I may even use the ordering capability in the way they intend, since I order from DigiKey fairly frequently already.

Next Page »

%d bloggers like this: