First week of class W2020

I’ve just finished the first week of class for BME 51A (Applied Electronics for Bioengineers), and it has been a rather hectic week.

First, I had about 5 people sign up for the class at the last minute (from 56 to 61), which drove the BELS staff to having to do a lot of expensive last-minute purchasing of additional items for the parts kits.  It also meant that when we handed out the parts kits on Tuesday, several of them were incomplete and so more items had to be handed out during the Thursday lab period.  Since getting their kits, 2 students have dropped taking us down to 59, and one other has told me that they might drop (assuming that they can get a coherent plan together for an independent study to replace the course).  This enrollment is less than last year’s 82 students, but still a fairly large number, given that the BELS labs can only accommodate about 24 students.  I’m using two lab rooms for the first section and one for the second section, and have hired 3 group tutors (one for each lab room and time) to work along with me in the labs.

The first week’s labs consist of soldering headers onto the Teensy LC boards and installing software: Python with SciPy (using the Anaconda install), gnuplot, Arduino+Teensyduino, PteroDAQ.  For the first time in years, everyone got their soldering done within the allotted lab times!  I don’t know what was different this year that made it work as it was supposed to.  One thing that helped a little was that when students soldered the male headers on the wrong side of the boards (which happens every year, despite all the written, pictorial, and verbal warnings I can give), I had them cut apart the male headers to unsolder, then gave them new male headers to solder on.

Software installation is always a problem, because students are attempting to install on a variety of platforms, and something is always incompatible. Here are some problems that came up this year:

• Gnuplot on old Mac OS X systems would not install with homebrew, as qt could not be installed—it insisted that you needed the full Xcode (not just the command-line subset), but Apple will not provide the full Xcode for such old, unsupported systems.  There may be already compiled versions of gnuplot for these old systems, but I have not gone looking for them. I’m not sure what to recommend to these students, other than updating to a system that Apple still supports.  I can’t recommend updating to Catalina, as they will lose all 32-bit apps, some of which will never be ported to newer versions of macos (for example, I lost access to Finale Notepad, which is no longer available on Macs—probably because Apple makes it so very difficult for third-party software developers to maintain their code).
• Gnuplot on new Mac OS X systems installs and runs, but produces bad PDF files.  The workaround for this was posted at https://stackoverflow.com/questions/57698204/gnuplot-pdf-terminal-exhibits-font-issues-on-mac. The answer to that stackoverflow question had two lines to fix the problem:

  brew uninstall --ignore-dependencies pango
  brew install iltommi/brews/pango
  

  The problem is that the default homebrew “bottle” for gnuplot points to a broken version of pango, and you have to remove that version and install an older version.  This one I knew about, because it had bitten me when I was trying to finish the book in December.

• The standard Arduino+Teensyduino installation fails on macos Catalina (10.15). I knew about this one also, as I had tested the PteroDAQ install after I had “upgraded” to Catalina (which broke nearly everything).  The workaround is a beta release of Teensyduino (currently https://forum.pjrc.com/threads/59030-Teensyduino-1-49-Beta-5), which is pointed to from the Teensyduino installation page, but many of the students with macos Catalina did not notice the pointer.  One nice feature of the Teensyduino beta release is that it includes the Arduino IDE, so that you don’t need to install Arduino first.
• The PteroDAQ GUI seems to be crashing on macos Mojave (10.14)—logging the user out when they try to run “python daq” as usual.  I think that this may be a weird filesystem thing, as it seemed to make a difference whether PteroDAQ had been cloned from GitHub or downloaded as a zip file, and whether the files were on the hard drive or on a Google Drive.  The only thing that seemed to work consistently was to clone the files from GitHub onto the hard drive of the laptop, then run the app that is in pterodaq/extras/maclauncher/ .  Why the launcher app works, when invoking “python daq” directly from a terminal results in logging the user out, is a mystery to me.
• On macos Catalina, PteroDAQ can’t find the board if it is connected to the laptop through a USB2 hub connected to a USB3 port (I ran into this problem on my laptop before classes started also).  Apple has once again scrambled their USB stack, and USB Serial no longer seems to be visible through a USB2 hub (though the USB device is visible, the serial interface does not seem to be).  This hasn’t caused a problem for any students yet, as direct connection to a USB3 port works ok, and connection from hubs on USBC ports seems to be ok (probably because they are USB3 and not USB2).
• One student had trouble getting PteroDAQ to run on his Windows 8 machine, with the python program crashing on trying to access the list of ports.  No one else with Windows 8 was having trouble, and his machine was running extremely slowly (slower even than the ancient “Barbie” laptop that I used for testing Windows implementations of PteroDAQ), so I suspect he was having hardware or malware problems on his machine.

I think that nearly all the students got their software installed—at any rate, people left the lab early on Thursday and were not queuing up for installation help, as they had in previous years.  Only one student asked for installation help during Friday office hours (the one with the probably bad hardware or malware), and there were no requests on Piazza for help.  If I’m right that everyone got things installed, then we are well ahead of previous years.

On Thursday, before lab, one of the group tutors ran a tutorial session on , which about half the class attended.  That should help somewhat in the prelabs to be turned in on Monday.

I’ve changed grading logistics this year.  Rather than hiring just 2 graders, I hired 6, and rather than having them work whenever they have time, I have scheduled grading sessions where we get together and grade in the same room.  On Thursday, we took just over 2 hours (with 5 graders, counting me) to grade homework 1, and today 6 of us got homework 2 graded in about an hour.  I graded the first quiz by myself Wednesday night.  So far, we have managed to keep the turn-around time to about a day, which is much better than last year, when the 2 graders were overloaded.

Monday’s assignments will be stressing our grading system a bit, as we have a lot coming in, but I couldn’t schedule a grading session until Wednesday at noon.  I’m hoping that we can get both homework 3 and the first prelab graded before Thursday’s lab.  If necessary, we can just give a turned-in/not-turned-in grade for the first prelab, as they are turning in a more complete draft of the same prelab on Friday, but I’d much rather give them feedback so that they can correct mistakes before the Friday draft.

So far, the quiz and homework scores look pretty good, so I’m hopeful that this will be a high-performing class this year.  It would be really nice to give out more B+ (and even A-) grades and fewer C grades.  I’m keeping my fingers crossed (even though that makes typing the blog post hard).

OS 10.11.3 takes forever to install

I decided to upgrade the household iMac to 10.11.3 (from 10.8, I think), but not because 10.11 offered any valuable new features.  In fact, just the opposite—a student in my freshman design seminar was having trouble getting PteroDAQ to work on 10.11.3, and I need to debug the problem.

It took well over an hour to download the upgrade—6.9Gbytes!  All the text in Wikipedia could be transmitted in about 9GBytes.  What junk is Apple loading my computer with? Sometimes I long for the days when the operating system fit on a 700kB floppy disk.

After downloading, it supposedly takes about half an hour to install (I’m still waiting for that to finish). [Update: after over half an hour, I went to check on it, and it now claims 45 minutes remaining.] I don’t know how much fussing I’ll need to do to get the new OS working—there always seems to be something broken on each new upgrade of the system.  All that time, before I can even look at what they broke/changed in OS 10.11 to make PteroDAQ not work.

Because the Arduino software was able to download to the board, but PteroDAQ was not then able to see the board, I suspect that port listing method has changed, and PteroDAQ code will need to be updated to do something different in OS 10.11 than in the earlier Mac OS versions (there is already very different code for Mac, Linux, and Windows, which share almost nothing in how to list the USB ports—we may need two different Mac code versions).

I was able to solve one student’s problems with the Arduino system losing track of ports, by suggesting he update from 10.11 to 10.11.3.  It seems like the 10.11 release had a new USB stack, which Apple’s was beta-testing on live customers rather than spending the time to do their own QA.  (This didn’t use to be such a problem with Mac releases, but has always been a problem with iOS releases—it seems that bad software engineering practice is driving out good at Apple.)  I checked, and the student having trouble with PteroDAQ did have 10.11.3, so the problem is not so easily resolved as the Arduino problem was.

Getting KL25Z bootloader to work without Windows 7

The one thing I hate about the Freedom KL25Z boards is that they are shipped with a bootloader that doesn’t work.  More precisely, they are shipped with a bootloader that works only with Windows 7, not Mac OS X, not Linux, and not even Windows 8.  (Note: never get P&E Micro to do any software work for you—they are responsible for this crappy bootloader.)  There is a newer bootloader from P&E Micro, but you need a Windows 7 computer to download it, so that does no good at all (it’s worse than that: see below).

I bought a new KL25Z board recently, to replace one that my son borrowed for Futuristic Lights, and soldered up the headers today, and was ready to test it out. I was hoping that they were now using one of the newer P&E bootloaders, that supposedly works on Macs (I’ve never seen a P&E bootloader that works with Macs, but I was willing to believe that P&E got enough complaints that they eventually fixed their bugs). Unfortunately, what I got is

MicroBoot Kernel Version is: 1.05
Bootloader Version is: 1.09

but P&E Micro reports

Bootloader versions 1.10 and earlier are not allowing firmware update and MSD FLASH programming on my OpenSDA board, with the Linux, MacOS, or Windows 8/8.1 operating systems.

I understand that on some of their newer boards, Freescale is using the mbed.org bootloader, which is the one I want to install on my KL25Z board. My choices at this point seem to be:

• Take the 6.5-mile bicycle ride round-trip to campus to use an ancient Windows 7 machine from the circuits lab (this is what I’ve done in the past).
• Buy a Windows 7 laptop to use at home, for initializing KL25Z boards and for testing PteroDAQ multi-platform support.
• Look on the web for a workaround.
• Ask my son if he found a workaround for his Linux laptop.

I’m too lazy to bike to campus (I did that cycle ride 4 times in the last 6 days), and I can’t get a laptop delivered instantly (and it will take quite a bit to put aside my aversion to Windows enough to buy even an $85 used laptop with Windows). That leave the web search and asking my son.

I did find a workaround on the web for Windows 8 [https://community.freescale.com/message/440982#440982]:

Configure “Do not allow locations on removable drives to be added to libraries”  as discussed here: http://answers.microsoft.com/en-us/windows/forum/windows8_1-hardware/how-do-i-prevent-system-volume-information-files/815b0046-d631-4419-a43e-44083a3733f5 and you should be able to do updates from Windows 8.1.

I don’t know whether this workaround works, as I don’t have access to a Windows 8 machine to test it on.

My son pointed me to a workaround that has been posted for Linux systems [http://karibe.co.ke/2014/04/changing-the-firmware-on-freescale-freedom-boards-in-linux/], and that he has used successfully. I tried that on Mac OS X, but the utilities for manipulating mounting of disks is different—you need to use ‘diskutil unmount’ and there is no “modprobe”.

I tried doing the closest corresponding actions on Mac OS X, but they did not work.  I could unmount and mount the disk easily enough (it showed up as /dev/disk1, which can be most easily discovered with ‘diskutil list’), but copying the mbed sda file to the disk still had no effect.

We tried the original script from karibe.co.ke on my son’s Linux laptop, and it worked ok.  We could install the mbed.org firmware (from https://developer.mbed.org/handbook/Firmware-FRDM-KL25Z) using the Linux script with the unmount/mount trick. The mbed software worked as usual to download PteroDAQ to the board.

I think tried some more experimenting. We updated the Bootloader to v1.11, using the latest download from P&E Micro, as pointed to by their OpenSDA page. The update worked, to the extent that the Bootloader reported being v1.11, instead of 1.09, but despite P&E’s claims, it still did not work with Mac OS X. I could not download P&E micro .SDA files nor  the 20140530_k20dx128_kl25z_if_opensda.s19 file from mbed.org.

So we used my son’s Linux laptop again to put the mbed software on the KL25Z board, and I’ll have to be carefully not to get into Bootloader mode unless either he’s around with his laptop or I’m willing to cycle to campus.  (Or I break down and get a junky Windows laptop just for rebooting KL25Z boards.)

My son has been suggesting that we get a junky Windows box, so that he can test out the multi-platform features of PteroDAQ without having to cycle to campus with me.  I might do that while he is away at Ashland next week, if I can find a cheap enough laptop that I believe will actually function.

Mac OS 10.6.8 kernel bug

Today I managed to tickle a rather nasty kernel bug in Mac OS 10.6.8 on my laptop.  The bug resulted in unkillable zombie processes—including one stuck in a busy-wait loop that took up 100% of the CPU (in kernel mode). You can’t kill the processes (not even “kill -9” works), and you can’t shut down or restart the computer—only power cycling works.

Here is how I tickled the bug:

1. I had a KL25Z board plugged into a USB slot, providing data at 115200 baud.
2. I started a python program that read the stream from the USB and echoed it to the terminal window:

   #!/usr/bin/env python
   
   from __future__ import print_function,division
   from glob import glob
   from serial import Serial
   
   USB_serial_ports = glob("/dev/tty.usb*")
   print ("# Possible ports = {}".format(USB_serial_ports))
   usb_serial = Serial(USB_serial_ports[0],baudrate=115200,timeout=5);
   print ("# Opened {}".format(usb_serial))
   
   try:
       for line in usb_serial:
          print (line.strip())
   except KeyboardInterrupt:
       usb_serial.close()
   

3. I used ^C to kill the python program and close the port.  So far, everything was working great.  The program echoed the stream from the KL25Z board nicely to the terminal window, and the try-except block caught the ^C interrupt and (presumably) closed the port.
4. I started the python program again.  This time, it was unable to open the USB serial port (never getting to the “Opened” print statement), and could not be killed. The Activity Monitor showed that the kernel was using 100% of a CPU core (I have a dual-core MacBook Pro, so this was only half the available CPU). I tried ^C, Force Quit, and “kill -9” with the process number of the Python process—none had any effect. ^Z claimed that the process was suspended, but the kernel was still in a busy-wait loop, and attempting to kill the suspended process had no effect on either the kernel busy-wait or the Python process.
5. Unplugging the USB cable stopped the kernel busy-wait loop (at least the system CPU usage dropped from 100% down to 2%), but the Python process was still unkillable.

I could do this cycle repeatedly, since each time I plugged in the cable to the KL25Z board the Mac assigned a new device number, and I could thereby build up a lot of unkillable Python processes.  Eventually, I got tired of the large number of unkillable processes, and tried restarting the computer.  It got to the blue screen, then spun the waiting icon forever (well, longer than I was willing to wait), so I had to use option-power to turn off the computer.  It rebooted fine.

I have so such trouble with the Arduino Duemilanove board (which resets whenever the USB serial port is opened).  The same Python program also works fine with the Arduino Leonardo board, which does not reset.  I can run my little echoing program, ^C it, and run it again to continue getting the stream from the board.

There must be some difference between how the Leonardo sets up the USB serial connection and how the KL25Z board does it—a difference that causes a kernel busy-wait when reopening the stream from the KL25Z, but not when re-opening the stream from the Leonardo.  Since the default serial setup is the same for both, I’m a little mystified what that could be.

Of course, I’m now left with a bit of a dilemma—how do I record data from the KL25Z board without having to unplug and replug the USB cable for every event I want to record?

I looked around the Web to see if anyone else had similar problems.  It does seem to have been a problem with OS 10.6 in other USB contexts:

http://support.plugable.com/plugable/topics/baud_rate_switches_cause_driver_hangs_in_the_kernel

http://stackoverflow.com/questions/4064832/open-function-hangs-never-returns-when-trying-to-open-serial-port-in-mac-os

And the stackoverflow answers suggest that the problem is buggy device driver, but I’ve no idea which device driver either the Leonardo or the KL25Z board (with MBED firmware) causes to be used on the Mac.  Are they using different drivers?  How do I find out?

 

According to the USB Prober application, the KL25Z board opens with

           4: MBED CMSIS-DAP@4100000  <class IOUSBDevice>
               AppleUSBCDC  <class AppleUSBCDC>
               USB_MSC@0  <class IOUSBInterface>
                   IOUSBMassStorageClass  <class IOUSBMassStorageClass>
               IOUSBInterface@1  <class IOUSBInterface>
                   AppleUSBCDCACMControl  <class AppleUSBCDCACMControl>
               IOUSBInterface@2  <class IOUSBInterface>
                   AppleUSBCDCACMData  <class AppleUSBCDCACMData>
                       IOModemSerialStreamSync  <class IOModemSerialStreamSync>
               MBED CMSIS-DAP@3  <class IOUSBInterface>
                   IOUSBHIDDriver  <class IOUSBHIDDriver>
                       IOHIDInterface  <class IOHIDInterface>

while the Leonardo opens with

           3: Arduino Leonardo@6200000  <class IOUSBDevice>
               AppleUSBCDC  <class AppleUSBCDC>
               IOUSBInterface@0  <class IOUSBInterface>
                   AppleUSBCDCACMControl  <class AppleUSBCDCACMControl>
               IOUSBInterface@1  <class IOUSBInterface>
                   AppleUSBCDCACMData  <class AppleUSBCDCACMData>
                       IOModemSerialStreamSync  <class IOModemSerialStreamSync>
               IOUSBInterface@2  <class IOUSBInterface>
                   IOUSBHIDDriver  <class IOUSBHIDDriver>
                       IOHIDInterface  <class IOHIDInterface>

The biggest difference seems to be that the KL25Z board (with MBED firmware) offers one more interface—the mass-storage interface that is used for downloading programs to the board. I don’t understand a lot of what I can see poking around with USB Prober, but I’m not seeing anything else that looks like a significant difference between the Leonardo board and the KL25Z (with MBED firmware). Certainly the serial interface specs look identical, so far as I can tell.

Breaking every application on Mac OS X

John Burk, in his blog post Designing a new type of tech PD-feedback wanted, said

“There is almost nothing you can do from the keyboard to harm or permanently damage your computer.”

Ah, if only that were true. It took several days of debugging to get my son’s account working on our new Mac again, after he did something seems innocuous: changed “Sharing and Permissions” on his home directory and did “apply to enclosed items”. He had done it because he’d created a new crippled account for himself with parental controls that prohibited his favorite time wasters, but he wanted that account to have access to all his files on his regular account.  He gave the crippled account read and write access to his home directory and clicked the button to propagate that recursively.

We spent days trying to figure out why almost every application program (TextEdit, Pages, Photoshop Elements) was broken and how to fix it. The programs all worked on the other accounts, so it was clear that the problem was specific to his account. Just figuring out that it was changing the permissions on the home directory that broke everything took a while, though we pretty quickly saw that the permissions were messed up on his files (with “everyone” appearing twice and the owner not having access, for example).

It turns out that Apple did one of the worst possible implementations of Access Control Lists (ACLs), so that changing the permissions on your home directory messes everything up. The secret fix is to remove all the ACLs with

chmod -R -N ~

You won’t find that advice from Apple, but from disgruntled users trying to figure out why their user-friendly Mac no longer does anything right.

This is not a new Lion bug, but one that has been around for a while. I’m amazed that they have left something so damaging in the system for so long.