Serial-only USB boot loader

jcw
Sun Nov 01, 2015 3:51 pm
Has anyone explored this serial-only USB boot loader, as alternative for the DFU approach?

original repository: https://github.com/leaflabs/maple-bootloader
further developments, but still relatively old: https://github.com/j1rie/maple-bootloader

The interesting bit is that this might avoid the DFU vs VCP (serial) complexity. In particular, it looks like this approach avoids the need for any driver setup and config files on Windows, Mac OSX, and Linux, and that even pulsing D+ for a reattach may no longer be necessary, since the USB connection remains of the same type. Lastly, the use of the Atmel STK protocol in this driver means that “avrdude” might be all that’s needed to upload firware.

If all of this is indeed the case, it could be a very interesting avenue to explore IMO, but for some reason there seems to be no recent development going on in this area, at least not any I could find.

-jcw

Uodate – just found this thread: http://www.stm32duino.com/viewtopic.php?f=32&t=31
Seems related. Reading now. Apologies for the noise. Film at eleven.


jcw
Tue Nov 03, 2015 11:00 am
The exploration continues. I’d like to try extending or implementing a CDC ACM based serial USB boot loader.
This would jump to the loaded sketch, which continues to use the same driver – avoiding reset / handover issues.

Have been looking at libopencm3 as starting point, but as far as I can tell it only does USB in polling mode?
That would rule it out for sketches in the IDE. The demos from libopencm3-examples do work out of the box.

There’s the current libmaple implementation, but I’m not sure what its status is.
Working, clearly, but not using recent STM headers and conventions, it seems?
For future consolidation, using the same base code for all of the IDE runtime code would be nice.

There are a lot of projects for STM32 and USB out there 🙂 – if anyone has a suggestion for a good starting point for a small solid interrupt-driven USB serial implementation for STM32’s (F1 for now), please tell…

-jcw


mrburnette
Tue Nov 03, 2015 1:23 pm
jcw wrote:The exploration continues. I‘d like to try extending or implementing a CDC ACM based serial USB boot loader.
This would jump to the loaded sketch, which continues to use the same driver – avoiding reset / handover issues.
<…>

jcw
Tue Nov 03, 2015 3:10 pm
Ray,

My thought was: low 8 KB flash and low 2 KB RAM are for the serial USB boot loader. Uploads write to the flash memory above that.

Sketch is compiled with flash offset 0x2000 and RAM offset 0x0800, it doesn’t contain a USB driver, and doesn’t touch the USB hardware. Instead, the boot loader has a vector table in a known place, which the sketch then uses to call into, for plain character-based serial I/O over USB.

With interrupts, the sketch vector entries for USB (3, as I understand) will need to point to the same boot code once re-vectored.

So as far as USB is concerned, there is no interruption of service between when the boot loader ends (with inited and working USB device) and the sketch starts – as there’s no µC or device reset in between. The USB driver in the boot loader can re-use the LeafLabs Maple trick for detecting a special “self-reset” request.

It’s mostly conjecture and fantasy so far. But it would be nice find out ASAP if such an approach is flawed in any way.

-jcw


mrburnette
Tue Nov 03, 2015 3:52 pm
jcw wrote:Ray,

My thought was: low 8 KB flash and low 2 KB RAM are for the serial USB boot loader. Uploads write to the flash memory above that.

Sketch is compiled with flash offset 0x2000 and RAM offset 0x0800, it doesn’t contain a USB driver, and doesn’t touch the USB hardware. Instead, the boot loader has a vector table in a known place, which the sketch then uses to call into, for plain character-based serial I/O over USB.
<…>


jcw
Tue Nov 03, 2015 4:05 pm
That’s what the vector table is for – pointers which allow indirect calls to the boot loader code.

Here’s an example from NXP for LPC µCs, used in one of my projects: https://github.com/jeelabs/embello/blob … #L156-L170 – the struct defines pointers to code in ROM, and that struct is located at a fixed known address.

To put it differently: the hookup is done at run time, not at compile/link time.


RogerClark
Tue Nov 03, 2015 8:04 pm
Jean-Claude

I must admit, I’m still not sure why you want a serial bootloader, or one that doesnt re enumerate the USB?

Is it for use with serial OTA updates.

Although the current bootloader is not perfect, most people seem to be able to use it.
(I know you have problems with it on OSX, but I know it works for many people on OSX, so I think you have a specific config problem with your Mac).

BTW. To use OTA
I think its better to use an external processor / radio module, e.g. ESP8266 or nRF51822, then the STM32 could be flashed using SWD by the co-processor, or even by using the internal serial protocol hardware bootloader.

But I know the range on BLE and Wifi is not as good as 433MHz.
However you could use the nRF9E5, as I think someone has this working with an open source compiler. (I know the 9E5 is an obscure device and not used much, but perhaps someone else makes a 433MHz device with MCU)


jcw
Tue Nov 03, 2015 11:14 pm
It’ll all become clearer I hope once the code exists, and I can describe the workflow this leads to.

No, this is not for OTA uploads via wireless, which are indeed on my radar. That will also need a boot loader, but I’ll be using a completely different mechanism for that. That code exists, but still needs to be adapted to the STM32. For now, I first need to learn how boot loaders on STM32 work, and try to understand enough (but no more) of USB to get things working.

It looks like the libopencm3 USB driver (same as used in the BMP) can probably be used in interrupt mode after all, so I’m going to stick with that for now.


RogerClark
Wed Nov 04, 2015 12:34 am
It looks like the libopencm3 USB driver (same as used in the BMP) can probably be used in interrupt mode after all, so I’m going to stick with that for now.

Libopencm3 is much more up to date than the code the the current bootloader uses, so I think its a good choice


jcw
Thu Nov 05, 2015 12:01 am
I’m reporting this here, though it touches on another thread, about having to call Serial.begin() in the sketch – viewtopic.php?f=3&t=703

Yes, serial USB has to be running & attached for it to pick up a reset request (I’ll call it “The Maple Trick”). This may also imply that anyone using a different USB mode such as HID or MEM disk will run into this same issue.

This is one of the issues an all-serial-non-re-enumerating boot loader can solve, I hope – with the help of the IWDG watchdog:

* on reset, the boot loader sets up USB serial
* when the sketch runs, it continues to use the boot loader’s USB driver
* we set up a watchdog and trigger it main, just after each call to loop() for example
* a stuck sketch will thus reset, maybe even periodically
* each time, the boot loader will start and re-init USB serial, and wait for boot requests for some time
* the serial USB driver continues to implement The Maple Trick

What this would do, is to keep the board entering USB serial mode every once in a while, even if the sketch crashes. With a watchdog timeout of 5 seconds, and with the boot loader waiting at least one second for incoming upload-/reset-req’s, that means the board should be able to replace a faulty sketch at least 20% of the time. Finally, if the uploader on the PC side were to keep trying for at least 6 second, it would be guaranteed to succeed.

There are still failure modes where this may be insufficient (keeping the watchdog triggered while the USB driver isn’t functioning properly), but hopefully this would be considerably less likely. In that ultimate case, the perpetual boot mode trick would still be needed, but it could be a manual jumper-wire action, as last resort.

The two key points which will make this work, are: 1) that the boot loader gets control first and cannot easily be damaged (as it’s not part of uploaded code), and 2) that the watchdog reset kicks in for sketches which are in serious trouble (e.g. crashed, interrupts disabled, infinite loop).

All of this is theory for now. First, there has to be a working serial boot process. I’m still many steps away from that.


RogerClark
Thu Nov 05, 2015 1:10 am
@jcw

I think adding a WDT is going to cause all sorts of problems, unless people know its running and you keep needing to feed it.

i.e you’d need to feed the WDT in the delay() function (I had to do this in some other firmware, as I realised that the original author, had not taken into consideration that someone would call their delay function with values as large as 1 second). I know this is easy to add.

Also, you are relying on people using setup() and loop() in the prescribed manner. I personally don’t always bother to use loop, I sometimes just put a

while(1)
{
// mycode goes here
}


mrburnette
Thu Nov 05, 2015 1:34 am
jcw wrote:
<…>

* on reset, the boot loader sets up USB serial
* when the sketch runs, it continues to use the boot loader’s USB driver
* we set up a watchdog and trigger it main, just after each call to loop() for example
* a stuck sketch will thus reset, maybe even periodically
* each time, the boot loader will start and re-init USB serial, and wait for boot requests for some time

jcw
Thu Nov 05, 2015 1:38 am
I’m just trying to come up with a solution, so that a runaway sketch can be replaced by a new one when USB isn’t working. This seems like one way to accomplish that.

The idea is not to feed the IWDG in user code (better not actually, to avoid “false” safety), but in the background. The Arduino loop() would work, even if you keep it empty, especially in combination with delay() in the rest of the code. Other useful places could be Serial’s available + read + write. All of these require no change in user sketches.

There are several ways to keep the IWDG at bay, i.e. we could feed it from the USB interrupt, or from the SysTick interrupt (after checking as much as it can that USB is operating properly).

I don’t think it’s hard to explain to people that they need to periodically call either “delay()” or “yield()” in any code which can take more than 2..3 seconds. Most people will be using delay() all over the place anyway. And having a watchdog running is a good practice, worth exposing, IMO.


mrburnette
Thu Nov 05, 2015 1:46 am
jcw wrote:
<…>
I don’t think it’s hard to explain to people that they need to periodically call either “delay()” or “yield()” in any code which can take more than 2..3 seconds. Most people will be using delay() all over the place anyway. And having a watchdog running is a good practice, worth exposing, IMO.

jcw
Thu Nov 05, 2015 1:55 am
Yes, I understand the risk. But all code does something, and naive code will call library stuff all the time (even just digitalWrite) – maybe we can identify a handful of places where we insert the yield() in the runtime ourselves?

If all else fails, this warning could perhaps be added to the troubleshooting section:
“If your code keeps starting over for no good reason, insert a yield() inside the main loops in your code”

I’m not saying this is trivial. But I don’t see why it would be impossible to resolve under the hood.


jcw
Thu Nov 05, 2015 1:58 am
mrburnette wrote:When contemplating changes to functionality, I would strongly suggest not taking the Arduino paradigm too far away from the mother ship.

jcw
Thu Nov 05, 2015 2:07 am
Ok, here’s a bare bones variant of my original notes: the SysTick handler keeps the IWDG at bay.

That in itself ought to handle the most serious crashes and things like disabling interrupts far too long. Then, as we gain more insight, we can add more smarts to SysTick, so that say once a second, it checks a bit more whether the USB driver is really operating as intended.

The SysTick mechanism is part of current Arduino folklore (well, interrupt-driven elapsed time is), and the watchdog adds no perceptible runtime effects as long as it doesn’t fire.


mrburnette
Thu Nov 05, 2015 2:20 am
jcw wrote:Ok, here’s a bare bones variant of my original notes: the SysTick handler keeps the IWDG at bay.
<…>
The SysTick mechanism is part of current Arduino folklore (well, interrupt-driven elapsed time is), and the watchdog adds no perceptible runtime effects as long as it doesn’t fire.

stevech
Thu Nov 05, 2015 6:45 am
Can bootloader pain be avoided by using an STM32Fxxx that has a bootloader in ROM, e.g., the F3, F4 can boot from serial, USB and in some cases, CAN and SPI.

RogerClark
Thu Nov 05, 2015 8:41 am
F1 can upload via serial bootloader (hardware bootloader), but this requires external hardware.

F3 and F4 have USB DFU bootloaders in their hardware, but as the majority of low cost / (cost effective) boards are based on the F103 this generally rules out using the hardware bootloader


jcw
Thu Nov 05, 2015 4:45 pm
Ok, I’m not at all sure this is the best way to go, but here’s an example of an STMF103 accepting an Intel hex file sent to it via “avrdude” over USB serial, using the STK format:

$ avrdude -p m328p -c arduino -VeF -U test.hex -P /dev/cu.usbmodem2441

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x141516
avrdude: Expected signature for ATmega328P is 1E 95 0F
avrdude: erasing chip
avrdude: reading input file "test.hex"
avrdude: input file test.hex auto detected as Intel Hex
avrdude: writing flash (464 bytes):

Writing | ################################################## | 100% 0.61s

avrdude: 464 bytes of flash written

avrdude: safemode: Fuses OK (E:01, H:19, L:19)

avrdude done. Thank you.


Rick Kimball
Thu Nov 05, 2015 5:03 pm
: ) .. that is just so wrong …

jcw
Thu Nov 05, 2015 6:22 pm
Heh. Well, given that ROM-style serial uploads for STM32 are already supported in the IDE, it probably makes more sense to implement STM’s native protocol for USB as well. I just wanted to try this out…

mrburnette
Thu Nov 05, 2015 7:25 pm
I saw a hack the other day for another non-AVR uC, ESP8266 I think, but did not give it much thought at the time.

Ray


RogerClark
Thu Nov 05, 2015 8:52 pm
RedBearLab use the AVRDude technique, but they use an external (Freescale) processor do handle this.

I guess it saves them having to maintain upload s/w.

But as all of our upload s/w works OK, I personally prefer DFU, because AFIK this is a binary protocol


jcw
Thu Nov 05, 2015 9:21 pm
RogerClark wrote:… because AFIK this is a binary protocol

RogerClark
Thu Nov 05, 2015 9:26 pm
Anyway, I’ll probably switch to STM’s serial protocol.

There is code for that in the original Maple bootloader. Serial bootload branch – but you probably already know that


jcw
Fri Nov 06, 2015 12:09 am
Yes, thx. Ok, the STM protocol is actually a bit simpler. Looks like a basic test works (not actually saving to flash but to a RAM buffer for testing):

$ stm32loader.py -p /dev/tty.usbmodem1421 -ewvX test.bin
Reading data from test.bin
Bootloader version 0x20
Chip id 0x410, STM32F1, performance, medium-density
Writing 27 bytes to start address 0x8000000
Write 256 bytes at 0x8000000
Read 256 bytes at 0x8000000
Verification OK
Writing to Reset Register


Rick Kimball
Fri Nov 06, 2015 12:16 am
jcw wrote:Yes, thx. Ok, the STM protocol is actually a bit simpler. Looks like a basic test works (not actually saving to flash but to a RAM buffer for testing):

jcw
Sun Nov 08, 2015 12:36 am
Progress report. And let me reiterate that libopencm3 is well-designed, IMO – a solid foundation to build on.

I now have a first crude approximation of a boot loader which recognises the STM32 serial USART protocol, and handles erasing, flashing, and reading back. For now, It’s flashing to a fixed unused area, so this can’t actually run what it saved yet, but all the main pieces appear to be working: usb serial I/O in polling mode, proper STM protocol handling, and writing to flash memory. The code is only 6.5 KB so far, so I expect the final boot loader to still fit within 8 KB.

Haven’t published the source yet, for the silly reason that I can’t make up my mind on what to call it and where to put it. But I’m sure these hurdles can be overcome with some additional effort 🙂

With this boot leader, the same stm32upload.py script (or some equivalent) can in principle be used for “real” USART-based uploads and for this USB-based one, but there’s a little snag… if this boot loader sits in lower 8 KB flash, then sketches will need to be compiled with a base address above that, i.e. 0x8002000 – just as with the Maple boot loader. That would mean YET another upload tool variant to add to the IDE menus. Yuck.

It can be resolved with the boot loader at the TOP of flash memory, I think. But that’s something for later.


RogerClark
Sun Nov 08, 2015 1:01 am
@jcw

Sounds like you are making amazing progress.

I’m not sure if there is much we can do to overcome the bootloader menu’s problem, but perhaps some sort of Arduino Plugin could do it.

I noticed that the ESP8266 team (IGGR) seems to have developed a plugin to manage their uploads.


jcw
Sun Nov 08, 2015 1:04 pm
More progress: the boot loader now correctly saves and launches sketches. It’ll definitely fit in 8 KB.

I’m not able to upload from the IDE yet, though. For some reason I get this error:
stm32flash 0.4

Error probing interface "serial_posix"
http://stm32flash.googlecode.com/
Cannot handle device "/dev/cu.usbmodemUSU1"

Failed to open port: /dev/cu.usbmodemUSU1
Using Parser : Raw BINARY

the selected serial port
does not exist or your board is not connected


jcw
Sun Nov 08, 2015 1:32 pm
Here is how the upload cycle could work – maybe:

* host asserts DTR low
* target USB driver detects this and enters the boot loader
* disable all interrupts, stopping everything the sketch may have been doing
* (is there an easy way to reset all I/O + DMA peripherals to stop them from doing stuff?)
* do not re-init / re-enumerate the USB device from scratch
* host starts uploading using STM serial protocol
* target performs the upload as requested, protecting the boot loader against overwrites
* host ends with a reset/start request
* target does a self-reset (with some flag in RAM to detect this case?)
* target starts up with boot loader again
* target sets up USB device again (will probably have been re-enumerated)
* since this is a requested reset, boot loader then sets up and launches user sketch

There are some potential problems with this, but I’m trying to get around the fact that re-enumeration takes a little time. I’m not sure what a full reset does for the upload process – it would be simpler to just reset on DTR assert, and then re-enumerate, wait for the device to re-appear, upload, and launch all in one sequence. It’s how lpc21isp works for NXP’s chips (UART-only), and it lets one utility handle uploading as well as turn itself into a serial port terminal console.

The re-enumeration is always from serial to serial endpoint, since there is no DFU mode involved here at all, but that still means the current terminal session will be forcibly closed. Which is tedious, when you have a screen/putty session running during development, for example.

If there were a way to re-init all µC hardware except USB, life would be a lot simpler. Probably not possible?


mrburnette
Sun Nov 08, 2015 2:32 pm
jcw wrote:
<…>
An interesting property is that the boot loader does not use interrupts at all, it only needs to init and poll USB to get the job done. So in essence, the sketch starts as if coming out of reset, except that there’s an attached USB device which hopefully the sketch can hook into – maybe it can check for this, and conditionally re-use the open connection, or else just init USB as it has always done until now.
<…>

jcw
Sun Nov 08, 2015 6:58 pm
Thanks 🙂

RogerClark
Sun Nov 08, 2015 8:08 pm
JeanClaude

There was an issue with the stm32flash where it didnt like the names of some serial devices.

I thought Id done a fix for that. I forget where it is in the code, but It checks / validates the name of the port ( just the name)

You could try commenting that code out and rebuilding.

Edit.

Sorry. It looks like I already took out that check

https://github.com/rogerclarkmelbourne/ … 550e3afe01

You could just try rebuilding it.

(I assume you are using the binary in my repo and not some other one, as the original version had other issues as well)


RogerClark
Sun Nov 08, 2015 8:38 pm
See this thread

viewtopic.php?t=376


jcw
Sun Nov 08, 2015 9:02 pm
Aha, thanks – that’s interesting. I see something else which interferes with my assumptions: the IDE is calling …/serial_upload on Mac OSX, which is a shell script setting the baudrate to 230400 iso 115200, and the upload address to 0x08000000, whereas this new serial upload mechanism needs 0x08002000.

It’s not the whole issue, but definitely also something relevant for this boot loader.

There’s no guarantee that we have python available in the IDE, I assume.
Hmmm… maybe I’ll rewrite the uploader in Go 🙂


jcw
Sun Nov 08, 2015 9:05 pm
RogerClark wrote:(I assume you are using the binary in my repo and not some other one, as the original version had other issues as well)

RogerClark
Sun Nov 08, 2015 9:17 pm
You can pass in paramaters from boards.txt via platform.txt into the script.

(as you probably already know ;-)


jcw
Sun Nov 08, 2015 9:48 pm
Thx – still getting to grips with boards.txt & parameters.txt …
I can reproduce the failure from the cmd line now.

Well, it seems that – unlike stm32loader.py – stm32flash does not like a virtual USB port at all:

$ ../../../macosx/stm32flash/stm32flash /dev/cu.usbmodemD5D4C5E3
stm32flash 0.4

http://stm32flash.googlecode.com/

Error probing interface "serial_posix"
Cannot handle device "/dev/cu.usbmodemD5D4C5E3"
Failed to open port: /dev/cu.usbmodemD5D4C5E3


RogerClark
Sun Nov 08, 2015 10:13 pm
Last time I looked stm32flash was not being actively maintained, it languished on sourceforge for ages, without any updates, dispite people noting bugs.

I forget if I had to fix anything other than it only accepting ports named tty.


jcw
Sun Nov 08, 2015 10:30 pm
No worries. There’s are some tty control settings differences, when I ignore one of the checks, stm32flash proceeds.
It does make a few more requests to the boot loader than what it supports so far, so need to add a bit more functionality in.
Getting there…

RogerClark
Sun Nov 08, 2015 10:41 pm
OK.

Its always the same with S/W development… You can always end up with problems from the area you least expect.


jcw
Mon Nov 09, 2015 12:12 am
Ok, got a first test upload going through the IDE using a hacked stm32flash. Mostly by disabling a few checks and features.

The stm32flash utility is considerably more elaborate than we need it to be here (and not just for this USB serial boot loader, I believe). As far as I can tell, all we need is an erase + write into a contiguous area of flash memory, optional read-back for verification, and then a jump to the base address. And this on Windows and POSIX systems, for at least real and virtual serial ports.

In summary: apart from having to press the reset button, the basics of uploading and running a sketch through this new boot loader seems to work. At least as proof of concept. No handover of serial USB from the boot loader to the sketch yet, that’s still a big TODO.

I’ll ponder a bit on a way forward. A small portable scripting environment which does serial I/O would be enough for this simple task (e.g. Lua, JimTcl). Basic text manipulation and launch could also help simplify the boards/platform.txt situation, though it’s a bit of a stretch to start changing all that. I’ll have a look at the ESP-Arduino setup. Maybe we can share some approaches with them. Too bad this whole Arduino IDE thing isn’t a bit more modular.

Thanks Roger, for your help, tips, pointers, and background info – it was just the right mix to keep me from getting stuck 🙂


RogerClark
Mon Nov 09, 2015 1:03 am
Hi Jean-Claude

Sounds like you have made amazing progress.

Looking at what the ESP guys are doing is a good idea. They seem to have an upload plugin for the Arduino IDE.


jcw
Mon Nov 09, 2015 12:27 pm
Let’s talk about the boot protocol for a moment. I have the STM32 serial protocol working for uploads, but this depends on either stm32loader.py (which requires Python to be installed) or stm32flash (which requires some changes, and will need to be re-compiled for Win/Mac/Lin32/Lin64).

An alternative is to have the boot loader act as STK500-compatible boot loader, as used on ATmega chips. This would mean that “avrdude” can be used for uploads, which is well-supported and maintained, and part of the standard Arduino IDE install, hence effortless for us. The emulation is slightly tricky, as the boot loader has to ignore things like “fuses”, and report itself as some sort of fake AVR µC to be accepted by avrdude.

Tools like avrdude, stm32flash, etc solve the important problem of talking to serial devices in raw mode, on Windows, Mac, and Linux.

Sooo… what if the boot loader did indeed adopt the STK500 protocol for uploads? Seriously. What would be the drawbacks?

And while I’m at it: why can’t we avoid the whole issue of selecting the right chip type, memory size, board model, manually? Why can’t we just select a serial interface, and ASK the attached board what µC type it is and what board type even? It would be trivial for a boot loader to report this. Shouldn’t we be stepping out of the stone age by now? 🙂


mrburnette
Mon Nov 09, 2015 1:37 pm
jcw wrote:
<…>
And while I’m at it: why can’t we avoid the whole issue of selecting the right chip type, memory size, board model, manually? Why can’t we just select a serial interface, and ASK the attached board what µC type it is and what board type even? It would be trivial for a boot loader to report this. Shouldn’t we be stepping out of the stone age by now? :)

jcw
Mon Nov 09, 2015 2:05 pm
Thanks, Ray – lots of encouragement in your comments, from where I stand…

I’ve been battling boot load chickens-and-eggs for years now. First on ATmega chips, and now the whole circus again on STM32.

If we could settle on say up to 3 boards for people to use as “boot loader installers”, then a page on the web with exact info on how to wire things up to install (over ROM-based serial) a specific boot loader on a second board would be feasible. It gets confusing, but say you take a standard Arduino Uno, connect it with a few wires to your own board (STM32, LPC, whatever), put your own board in serial-upload mode, and upload a special installer sketch via the IDE to the Uno. After that, your own board ends up with the proper boot loader (which could be any of numerous types, Maple DFU, USB serial, wireless, etc).

I’ve been through this mess several times, the important challenge IMO is to end up with brief and clear instructions.

The only thing we need is an easy starting point, i.e. a board supported by the IDE, with a few spare I/O pins.
This is the unavoidable part: you need ONE board to install a boot loader on a SECOND one.
But I’d like to stress that they don”t need to be the same. A Uno could program a Maple Mini just fine.

To rephrase: what cheap boards are out there which work in the IDE out of the box? Which ones are the really common ones?

(this same approach can also be used to install Rick’s BMP build on most STM32F103C8’s, BTW)


Rick Kimball
Mon Nov 09, 2015 4:52 pm
If the bootloader is small enough, maybe you could make an arduino sketch that contains the bootloader binary as an array of bytes.

The sketch would take those binary bytes and upload them via the ROM based stm32 factory serial bootloader. As it is a sketch that is only going to use serial communication and gpio pins ( to toggle the the chip into bootload mode ) it would be easy to port to any board running an arduino port. It could be an arduino board, teensy, stm32, texas instruments .. etc

-rick


jcw
Mon Nov 09, 2015 5:23 pm
Yep, exactly. Could do this with a $2.30 Arduino Nano clone off eBay, for example.
And optionally modify the code a bit, say to add a board-type ID before flashing it.
There’s also a unique H/W ID in each µC, it can be used to distinguish multiple units.

That’s 2 bytes µC type (0x410 etc), 2 bytes board ID, 12 bytes h/w ID = 16 byte “tag”.
The boot loader could report these for a read from some “impossible” address, perhaps.

-jcw


mrburnette
Mon Nov 09, 2015 8:10 pm
jcw wrote:Yep, exactly. Could do this with a $2.30 Arduino Nano clone off eBay, for example.
<…>

RogerClark
Mon Nov 09, 2015 9:12 pm
I have managed to flash the bootloader using a Uno, but it was more by fluke than design.

The issue is that I think the internal bootloader uses Even Parity by default, and Software serial does not support this ( as far as I can tell) on AVR.

I found if I set the baud rate to 56k, and used the default settings, in software serial, I could use the Uno as a serial bridge.
But slower or faster speeds would not work.

I had considered if there was a way to get a Uno to act like a Black Magic Probe, and connect via swd, but I didnt have time to continue looking into this.

There is also the option to update the bootloader using a sketch that contains the binary of the new bootloader ( as an array).

We already have this, but it only works ( at the moment) if the new bootloader is not bigger than the old one. Otherwise you would end up overwriting the sketch that is flashing the bootloader.

BTW. As an aside, RedBearLab use Avrdude as the upload program for their arduino compatible stuff, and implement the protocol in a MCU ( I cant remember what they use but its not a stm32)

I dont have any problems with using Avrdude, in fact its a great idea, but getting people to change the pre installed bootloader in the maple mini, may be an uphill struggle.


jcw
Tue Nov 10, 2015 10:20 am
To continue on my own post:
what cheap boards are out there which work in the IDE out of the box? Which ones are the really common ones?
The boards I see as most suitable for turning into a “boot loader bootstrap” (bolobo?) are:

* Arduino Uno
* Nucleo F103RB
* Maple Mini

These should all work with the Arduino IDE out of the box, so if you happen to have one of these and upload a yet-to-be-written sketch to them, you will then be able to load a boot loader into your own target board of choice (at least for STM32F10x µCs). This will use the STM32’s ROM-based serial boot loader.

It needs 3 wires to the target board (RX/TX/GND) and a description how to manually put the target board into the proper boot mode.

It would be nice to have a few versions of this sketch, e.g. one with the current Maple v2 boot loader, and one with the new USB Serial Loader I’m working on (both are under 8 KB). On boards with at least 128 KB flash, a third useful option IMO, would be a version which flashes Rick’s BMP build.

The adventure continues…


RogerClark
Tue Nov 10, 2015 10:31 am
The Uno or its predecessor are probably the most widely used of the Arduino boards

I’d recommend that you focus on getting a reliable way to flash the bootloader using that board.

However, I’m not sure its going to be that easy, because of the lack of parity control with software serial.

You can of course use the hardware serial on the Uno, but you loose debug to the PC if you have to use that.

People could of course buy a $3 USB to Serial, but I agree that if we could reflash using hardware people already have, it would be easier.

BTW. If you are considering using a Maple mini to flash another board, I think using the BMP may be one option.

I”m looking at building a verison of the BMP that is loaded by the existing bootloader in the Maple mini into address 0x8005000 (just in case they have the old bootloader in the Maple mini)

I’m in the process of forking the BMP and adding the GD32, but doing a Maple mini version is next on my list, but the vector table and linker script would need to be changed, and its 3rd on my list of priorities for the BMP.

1) GD32 in my repo
2) binaries for GD32 in my repo
3) Generic STM32 in my repo (based on Rick’s j66_coreboard)
4) binaries for j66 in my repo
5) Maple mini without bootloader
6) Maple mini with existing bootloader

However I have not had much time to work on this today, as I helped Rick out with his nRF51822 stuff for a while today, and its been a busy day with work, and I’m not a “night owl” ;-)

So it will need to wait until tomorrow

But if you do a Maple mini version of the BMP, especially one that runs with the existing bootloader, in the next 10 hours, let me know, as it will save me having to do it ;-)


jcw
Tue Nov 10, 2015 12:40 pm
RogerClark wrote:However, I’m not sure its going to be that easy, because of the lack of parity control with software serial.

zmemw16
Tue Nov 10, 2015 6:27 pm
why does it have to be fast for a potentially once only happening?
after that you have the egg.

so maybe it could be bit banged?

stephen ducks down behind big wall. :D


jcw
Tue Nov 10, 2015 7:19 pm
Bit-banged sending is easy, it’s the reception that is more tricky. And that’s exactly what SoftwareSerial does.

I’ve hacked up a modified version of SoftwareSerial which sends out even parity, and ignores the incoming extra parity bit.
Getting this going on an ATmega is actually the best option for me after all. Here’s output from an ATmega328 sketch:

[stm32f1init] boot-usbSerial-v01.h

(When you see a question mark, RESET your TARGET board!)

Connecting? .. OK
Boot version: 0x22
Chip type: 0x410
Unprotecting: OK
Resuming? ...... OK
Erasing: OK
Writing: ........................... OK
Done: 6661 bytes uploaded.


jcw
Tue Nov 10, 2015 7:40 pm
mrburnette wrote:Just saved you $0.40: http://www.aliexpress.com/wholesale?pro … 2341832857

RogerClark
Tue Nov 10, 2015 7:53 pm
umm

That link just takes me to a search on Aliexpress.

The product still active?

@jcw

Well done for hacking sw serial to do even parity.

I dont think there is any harm to unprotect, even every time.
The only board that I have that arrived protected was a STLInk dongle, which I then reflashed with the bootloader, and it worked fine even though it was a STM32L03 I think !!

But I only needed to do it once.

I’m not sure its such a big deal for someone to need a usb to serial converter or even a STLink.
The STM32 board will be a new board for them, and if they have been doing any research they will know that unless its a Maple mini, they need to flash the firmware.

Well, no, I suppose we have people on this forum who have bought a Blue Pill etc as the vendor claimed Arduino compatibility, but with no bootloader.

But I dont think its a lot to ask to buy usb to serial.

One thing that you may want to consider is updating the bootloader on the Maple mini via a sketch.
We already have this, to update from the old maple bootloader.

But this would only work if you new bootloader was not larger than the existing one


jcw
Tue Nov 10, 2015 8:14 pm
Code is here – https://github.com/jeelabs/embello/tree … tm32f1init – docs to follow later this week.

One thing that you may want to consider is updating the bootloader on the Maple mini via a sketch.
We already have this, to update from the old maple bootloader.

There will be many different ways to upgrade & replace boot loaders, I expect. And more to come, no doubt.
But we have to be careful, IMO – this is the sort of “capability” which can massively confuse newcomers.
This particular one (now called “stm32f1init”) is really only meant as last resort, or when you start with next to nothing.

Onwards, to the next level!


jcw
Tue Nov 10, 2015 8:21 pm
RogerClark wrote:But I dont think its a lot to ask to buy usb to serial.

Rick Kimball
Tue Nov 10, 2015 8:33 pm
jcw wrote:Code is here – https://github.com/jeelabs/embello/tree … tm32f1init – docs to follow later this week.

jcw
Tue Nov 10, 2015 8:47 pm
Yes, I’ll definitely post it – but the new USB serial boot loader is not a sketch, it’s a libopencm3 project with a makefile.
And it’s only marginally usable so far (it still requires manual reset to pick up a new upload).

I do have a little “bin2h” tool to turn any binary file into a .h file suitable for this stm32f1init utility.
The idea being to collect header files and insert the proper one in the sketch.
I’ll add the Maple v3 boot loader header.

Here’s the “bin2h.c” code I’m using on OSX (should also work on Linux). Inlined below for now:

// Convert a binary file to a format usable as data inside C source code.
// -jcw, 2015-11-09

#include <stdio.h>

int main (int argc, const char** argv) {
if (argc != 2) {
fprintf(stderr, "usage: bin2h infile >outfile\n");
return 1;
}

FILE* inf = fopen(argv[1], "rb");
if (inf == 0) {
perror(argv[1]);
return 2;
}

int width = 0;
while (!feof(inf)) {
width += printf("%d,", fgetc(inf));
if (width > 75 || feof(inf)) {
putchar('\n');
width = 0;
}
}

return 0;
}


jcw
Tue Nov 10, 2015 8:52 pm
Ooh, msp430 Launchpad would be nice, tons of those in people’s drawers too, I suppose.
How much free flash does it have? If there’s enough room, you could stash a BMP image in there.

Rick Kimball
Tue Nov 10, 2015 9:01 pm
jcw wrote:Ooh, msp430 Launchpad would be nice, tons of those in people’s drawers too, I suppose.

RogerClark
Tue Nov 10, 2015 9:24 pm
JeanClaude

There are definitely some people on the forum who are capable to build using via Makefile etc

I can think if at least 3 or 4 people straight away.

I can test compile on Windows 7 if you want, as its my main OS, (though I do run OSX and Linux)


jcw
Tue Nov 10, 2015 9:36 pm
Oops – I wasn’t doubting that for one second! Sorry, I’ve been in the “for noobs” mindset too long.

Ok, I’ll think of a name and a spot to put it. Somehow, “Usb Serial Bootloader” doesn’t cut it, I think 🙂
I wasn’t trying to keep this to myself. Will report here as soon as it’s online.

Meanwhile, I’ve added the binary data for Maple boot 2.0 to stm32f1init. Untested.

And yeay, I’ve finally got a board which Maple-boots as it should here: std “Baite” Maple Mini clone, as pre-loaded.
The only quirk is that I seem to need to press the reset button after an upload. Otherwise fine, also resets/re-flashes.

Update – there’s more to it than the USB serial boot, for getting things to actually work: it also needs a new board entry to compile sketches at 8KB offset (like Maple, at least for now). And stm32flash has to be tweaked to talk to the CDC-ACM device. Hmm… the backlog is piling up 🙁


RogerClark
Tue Nov 10, 2015 10:13 pm
Hmm… the backlog is piling up :(

I know the feeling ;-)


jcw
Wed Nov 11, 2015 1:07 am
The stm32f1init sketch now also compiles ON an STM32F103 µC, using h/w serial with h/w even parity.
Well… at least it launches on Maple Mini and Nucleo F103, which means those could also be used as boot-the-bootloader setups.
Haven’t tried hooking either of these boards up to a target yet, though.

Lastly, I’ve converted the BMP-jc66 binary, so with a bit of luck this can now also be flashed – the sketch should fit on F103xB’s.
Again, untested, but it’s there for the brave/early/adventurous soul/bird/adopter who wants to give this a try.
The only missing link is to figure out the wiring connections between the stm32f1init and target boards, I think.

If you try this out, please let me know how it goes. Would like to document a couple of these scenarios in the next few days.


jcw
Wed Nov 11, 2015 2:12 pm
Looks like ARM builds aren’t quite working yet. I’m trying this on a Nucleo F103:

[stm32f1init] boot-usbSerial-v01.h

(When you see a question mark: RESET your TARGET board!)

Connecting? . OK
FAILED - got 0x1F


Rick Kimball
Wed Nov 11, 2015 4:42 pm
Instead of having the user press buttons, could the sketch toggle the BOOT0 and the Reset pins itself?

jcw
Wed Nov 11, 2015 4:59 pm
I’d expect stm32f1init to be for mostly-one-time use, but sure.

Just to confirm: the sketch pulls B0 low, toggles reset down briefly, uploads, pull B0 high, and toggles reset again to start the target?
I can probably manage that 🙂 – it needs two more wires and matching documentation, but their use can remain optional.


Rick Kimball
Wed Nov 11, 2015 5:05 pm
jcw wrote:I’d expect stm32f1init to be for mostly-one-time use, but sure.

Just to confirm: the sketch pulls B0 low, toggles reset down briefly, uploads, pull B0 high, and toggles reset again to start the target?
I can probably manage that :) – it needs two more wires and matching documentation, but their use can remain optional.


jcw
Wed Nov 11, 2015 5:27 pm
MSP430G2553, I presume? That’s not much RAM. Would be neat though. Yes, LEDs would be useful too.

Rick Kimball
Wed Nov 11, 2015 5:49 pm
jcw wrote:MSP430G2553, I presume? That’s not much RAM. Would be neat though. Yes, LEDs would be useful too.

jcw
Wed Nov 11, 2015 6:09 pm
Great, I’ve added a LED option and the ISP + RESET control lines. Latest in GitHub.
Will be happy to add support for MSP to this, just submit a pull request.

Edit – LED blink rate, on ATmega: 0.25 Hz = trying to connect, 1 Hz = programming, 5 Hz = error, steady = done.


Rick Kimball
Wed Nov 11, 2015 6:25 pm
How about abstracting out the Log code…
#ifdef __MSP430__
#define Target Serial
#define LED_PIN P1_6
#define RESET_PIN P1_5
#define ISP_PIN P1_4
#define Log_begin(s)
#define Log_print(s)
#define Log_println(arg1, …)
#define Log_flush()
#elif ARDUINO_STM_NUCLEO_F103RB
#define Log Serial1
#define Target Serial
… implement real logging to second device
#else
#define Log Serial
#define Target Serial1
… implement real logging to second device
#endif

jcw
Thu Nov 12, 2015 1:14 am
First commit of the USB serial uploader is now here: https://github.com/jeelabs/embello/tree … s/usbserup
Needs a small change to stm32flash, to be submitted as PR on GitHub later.

This needs an extra entry in boards.txt (can be avoided by moving the boot loader to high flash, but that’s a bit more work).
The current code emulates a minimal subset of STM’s USART upload protocol, i.e. compatible with F1’s ROM boot loader.
If this gets replaced with an avrdude-STK500 mode, then stm32flash & patch would not be needed anymore.


jcw
Sat Nov 14, 2015 9:30 am
The build instructions for usbserup have changed. You no longer need an old copy of libopencm3, see README.

Reflecting a bit on all the little details to make this new boot loader practical, here are the key TODO’s, I think:

* switch to the STK500 protocol, so uploads can be done with avrdude
* put the boot code in high-flash, so sketches can be loaded at 0x08000000, as usual
* listen to DTR and RTS low, to act as self-reset request (if possible, need to investigate)
* find a way for the loaded sketch to re-use the open serial connection

That last one is actually optional. With a sketch which simply re-inits the USB device, things already work. And if that sketch also includes the logic to listen to DTR+RTS, then it would be able to self-reset just like the Maple boot loader does. No more button pressing, which was one of the goals for all this.

With the above changes, a new upload method needs to be added, called “Serial (avrdude)”. Since the load offset will be gone, the same code can then also be uploaded using Serial, ST-Link, or BMP. The only difference in the IDE is which TRANSPORT is used to get the code into the µC. This will make build settings and upload settings independent (orthogonal).

There’s an interesting feature in avrdude to connect to a remote network port and upload the sketch there. Maybe one day we can get that going as well. I’ve been using socket-based uploads in other contexts, and in fact the “esp-link” (https://github.com/jeelabs/esp-link) project also supports it.


jcw
Fri Nov 27, 2015 2:46 pm
Here’s an update from the USB serial loader front.

The “usbserup” loader now resides at the end of flash memory. This means that the same sketches should run regardless of how they are uploaded (serial, usb serial, or BMP), because they can all be compiled the same way: loaded at the normal 0x08000000 flash base address.

There’s some trickery involved to make this work, see https://github.com/jeelabs/embello/tree … s/usbserup

The other piece of the puzzle is that you need a way to get usbserup installed properly. I’ve written a small sketch which does this, see https://github.com/jeelabs/embello/tree … nstallboot – you can use whatever upload method you currently have at your disposal.

In a nutshell: with this version of usbserup, you can press RESET and upload your sketch over USB using the “serial” method.

This is far from done. Heck, it’s almost untested, even. Having to press RESET is a step back w.r.t. the Maple loader, so there is no point using this boot loader if that works for you. Next on the list, is to allow triggering a reset via USB (I’m aiming for detection of the DTR-low edge, just as with AVR type Arduino boards).

Would love to hear from someone trying this out (please do keep in mind that usbserup will replace whatever boot loader you had!).


zmemw16
Fri Nov 27, 2015 5:52 pm
i can find one each of baite mini, red pill and blue pill, no pins fitted ie not soldered as yet.
also the same again, but soldered, contents unknown.
i also may have a couple of rbt/rct, vet and zet boards.

i normally use sw-link for programming

is blinky enough of a sketch or is there anything else needed as well?

what would be your expectations as to results?

stephen


jcw
Fri Nov 27, 2015 6:17 pm
Yes, blinky should be fine as test.

Please hold off a bit though – it’s not quite working as I thought. Also, the current boot loader config is for a 128 KB flash, and since it lives “up there”, flash size does matter for the current code & setup. Hm, I suppose I could just hardwire for 64 KB instead, then it’d work on both.

Thanks for the offer – I’ll follow up as soon as things have a slightly better chance of actually working…

Result I’m hoping for: get through the install unharmed, press reset, upload from IDE over serial pointed at the USB port, blink!


jcw
Fri Nov 27, 2015 6:58 pm
Ok, the test should work the same for both 64K and 128K F103-MD’s now (latest commit).

Something still bad. The protocol worked in low-flash, but now seems to fail:
$ stm32loader.py -p /dev/cu.usbmodem421 -b 115200 -ew fader.bin
Reading data from fader.bin
Bootloader version 0x22
Chip id 0x410, STM32F1, performance, medium-density
Writing 6268 bytes to start address 0x8000000
Write 256 bytes at 0x8000000
WRITE FAIL - try and recover
WRITE FAIL - wait
WRITE FAIL - retry
Traceback (most recent call last):


RogerClark
Fri Nov 27, 2015 7:54 pm
@jcw

We have found that all STM32F103C8 board have 128k flash memory even though they should only have 64k

So I think everyone should be able to test this.

However in the long term you would need to do a 64k version as well, in case STM changes their manufacturing processes and C8 devices become 64k

(BTW. GD32F103C8 is only 64k as it is a completely different device)


Leave a Reply

Your email address will not be published. Required fields are marked *