IMO: time to freeze the F103 core codebase

mrburnette
Thu Jun 30, 2016 12:37 pm
I had a hissy in another thread regarding major changes to the serial (non-usb) code. Particulars are not important.

What I think is important is that after 3 years, it is time to close the chapter on F1xx corecode enhancements. Much work was done early in “fixing” and migrating the Leaflabs’ code to Arduino 1.5x standards and that was most successful. We are not at 1.6.9 and STM32duino is working well. No doubt, as the ArduinoIDE advances, some changes may be required to the F1xx base, but IMO it is time cease thinking of enhancements at the core-level.

The above being said, enhancements that are of immense value (such as the DMA implementations) could be put up to a vote. Or better, I think these changes should go to another Github branch. Currently, we have “master” and “development” and I’m proposing that soon that master and development become the same and that a new branch called “breakfix” be created to only be used to “fix” things that are found to be a critical issue. The current development branch would then become whatever you guys want it to be … put in the new serial code, add parameters, etc. But this branch would never find its way into “master”.

My opinion is that we owe it to our constituent base to stabilize and maintain stability in “master”. Users need to know that code they write today and publish will run without issue moving forward.

As we are today, there is no formal promotion mechanism from development to master. A few members try it, it seems to work, and the code is promoted. If breakfix is implemented, I suggest a formal promotion scheme and perhaps adopt the ESP8266 paradigm… or another if one is known. In a corporate environment, there are departmental stockholders and business unit managers. When code issues are brought forward formally, they are investigated, a corrective code fix (sometimes procedural) is documented and all application areas are made aware of this documentation. In the meanwhile, the break-fix team is doing the coding and putting the fix up for testing. A regimen of tests using known-datasets is used to ensure that the proposed fix does not break anything else! After the break-fix team is satisfied themselves then they move the code to pre-production where the business unit application owner (may be an internal IT entity) certifies the fix as well as attests no other issues have occurred. Everyone is notified that the pre-production will be promoted to production at a specific time and date.

I am not proposing forming teams and identifying roles, but I am saying that some good process procedure can be gained by formalizing the fix-test-promote process. It is a Roger decision in my mind.

Ray


martinayotte
Thu Jun 30, 2016 12:46 pm
mrburnette wrote:But this branch would never find its way into “master”.

mrburnette
Thu Jun 30, 2016 1:00 pm
martinayotte wrote:mrburnette wrote:But this branch would never find its way into “master”.

Rick Kimball
Thu Jun 30, 2016 1:04 pm
I think this could be accomplished by just creating a release

https://help.github.com/articles/creating-releases/
https://github.com/rogerclarkmelbourne/ … 2/releases

No need to freeze it, just capture a moment in time and name it. The Arduino API/IDE marches forward with or without us. To say we are going to never change is ignoring the reality of the Arduino IDE.

So I say yes to creating a release!


mrburnette
Thu Jun 30, 2016 1:35 pm
Ok… a Release … ;)

Rick Kimball
Thu Jun 30, 2016 1:38 pm
Since you posted the topic of github and branches and all, I think we should move to an issue based branching scheme on github. It would go something like this:

o Some user has an interest in some feature or fix, they create a new issue. This gives other people a chance to comment or suggest a solution
o The original user forks the master repo and creates a new branch called issue_xxx. Where xxx is the issue #. They then go ahead and make the required changes and submit a pull request.
o If a consensus of people think it is a good thing, it can get incorporated into the master.

This approach gives people a chance to see what has been changed, suggest improvements or problems and in general they can try it out. This allows you to try out specific features by checking out that branch. The reason for the changes are attached to a specific issue and the documentation and comments are also available for perusal.

This has worked pretty well for the Energia project. For example, I wanted to add a parity feature to the Serial port. I created an issue and then a branch:

https://github.com/energia/Energia/issues/766
https://github.com/energia/Energia/commits/issue_766

Even though they haven’t incorporated my change, other people can find my specific changes to implement the feature.


martinayotte
Thu Jun 30, 2016 1:54 pm
Yes ! … and like most projects are doing, the github repo would simply tagged with this release number …
We can also do the same as kernel guys, with even/odd numbering for alternate of stable releases and development releases.

mrburnette
Thu Jun 30, 2016 3:46 pm
Honestly, Github is fine by me as the ESP8266 has used that approach well.

But, just like the Arduino IDE releases, STM32duino really needs a forward-only approach to release code. So, if I have a reader off somewhere that is trying to duplicate one of my many V-USB projects, I know that I must tell them to use Arduino IDE 1.0.5 … the way things are going now with STM32, I’m keeping ZIP copies by achieve date up on my Linux server. While I can go backward, the end-user is less likely to understand how to rollback github to a single point in time… although, I do remember one of us explaining in the forum… but, releases are a better approach.

Ray


martinayotte
Thu Jun 30, 2016 4:10 pm
I’m agree with you, Ray.
It all goes together, github tags, ZIP files, docs and even IDE associated with them.
Of course, it doesn’t mean to do that on daily basis, but only on official releases … ;)

RogerClark
Thu Jun 30, 2016 10:59 pm
I am fine with creating releases, but I suspect 99% of people will just grab whats in the master repo.

Re:issue related branches

Yes. I have used this system for commercial projects and it works well.

But, generally I find contributors do not fork the repo, and do a PR etc

Its far more likely that someone posts a code snippet or replacement files on the forum.

So I have to update the repo and test the change.

Hardly anyone seems to even look at the development branch and give me feedback.

I usually only get stuff tested when I merge the development branch back into master and a new user downloads it and finds a problem.

Perhaps posting to the announcements thread is not enough to let people know there are changes. But spamming everyone with a forum wide PM seems invasive as a know a lot of the forum members are dormant or have moved on to use other MCUs e.g. the ESP8266


mrburnette
Thu Jun 30, 2016 11:44 pm
<…>

  • Its far more likely that someone posts a code snippet or replacement files on the forum. So I have to update the repo and test the change.
  • Hardly anyone seems to even look at the development branch and give me feedback.
  • I usually only get stuff tested when I merge the development branch back into master and a new user downloads it and finds a problem.

That’s kind of my point. Unless something is broken, dev-branch is the next release. If something is broken; then you have the option to fix the master branch or fix the development branch & promote. Your decision would likely depend on your assessment of “how broke?”

Since everyone has access to development, if is the end-user’s decision on which to run. But when I download or tell someone to download master, I think it should be version controlled.

IMO, promotions from development to production is something like 2x per year.

While we all appreciate your testing prowess; testing in a perfect world would be hands-off, script driven. As you and I live in reality, I propose that the individual submitting an enhancement also submit a test plan & test sketch. 30 days before a planned release, a call for volunteers can relieve you of testing beyond what you wish to do. If the test plan identifies related areas of concern (USART, SPI, I2C, USB, etc.) then volunteers could use their own sketches for further testing.

I’m all for enhancements but with changes comes a responsibility to thoroughly soak and validate and exhaustively test. With so much effort being directed to F4, perhaps the introduction of new ideas there 1st and backport to F1 afterward… but that is just food for thought.

Ray


RogerClark
Fri Jul 01, 2016 12:05 am
Ray

Testing is always a problem, its time consuming and often requires a specific board or specific peripheral and often it involves a specific OS.
In reality there are just too many permutations for me to test some things at all. So I’m quite reliant on other people testing and letting me know whether things work or not.

Re: 2 year release cycle

ummm. 2 years in computing is a very long time. If I look at a project on github and its not done a new version in the last year, I generally avoid it, as I presume its not active enough.

Looking at what @IGGR does for the ESP8266, he seems to have 2 new releases per month (on average)
https://github.com/esp8266/Arduino/releases

Also, this master branch is updated very frequently https://github.com/esp8266/Arduino/commits/master
(Several times a week)

He also doesn’t seem to use branches. (Not that I agree with that method, I’m just noting what he does on a very successful project, and I really respect @iggr as he puts way more effort into ESP8266 than I do into STM32 and his programming skill level is far above mine)


martinayotte
Fri Jul 01, 2016 2:29 am
Hi Roger,

Ray mentioned “2x year release cycle” not “2 year release cycle”, but even 6 months is quite long, maybe “every quarter” would be nicer.

He also doesn’t seem to use branches

@igrr doesn’t create branches, but he use tags all the time, just to “take a snapshot/picture” at the release time.
Branches could then be created out of those tags if there a need to major bug fixes on this specific tags, otherwise branches will never been created.
At the GitHub Web site, tags appear like branches, people can checkout those specific tags for testing/debugging, but it doesn’t consume storage (anyway github space is unlimited).


RogerClark
Fri Jul 01, 2016 2:46 am
Martin

Thanks for the clarification. about x 2

I guess it depends on whether something is a bug fix or an enhancement. What some people think is an important fix, is viewed by others as an enhancement
e.g. Hardware Serial write, being much slower than AVR and means someones project doesn’t work like it did on AVR, in what case it can be viewed as a bug fix, as we don’t support what AVR already does, but equally it is an enhancement to libmaple.

Generally I try to make sure libmaple is as compatible with AVR code as possible, as I think this was one reason people didn’t take to the Maple Mini in large numbers (in addition to the high cost)

e.g. you had to change all references of Serial to SerialUSB, and had to set STM specific pin modes to do analogRead etc

But each case needs to be viewed individually and perhaps go to a vote in some circumstances to changes can benefit the majority.


martinayotte
Fri Jul 01, 2016 3:07 am
Hi Roger,

I’m agree with you ! (and I think we have common voice on that, and even @igrr ;) )

It is really rare that a branch is created out of the tag, even @iggr never done it yet.
In Linux Kernel, it occurs more frequently, mainly to satisfy old kernels still frequently in use by doing back-porting of some features done on Mainline.
(But we are far to have those scenarios where old versions of stm32duino is in production and absolutely need those back-ports)

In other words, you don’t have much more to do that you actually doing … :)

Simply add few minor tasks on your list :

– decide a Release schedule that meet you free time.
– be convince of the stabilty of the Master branch, with the help of the community.
– drop down the Big Damier Flag (and here, your are the Ultimate Judge !), and apply a github tag, and produce some JSON packages like ESP8266 using the build steps I think that @ddrown got working. (although this packages doesn’t to be done by yourself, since the tag will allow some other contributor to help here)
– of course there are maybe some other tasks that could be added, such as creating new Release.log, but this can also be done by someone else.

Having to small steps (except maybe the packaging), the whole process will bring stm32duino release process almost the same as esp8266 with almost not much effort from your side.

In other words, “tags” are the magic … :D


RogerClark
Fri Jul 01, 2016 3:51 am
Thanks Martin

I know @ddrown as some perl code that produces JSON packages, I will need to check if my cygwin runs them OK, as don’t want to have to switch to a linux box or a VM.
(Well, perhaps keeping a VM just for the JSON packages for releases would be a better option.)

I’m happy to make branches for bug fixes, though we will end up with lots of branches, which is perhaps why @iggr uses tags.

One big question however. What is the initial release number. I’m not sure it can be called 1.0, as I suspect libmaple doesn’t have all the latest Arduino API functions and macros etc etc

So it would have to be

0.0 < release_number < 1.0


martinayotte
Fri Jul 01, 2016 4:19 am

Well, perhaps keeping a VM just for the JSON packages for releases would be a better option.)

Of course that would help !
Even myself, I’m using Linux VMs under Linux, to avoid environment pollution, in my case running Xenial 16.04 under 14.04 to compile kernels for aarch64-linux-gnu- pineA64 or for several H3 OrangePi flavours kernels without polluting my main desktop. (I have even kept an older Ubuntu 12.04 in a VM because ffmeg was producing much smaller MPG from MythTV that the current releases)

Don’t worry about branches, as I said @iggr never done branches for bug fixes, those bug fixes always going into HEAD (unless major), @igrr never do any back-ports never for features or bugs, HEAD is the bleeding edge if you need/wish it.

For version numbering, my thought : like Ubuntu, quarterly based, 16.06 for tonight tag ! :D


RogerClark
Fri Jul 01, 2016 4:40 am
Re: Version numbering

Good idea.

Actually I think I dicussed something similar with @ddrown for the json packages and it is OK as long as there arent any leading zeros e.g. I don’t think the IDE like 2016.06.05 as a version number. I think it insists on 2016.6.5

Anyway. Good idea to use some sort of date base system


martinayotte
Fri Jul 01, 2016 4:56 am
… and again, we all don’t wish to this release management becomes a burden for you !

(I’m not yet retired as Ray ;-), but I’ve been part of SoftImage Microsoft for 4 years before they selled us in the 199x, I was in charge of putting in place and maintain all the Release process)


RogerClark
Fri Jul 01, 2016 5:08 am
Martin

I must admit, its nice that @ddrown is doing the json packages, as its one less thing for me to have to do ;-)


martinayotte
Fri Jul 01, 2016 5:18 am
If those packages scripts are fine, yes, simply build automation will do the job !

Your main job should not be coding related, it should only be to declare Damier Flag timestamp !
(of course, you can do some QA yourself, but that is up to you with your availabilities)

Lets go for a first try soon, 16.07 ! :P


mrburnette
Fri Jul 01, 2016 12:20 pm
martinayotte wrote:
<…>
Your main job should not be coding related, it should only be to declare Damier Flag timestamp !
(of course, you can do some QA yourself, but that is up to you with your availabilities)
<…>

RogerClark
Fri Jul 01, 2016 9:29 pm
mrburnette wrote:
….
At some point in the future, China may run out of their seemingly unlimited supply of STM32F103 devices and the $4 boards may become history.

Ray


ekawahyu
Mon Jul 04, 2016 12:53 am
martinayotte wrote:Lets go for a first try soon, 16.07 ! :P

RogerClark
Mon Jul 04, 2016 1:18 am
ekawahyu wrote:
Have you guys ever thought about having the official STM32Duino board of our own? Let’s call it as WhitePill for example? ;) Just a thought to spice things up as we are about to have 16.07 release soon.

I can design and test the board. We could also try to raise a campaign to get it mass produce here in the US for sub $5. All you guys need to do is to give me inputs and suggestions of the STM32 part number and package, size of the board, number of buttons, number of header pins, USB port, etc. I can definitely spare my time to catch up with the 16.07 release! What do you say?


martinayotte
Mon Jul 04, 2016 2:04 am
If there i a need to have Community board design, of course, as @roger said, it won’t based on F1xx since China has plenty of them and we can’t compete with their pricing.
But for newer F4x and F7x, that would be great to have some Community board, but it need to be stuffed with at least what can be be found on current boards such Arch_Max : SdCard holder and ETH, all for less than US$19.95

RogerClark
Mon Jul 04, 2016 2:13 am
And @grumpoldPizza has a small L4 board, which I think is a good niche, i.e like the STM32F4 stamp board

However, I suspect that the price of a F4 board would be in the $10 region, and most people would not need the speed etc, so would still just by the Blue Pill for $2

When you get to the $10 price point, you start competing with Rpi Zero and the Orange PI and C.H.I.P etc etc

So the market size is much smaller as you only get a fraction of the total market

IMHO there isnt much competition for a $2 Blue Pill. You can get a Arduino Pro mini for $1.25 including shipping, but value for money, the Blue Pill wins as it has USB and is also much faster and much more ram and flash etc etc.

The only reason I now occasionally use a Pro Mini is when I need some feature which we have not ported to the STM32, e.g. I2C slave.


ekawahyu
Mon Jul 04, 2016 3:04 am
Well, I just thought that having at least one official Open Hardware STM32Duino board would be cool. The ultimate goal is not the selling, but providing makers who love to build things from scratch with a reference design that works and fully supported by the core. I am sure some of people in the forum would try to build one themselves, if we can proof that the BOM would cost them less than $5.

@RogerClark: What about the use of STM32Duino infinity logo? Is it protected somewhere? Is @grumpoldPizza’s board an Open Hardware one? Maybe we can take his design with his permission and make it official? I think it would be great to have at least one official board fully supported by the community.

martinayotte wrote:But for newer F4x and F7x, that would be great to have some Community board, but it need to be stuffed with at least what can be be found on current boards such Arch_Max : SdCard holder and ETH, all for less than US$19.95


Rick Kimball
Mon Jul 04, 2016 3:08 am
I know the whole reason I was drawn to the stm32 stuff was because of the availability of cortex-m3 arm boards costing less than $5 and ready to go without me having to do anything other than order. If I was going to spend more than $5, I’d buy the STM Nucleo or Discovery boards and it would still cost less than anything I could build.

RogerClark
Mon Jul 04, 2016 3:59 am
ekawahyu wrote:
@RogerClark: What about the use of STM32Duino infinity logo? Is it protected somewhere?

Ollie
Mon Jul 04, 2016 5:49 am
@ekawahyu,

If you can have STM32F745VET6 and ESP32 on the same board with all pins – excluding osc and boot – exposed on 0.1″ grid on a board costing less than $15, I would be very interested.

For headers, there could be the following alternatives
– no headers, which would then imply a “mandatory” shield
– normal headers for SWD (or JTAG), one or two UARTS

Two user buttons and two LEDs. One or two USB. No Ethernet due to built-in wireless. In the mandatory shield, I would like to see large number of 3 pin clusters for direct wiring of sensors and servos.

For $10 board, I would be happy for L4 + ESP32.

For $5 board, I am happy with existing alternatives and the new ESP32 boards.

Cheers, Ollie


Leave a Reply

Your email address will not be published. Required fields are marked *