I’m with a problem to handle the 64 Kbytes of my STM32F103ZE with STM32Duino.
I did a simple code, and it just work until it uses 31kb, more then that the system halt before run. Could some one help me?
String ramtest[2160]; // <= WITH THIS THE SYSTEM WORK
String ramtest[2170]; // <= WITH THIS THE SYSTEM HALT
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
}
void loop() {
// put your main code here, to run repeatedly:
delay(3000);
Serial.println(SERIAL_TX_BUFFER_SIZE);
}
It’s hard to say why your sketch is crashing, but besides the memory you assign for variables, there is also the Stack and the Heap in RAM.
They grow towards each other, so you can run in a situation where the Stack overwrites variables in the Heap, or the Heap overwrites data in the stack. Either case causes a corruption of memory and likely a crash.
I can’t say how much memory you are using in either, but it you use a lot of “new” or “malloc” to alocate space for variables, that will come from the heap and can end up crashing with the stack.
On the other hand, if you have some code that nests functions calls a lot before unwinding it all, that will grow the stack a lot. Also if you declare objects that take a lot of ram, like those strings, as local, they take a lot of memory from the stack.
String ramtest;
String ramtest2;
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
}
void loop() {
// put your main code here, to run repeatedly:
Serial.print((int)&ramtest,HEX);
Serial.print(" ");
Serial.println((int)&ramtest2,HEX);
delay(1000);
}
I ran the same test on STM32GENERIC and I get
20000E0C 20000E24
So I think potentially there is an issue with the bss location on Libmaple
[Pito – Thu Aug 31, 2017 7:46 am] –
I saw the issue with F103ZE while running the simple USB TX test. With F103ZE I cannot allocate bigger buffer than 16kB (ie. 32kB).
I think someone who understands linker files needs to look at the libmaple linker files, as there seems to be a big chunk of RAM which can’t be used.
I guess this could be for the stack, but I thought thats normally put at the top of ram and moves downwards
For example: http://www.stm32duino.com/viewtopic.php … 787#p33787
You may combine Stack vs Heap Buffers allocation.
BSS are the variables initialized to 0. It goes in memory after the variables initialized with any value, that’s why they start higher than 0x2000000
The test Roger did is not the same the OP did though, since the OP used an array of strings, so Roger’s doesn’t take as much memory.
I just had a quick look at the linker script for the ZE and don’t see the problem, the start address looks right and is set to 64KB size, and the linker script, and the startup code that initializes BSS to 0 looks all correct, using 32bit pointers and no masks or anything like that could stop initialization beyond 32KB. Perhaps Rick can find something I missed.
[Pito – Thu Aug 31, 2017 7:46 am] –
I saw the issue with F103ZE while running the simple USB TX test. With F103ZE I cannot allocate bigger buffer than 16kB (ie. 32kB did not work).
Pito that issue may have been different though, depending how the buffers are managed there may be masks, pointers, etc, smaller than uin32, and so limiting the buffer size that can be managed.
A uint16 would allow 64KB, a int16 would allow 32KB, since negatice numbers probably don’t make much sense for a pointer depending how the code is written, so if the are conversions between int and uint, anything larger than 32KB could result in non-sense math, and if there is a division somewhere perhaps that’s turning 32KB in 16. Not saying that’s the case for sure, but can’t say that the issue is the same as the OP’s either.
I think much more context to his code is needed. Are his variables global or local? is he using malloc or new accross the sketch for something else? any use of malloc or new will not be accounted for as RAM used in during compilation, neither any local variable.
He may be copying those strings to local ones somewhere in his code, so for every KB in size he increases those variables, there may be several more using in Heap and Stack at runtime…
Just did a compilation of the sketch in the first post, and this is the relevant chunk in the map file.
.bss.ramtest 0x20000ddc 0x65b8 .\sloeber.ino.cpp.o
0x20000ddc ramtest
.bss.pbreak.4444
0x20007394 0x4 C:/Users/Victor/workspace/ZE_RAM_ALLOC/Release/arduino.ar(syscalls.c.o)
.bss.line_dtr_rts
0x20007398 0x1 C:/Users/Victor/workspace/ZE_RAM_ALLOC/Release/arduino.ar(usb_cdcacm.c.o)
.bss.vcomBufferTx
0x20007399 0x100 C:/Users/Victor/workspace/ZE_RAM_ALLOC/Release/arduino.ar(usb_cdcacm.c.o)
*fill* 0x20007499 0x3
.bss.tx_head 0x2000749c 0x4 C:/Users/Victor/workspace/ZE_RAM_ALLOC/Release/arduino.ar(usb_cdcacm.c.o)
.bss.rx_hook 0x200074a0 0x4 C:/Users/Victor/workspace/ZE_RAM_ALLOC/Release/arduino.ar(usb_cdcacm.c.o)
.bss.vcomBufferRx
0x200074a4 0x100 C:/Users/Victor/workspace/ZE_RAM_ALLOC/Release/arduino.ar(usb_cdcacm.c.o)
.bss.transmitting
200010B4 200010C0
[RogerClark – Thu Aug 31, 2017 7:41 am] –
I did a quick test on the Maple mini to determine where in RAM these global variables would get allocated.
String ramtest;
String ramtest2;void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
}void loop() {
// put your main code here, to run repeatedly:
Serial.print((int)&ramtest,HEX);
Serial.print(" ");
Serial.println((int)&ramtest2,HEX);
delay(1000);
}
My main code is crashing with using more than 20k. So I just did this simple test to check what whas happening with my RAM, without any Stack. My processor have 64k and I really need to use it for local variables and Serial Buffers. There is any way that can I use it all?
[victor_pv – Thu Aug 31, 2017 2:49 am] –
Not sure why you would need so many strings at the same time, but well that’s off topic.It’s hard to say why your sketch is crashing, but besides the memory you assign for variables, there is also the Stack and the Heap in RAM.
They grow towards each other, so you can run in a situation where the Stack overwrites variables in the Heap, or the Heap overwrites data in the stack. Either case causes a corruption of memory and likely a crash.
I can’t say how much memory you are using in either, but it you use a lot of “new” or “malloc” to alocate space for variables, that will come from the heap and can end up crashing with the stack.On the other hand, if you have some code that nests functions calls a lot before unwinding it all, that will grow the stack a lot. Also if you declare objects that take a lot of ram, like those strings, as local, they take a lot of memory from the stack.
I haven’t had time to run it yet. If that doesn’t crash, then there may be something else in your code.
In normal conditions you should be able to use all 64KB, but there could be a bug in the core, depending how exactly the memory is going to be used.
If you have a debugger, stlink or any other, it would be great if you can check where exactly is crashing. That’s when I plan to do when I have time to run a test.
The String class constructor allocates 16 bytes of memory per string in the heap, in adition to the other 12bytes it allocates in global space, in BSS.
When the string class constructor is called, with:
String ramtest[nnn]
What it does it copy a string object nnn times (it loops nnn times thru the constructor doing copies) and each of those times it allocates memory from the heap.
Eventually the heap reaches the stack and it crash the MCU.
The ramtest[] object only contains pointers, 12 bytes total, and one of the pointers is to the ram address in the heap where it allocates a buffer, I guess to write the actual string of text you are supposed to use this for, and I can see how each of those String objects contain a pointer 16 bytes ahead of the previous one, so it’s allocating 16 bytes of heap for each String object it initializes. The last pointer it writes in the ramtest array before it crashes in my case is in 2000FFA0, and the stack pointer is at that same address, so the heap has crashed the stack.
So if you want to use the String class, you need to understand it will allocate much more than just 12 bytes per string, and that’s with empty strings, I can imagine if this did not crash and you wanted to then give a string object a content that’s let’s say 100 bytes long, it will have to call the copy constructor again and allocate another 100bytes from the heap.
I do not know anything about the string class other than what I saw now in the bdebugger, so unless someone else in the forum can give you some advice, your best bet is to post in the Arduino forums, or the Teensy forum, since it looks like Paul Stoffregen rewrotte the whole String class.
I think this brings us up to my initial question, that now is not off topic any more, but why do you want to create thousands of String objects?
It seems to me that even if you got a different MCU that could fit all those, as soon as you start modifying Strings and they have to be copied again to get more space, you would deplete the memory of any MCU.
Good link about what’s happening to your:
https://learn.adafruit.com/memories-of- … izing-sram
Thanks for the info on the String class.
But, I suspect there may still be a problem with LibMaple
Looking at the addresses in RAM which it assigned to the Strings in global scope, (and also things like Serial and Serial1 etc)
The addresses its using are around 5k from the base of RAM, and from the map file I can’t see very much using the space below the 5k region
STM32GENERIC seems to allocate global vars much lower to the start of RAM (within a few hundred bytes).
But their memory organisation may be totally different.
Unfortunately I don’t know enough about the linker directives to understand why .bss seems to be 5k from the base of RAM
I assume its reserving that space for something else, but I’m not sure what.
Before that all looks normal, but there is a big chunk related to libc impure_data. I believe there was a longer discussion about that a while back in the forum, and it’s due to using malloc. Since the string class uses malloc, it’s pulling all this in. I dont think there is a problem with libmaple, but there is a problem with String for sure. Also there is some interesting posts from Paul in the Arduino forum explaining how the Arduino team refused to implement most of his fixes for that class and picked only some of them. Some of those threads are old, and more changes may have been brought over, but still may be convenient to look at how is the String class for the Teensy3, since Paul apparently had devoted a lot of time on corrections the Arduino team did not want to merge, so he applied those fixes to the teensy cores.
*fill* 0x2000057c 0x4
.data.impure_data
0x20000580 0x428 c:/sloeber/arduinoplugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/lib/armv7-m\libc.a(lib_a-impure.o)
.data._impure_ptr
0x200009a8 0x4 c:/sloeber/arduinoplugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/lib/armv7-m\libc.a(lib_a-impure.o)
0x200009a8 _impure_ptr
.data.__malloc_av_
0x200009ac 0x408 c:/sloeber/arduinoplugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/lib/armv7-m\libc.a(lib_a-mallocr.o)
0x200009ac __malloc_av_
.data.__malloc_trim_threshold
0x20000db4 0x4 c:/sloeber/arduinoplugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/lib/armv7-m\libc.a(lib_a-mallocr.o)
0x20000db4 __malloc_trim_threshold
.data.__malloc_sbrk_base
0x20000db8 0x4 c:/sloeber/arduinoplugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/lib/armv7-m\libc.a(lib_a-mallocr.o)
0x20000db8 __malloc_sbrk_base
0x20000dc0 . = ALIGN (0x8)
*fill* 0x20000dbc 0x4
0x20000dc0 __data_end__ = .
Thanks…
On the Maple mini .bss seems to be df0 which is exactly 3k from the start of RAM, which makes me think there is a setting somewhere which controls this
I agree we can’t really change this, as the RAM below that area will be in use for other things.
At the moment I think we have bigger fish to fry than optimising Strings to make better use of memory.
IMHO the String class should be avoided for embedded development. I hardly ever use it with the STM32, however I noticed the ESP8266 core makes extensive use of Strings (but they probably have more RAM to play with)
[stevestrong – Fri Sep 01, 2017 9:31 am] –
Can’t we just look at Teensy’s String class and take over some parts (all?) of it?
I don’t have time to do it.
[RogerClark – Fri Sep 01, 2017 4:32 am] –
IMHO the String class should be avoided for embedded development. I hardly ever use it with the STM32, however I noticed the ESP8266 core makes extensive use of Strings (but they probably have more RAM to play with)
+1
I dont use String either, I have already spent enough time to find if there was a bug in the core and there is not. If someone is using String and has the time to compare to teensy etc, then submit a PR, we can revisit.
The original poster doesn’t seem to be showing us the code that he is really running. So for us to try and figure out what he is doing is just a waste of time. I can’t really imagine trying to use an array of Strings the way he has it coded. I’m guessing he meant to allocate one String instance that is 2170 bytes long.
[Rick Kimball – Fri Sep 01, 2017 4:51 pm] –
I can’t really imagine trying to use an array of Strings the way he has it coded. I’m guessing he meant to allocate one String instance that is 2170 bytes long.
Right ! I think you put the finger on it !
In many forums, I’ve seen some newbies that could not even distinguish the difference between a struct allocations and a pointers to structs, causing null pointer exceptions.
commercial ?
srp
[zmemw16 – Fri Sep 01, 2017 5:16 pm] –
homework ?
commercial ?
srp
What do you mean ?
my normal response is a google search string
someone working on something commercial and trying not to give too much away – succeeding
or
someone yanking our chains
srp
[martinayotte – Fri Sep 01, 2017 5:36 pm] –
Here is an example I found this morning …
https://esp32.com/viewtopic.php?f=2&t=2870
And we thought that some questions here didn’t make sense…
[Rick Kimball – Fri Sep 01, 2017 4:51 pm] –
Sometimes I wonder if people come in here and post questions just to troll us?The original poster doesn’t seem to be showing us the code that he is really running. So for us to try and figure out what he is doing is just a waste of time. I can’t really imagine trying to use an array of Strings the way he has it coded. I’m guessing he meant to allocate one String instance that is 2170 bytes long.
Hi Rick. I don’t want to troll u. My main code is running out of memory with no reason. So I just try to fill all the memory to see what happen. I DON’T USE THAT CODE, it a simplr sample to test the FULL user of memory.
[Rick Kimball – Fri Sep 01, 2017 8:32 pm] –
Why don’t you post that code then?
My code has 9k lines of code. The code FREEZE without reason if I add any String or use Serial.print(); or Serial.print(F()), with just 23k of memory ALOCATED. I use a lot of SERIAL PROCESSOR in that code. That is the cause I did that sample code to test the RAM.
I did other test now with other sample and I can just use 23k of the RAM with String, but if I use other arrays of FLOAT and DOUBLE, I can use the full 64k.
Rick, I’m not expert of embedded processor, but I think the STM32DUINO is focus in our case, cause it help A LOT build embedded CODE without be a EXPERT.
Thanks for the attention!
Are you using an array of Strings or a single String and trying iterate though each item in the single String?
What does that 9k of code do?
Can you solve your problem not using String?
Do you understand how String works? Each String item allocates 12 bytes before you even put anything in it.
class String : {…
a buffer *
an int containing its capacity
an int containing its actual length.
4 bytes + 4bytes + 4bytes
}
When you actually put something into the String value it allocates more memory and sets the buffer pointer to that address. However, malloc does not ask for the exact number of characters you ask for. (Actually it uses realloc())
String mystring; // reserves 12 bytes in the .bss this is not available for heap or stack.
void somefunc() {
…
mystring = ‘A’; // allocates 16 bytes from heap
…
}
The line of code above actually allocates at least 16 bytes because String it assumes you are going to grow the string
(Otherwise why wouldn’t you be using some other data storage primitive) Also, malloc works better when it can allocate and free items of the same size.
So we don’t actually know how you are using your String objects. Without seeing your code flow we can only guess.
Is that what you expected?
You can also post the sections where you use Strings, as you have seen above String is a memory hog, so perhaps there perhaps there is a different way to achieve what you want to do in a different way that will not take all that mem.
If you can’t share those parts, then it’s impossible for us to know what’s going on. As I proved with your example, the heap can get depleted very easily.
[Rick Kimball – Sat Sep 02, 2017 10:55 pm] –
That isn’t what I observed in the debugger.
Did your test deplete heap like it did in mine? if so when did allocate for heap and for what?
If it did’t deplete heap like in my case, did it crash at all?
When I was debugging, I noticed the String.buffer variable being NULL. However, by the time you get to the setup() function, String.buffer does have a valid heap allocated value. The minimum allocation seems to be 16 bytes. So yes, it does reserve the .bss section space and heap memory allocation before setup() even runs.
Sorry for the confusion.
I see that the default constructor for String defaults to the String(const char *cstr = “”) version:
(gdb) p my_array
$3 = {{buffer = 0x20000c60 "", capacity = 0, len = 0}, {buffer = 0x0, capacity = 0, len = 0}}
(gdb) where
#0 String::String (this=0x20000b44 <my_array>, cstr=0x8002e32 "") at /home/kimballr/Arduino/hardware/stm32duino/STM32F1/cores/maple/WString.cpp:34
#1 0x080001ba in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/kimballr/Arduino/sketch_sep03a/sketch_sep03a.ino:1
#2 _GLOBAL__sub_I_my_array () at /home/kimballr/Arduino/sketch_sep03a/sketch_sep03a.ino:10
#3 0x08001216 in __libc_init_array ()
#4 0x08000214 in start_c () at /home/kimballr/Arduino/hardware/stm32duino/STM32F1/variants/generic_stm32f103c/wirish/start_c.c:84
#5 0xfffffffe in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
[gasparobr – Fri Sep 01, 2017 8:58 pm] –
The code FREEZE without reason if I add any String or use Serial.print(); or Serial.print(F()), with just 23k of memory ALOCATED. I use a lot of SERIAL PROCESSOR in that code. That is the cause I did that sample code to test the RAM.I did other test now with other sample and I can just use 23k of the RAM with String, but if I use other arrays of FLOAT and DOUBLE, I can use the full 64k.
As you saw String uses much more ram than the statically allocated at first, and depletes heap. Saying that you can only use 23KB with String is not correct, you are allocating much more, 23KB at compile time and the rest to complete 64KB at run time.
You should check if that is the case with your code, and you are depleting the heap or the stack.
I will update the title of this thread until you can provide some other test piece of code, since the one in the original post does not show any problem with memory allocation in the MCU or the core, but rather shows that creating an array of thousands of String class objects takes a lot of memory at run time.
One thing I have noticed is that malloc() does not seem to fail gracefully.
Its supposed to return null if it cant allocate the memory, but in practice if you keep allocating 8 bytes, eventually the code will crash.
I observed the same behaviour with the Nordic nRF51, which is also ARM based , when I compiled using GCC.
I am not sure if malloc() is the problem, or whether the rest of the core needs a small amount of heap to continue running.
I think I looked though the libmaple core a while ago, to see if malloc was used , and from what I recall, it does not use malloc.
So i presume somehow malloc its self is causing the crash, though I am not sure how that is possible, unless perhaps it allocates memory that is in use by the stack
In fact I’m sure because when we add that in a sketch, we see the extra code being pulled in.
malloc() it’s supposed to call new() if I remember right, and I believe new() does not check where the current stack is, neither it has a hard limit for heap other than the top of RAM (if I remember the code right).
So in that case, new () would continue allocating memory until causing a stack corruption and the core crashing, since no matter how small the stack is, it’s definitely going to be lower than the top of RAM at any given time.
I have seen linker scripts that place a minimum reservation for the stack, or a maximum size for the heap. In either case as long as the stack has not gone lower than that address, it’s all good, but there is always the chance stack grows larger than our estimate and runs into the heap. We could do that, would be a set of small changes to the linker scripts. We just need to decide what’s the minimum space we allocate for the stack, either for all MCUs, or per series.
I dont remember of the top of my head which cores or which linker scripts use that technique but we can look at SAM and Teensy for reference.
Only problem I see is that even if we return an error (or NULL) when the heap reaches a limit, that will happen at run time and most likely just crash the board, exactly as in the example in the first post. Also once something has been running for hours or days, the stack may be going lower, and the heap keeps going up… So it helps identify the problem if the system crashes right at boot up, but I don’t think it totally solves it since the stack size is variable.
Even if new() checked where the stack pointer is right when it’s called, there is no guarantee that the stack pointer won’t go lower once new() returns and a few other functions are called.
This is a good reading on the String class issues:
http://forum.arduino.cc/index.php?topic=115552.0
And there are pointers there to Bill Greyman to how using Heap is completely forbidden in some critical embedded systems:
http://forum.arduino.cc/index.php?topic … #msg869721
Perhaps Rick has some advice on whether there is any value in reserving some stack space.
