fast reading files by STM32F103C8T6

hi guys , I really appreciated for any help or hint about STM feature and capability for lighting project, actually I ran the POV project with teensy 3.1 which is read the image from SD card and show to LED strip line by line . for get benefits of new ESP webserver and websocket lib features I’ve moved to run the POV on ESP, however reading image files from SPIFF was so slow, whereas need to, at least, read 2.5 Mbyte/s , I’m thinking maybe it’d be good idea to upload and manage the animation show by ESP and let the slave devise such STM32F103C8T6 handle the POV algorithm, do you think STM chip with 64K Byte Flash & 20KByte SRAM could handle such task? I might add I’m using fastLED lib and hopefully can run on your STM lib
any tips really appreciated

Could you elabourate on what POV means in your context.

Re:File transfer speed

2.5 M bytes per sec is a very high data rate for SPI transfer to SD card.

See
http://elm-chan.org/docs/mmc/mmc_e.html

We can overclock the SPI to 36MHz but I dont know if any SD cards go that fast.

I suspct you would need to use SDIO, but that feature is not available on the STM32F103C8 and we dont have a SDIO library for the libmaple core.

I am sure a F4 series MCU could process that much data from SDIO, but as there currently isn’t an SDIO library, You should probably try to get hold of a Raspberry Pi Zero or another high speed unix board e.g. Orange Pi

@mkeyno, I remember seeing your discussion on ESP gitter…

About your tests with a Teensy 3.1, which performance did you achieved ?

You might also consider something like a SPI Flash chip rather than an SD card.
Something like these.

… and also avoid any filesystem’s layer, since your needs is simply to get huge array of RGB with a known length.

so sorry for late respond , I’ve just notified , and many thanks you all for reply and leave you notes,
@ RogerClark , POV is the vision come with rotation of one line of LED to product the whole image , I did it with teensy3.1 connected to SD card which is simply read image, line by line and put in LED strip , to get benefits of ESP feature , I’ve tried to use SPIFF for hosting the image files and webserver feature to upload the images , however after couple of tried and many discuss in ESP gitter room , noticed the ESP not fast enough to read the image and beside file fragmentation hold the speed back to optimum
in best try I reach around 60ms for each frame , for 25 frame/s I should deliver 144 LED pixel * 3 byte RGB data * 200 line of image = about 85Kbyte for each frame and about 2 Mbyte/s
I thought with STMF103 DMA feature can reach such performance whereas it is connected to ESP module as slave module , I’m not yet decide to test it but some guys told me may ESP32 with two extra separated core with free DMA feature could handle it
if you guys have any experience for SD card fast reading by STMF103 I’ve really appreciated to have your hints

mkeyno wrote:so sorry for late respond , I’ve just notified , and many thanks you all for reply and leave you notes,
@ RogerClark , POV is the vision come with rotation of one line of LED to product the whole image , I did it with teensy3.1 connected to SD card which is simply read image, line by line and put in LED strip , to get benefits of ESP feature , I’ve tried to use SPIFF for hosting the image files and webserver feature to upload the images , however after couple of tried and many discuss in ESP gitter room , noticed the ESP not fast enough to read the image and beside file fragmentation hold the speed back to optimum
in best try I reach around 60ms for each frame , for 25 frame/s I should deliver 144 LED pixel * 3 byte RGB data * 200 line of image = about 85Kbyte for each frame and about 2 Mbyte/s
I thought with STMF103 DMA feature can reach such performance whereas it is connected to ESP module as slave module , I’m not yet decide to test it but some guys told me may ESP32 with two extra separated core with free DMA feature could handle it
if you guys have any experience for SD card fast reading by STMF103 I’ve really appreciated to have your hints

martinayotte wrote:… and also avoid any filesystem’s layer, since your needs is simply to get huge array of RGB with a known length.

racemaniac wrote:
I’ve recently been encountering similar requirements with streaming small soundsamples i wanted to play on my maple mini, and to be honest, i never even bothered with SD cards… The current dma implementation is fast, but blocking, you can try altering that, but if you look at the code, the SD card filesystem overhead is HUGE (also the inconsistency of how & when it responds due to internal management doesn’t bode well either).
I don’t know how much data you need, but i ended up just buying some spi flash chips (the biggest ones i have are 1Gb (=128MByte)), and it’s sooo much easier working with them. A simple protocol to read data from them, i made a custom “filesystem” (the first 1k of the flash chip contains metadata & addresses of what is stored where, and i keep it into memory). And then i was able to quickly write a simple framework that allows me to stream many channels of wav audio data from the flash chip at the full speed of the SPI port, and nicely in the background with DMA .

So unless you realy need gigabytes of data, you should really consider the flash chip option. The only usage of an SD card in my project was to load the data into the flash chip XD.

mkeyno wrote:… 144 LED pixel * 3 byte RGB data * 200 line of image …

Also, if you don’t use the whole spectrum of 3*RGB(8bits)=24 bits for one LED, then you could maybe compress the color data information.

stevestrong wrote:mkeyno wrote:… 144 LED pixel * 3 byte RGB data * 200 line of image …

Hmmm, can a Teensy3.1 reach such a SD read-out speed and the STM32 not? I wonder why…

mkeyno wrote:racemaniac wrote:
I’ve recently been encountering similar requirements with streaming small soundsamples i wanted to play on my maple mini, and to be honest, i never even bothered with SD cards… The current dma implementation is fast, but blocking, you can try altering that, but if you look at the code, the SD card filesystem overhead is HUGE (also the inconsistency of how & when it responds due to internal management doesn’t bode well either).
I don’t know how much data you need, but i ended up just buying some spi flash chips (the biggest ones i have are 1Gb (=128MByte)), and it’s sooo much easier working with them. A simple protocol to read data from them, i made a custom “filesystem” (the first 1k of the flash chip contains metadata & addresses of what is stored where, and i keep it into memory). And then i was able to quickly write a simple framework that allows me to stream many channels of wav audio data from the flash chip at the full speed of the SPI port, and nicely in the background with DMA .

So unless you realy need gigabytes of data, you should really consider the flash chip option. The only usage of an SD card in my project was to load the data into the flash chip XD.

stevestrong wrote:Hmmm, can a Teensy3.1 reach such a SD read-out speed and the STM32 not? I wonder why…

racemaniac wrote:
I don’t know much about the ESP capabilities. But for the stm32: the SPI port can at max do 32Mhz, that’s 32 MegaBIT/s, so 4Megabyte per second. No software implementation will come close to that. And you can look into the SDFAT thread about benchmarks of sd cards on the maple mini, i think they barely surpassed 1Megabyte/s…. if you want 2.5 megabyte/s speed, i’d suggest going for a flashchip and an stm32 handling the data & controlling the leds. you’ll need something with either a hardware port, or a much higher cpu speed than the esp/stm32 to get such datarates, and sd cards addressed via spi won’t get it either….

mkeyno wrote:racemaniac wrote:
I don’t know much about the ESP capabilities. But for the stm32: the SPI port can at max do 32Mhz, that’s 32 MegaBIT/s, so 4Megabyte per second. No software implementation will come close to that. And you can look into the SDFAT thread about benchmarks of sd cards on the maple mini, i think they barely surpassed 1Megabyte/s…. if you want 2.5 megabyte/s speed, i’d suggest going for a flashchip and an stm32 handling the data & controlling the leds. you’ll need something with either a hardware port, or a much higher cpu speed than the esp/stm32 to get such datarates, and sd cards addressed via spi won’t get it either….

racemaniac wrote:Something like this for example: https://www.winbond.com/resource-files/ … _rev_g.pdf
There are many flash chips you can find on sites like aliexpress, all with similar protocols & features .

mkeyno wrote: I don’t know exactly how ESP can’t reach to best performance

mkeyno wrote:racemaniac wrote:Something like this for example: https://www.winbond.com/resource-files/ … _rev_g.pdf
There are many flash chips you can find on sites like aliexpress, all with similar protocols & features .

racemaniac wrote:the ESP doesn’t have a hardware spi port

martinayotte wrote:racemaniac wrote:the ESP doesn’t have a hardware spi port

martinayotte wrote:This can also be done with a SDCard without formatting it (no FAT16, no FAT32), simply accessing sectors by sectors as raw data stream, it will speed up things.

Still using low level part of SdFat, there are function like readBlock() and readBlocks() to read 512bytes sectors from the card.
I’ve never done that with SDCard yet, but should be doable. The thing I’ve done which is a bit similar is reading an SPIFlash on STM32 using readBytes()

racemaniac wrote:
the ESP doesn’t have a hardware spi port, so you indeed won’t be able to get high datarates from it…
So your best bet is to use an stm32 as slave to do the heavy lifting for you

why the esp btw? can you wirelessly send it new pictures?

I’m still convince that it can be done with ESP8266 simply by skipping the SPIFFS and using plain raw data stored in the flash, reading chunks using memcpy_P(), a bit like and PROGMEM is used.
To verify that, we need to write some kind of benchmark, then, if results are good, hookup your WS2801 RGB LED strip.

martinayotte wrote:Still using low level part of SdFat, there are function like readBlock() and readBlocks() to read 512bytes sectors from the card.
I’ve never done that with SDCard yet, but should be doable. The thing I’ve done which is a bit similar is reading an SPIFlash on STM32 using readBytes()

martinayotte wrote:I’m still convince that it can be done with ESP8266 simply by skipping the SPIFFS and using plain raw data stored in the flash, reading chunks using memcpy_P(), a bit like and PROGMEM is used.
To verify that, we need to write some kind of benchmark, then, if results are good, hookup your WS2801 RGB LED strip.

I did a small flashRead() performance test on ESP, and the following code gave be a throughput of 16MB/sec for 1MBytes length :

char mem[1024];
uint32_t flashStartAddress = (uint32_t)&_SPIFFS_start - 0x40200000;
uint32_t begin = millis();
uint32_t elapsed = 0;
p.println("Flash Mem Perf Test :");
for (int i = 0; i < 1024; i++) {
ESP.flashRead(flashStartAddress, (uint32_t *)&mem, sizeof(mem) / sizeof(uint32_t));
flashStartAddress += sizeof(mem) / sizeof(uint32_t);
}
elapsed = millis() - begin;
p.printf("\nelapsed time = %u \r\n", elapsed);
p.printf("throughput = %u bytes/sec \r\n", 1024 * 1024 * 1000 / elapsed);


martinayotte wrote:I did a small flashRead() performance test on ESP, and the following code gave be a throughput of 16MB/sec for 1MBytes length :

char mem[1024];
uint32_t flashStartAddress = (uint32_t)&_SPIFFS_start - 0x40200000;
uint32_t begin = millis();
uint32_t elapsed = 0;
p.println("Flash Mem Perf Test :");
for (int i = 0; i < 1024; i++) {
ESP.flashRead(flashStartAddress, (uint32_t *)&mem, sizeof(mem) / sizeof(uint32_t));
flashStartAddress += sizeof(mem) / sizeof(uint32_t);
}
elapsed = millis() - begin;
p.printf("\nelapsed time = %u \r\n", elapsed);
p.printf("throughput = %u bytes/sec \r\n", 1024 * 1024 * 1000 / elapsed);


The only part of the sketch that is missing and maybe you a struggling into is this that should be on top of the sketch :

extern "C" uint32_t _SPIFFS_start;


dear @martinayotte , I’ve made sketch base on your hints and other guys in gitter and uploaded in my repohttps://github.com/mkeyno/ESP_fast_reading , although it seems the speed performance is increased but it throw exception when try to using the read data from flash , can you check the sketch , it if fully tested and ready to upload
Thanks

Unfortunately, I can’t test easily, since I don’t have such RGB strips.
I can try to extract piece of code to debug, but I’m not sure which one : is it the raw reads pushed to LEDs or the copy from SPIFFS to raw writes ?
For the exceptions you still see, did you use the ExceptionDecoder to get meaningful stack trace ?

martinayotte wrote:Unfortunately, I can’t test easily, since I don’t have such RGB strips.
I can try to extract piece of code to debug, but I’m not sure which one : is it the raw reads pushed to LEDs or the copy from SPIFFS to raw writes ?
For the exceptions you still see, did you use the ExceptionDecoder to get meaningful stack trace ?

Do you means that you don’t have exception anymore ?

martinayotte wrote:Do you means that you don’t have exception anymore ?

Which memcpy giving you exception ?
Is it a memcpy() that you are doing with a raw Flash address ?
If Yes, then I can tell you that this is not allowed, you need to memcpy_P() instead.

martinayotte wrote:Which memcpy giving you exception ?
Is it a memcpy() that you are doing with a raw Flash address ?
If Yes, then I can tell you that this is not allowed, you need to memcpy_P() instead.