Non blocking SPI DMA – Added callback to the SPI DMA functions (dmaSend, dmaTransfer…)

Sorry, now I have foot in my mouth. The F1 and other STM MCUs share the same name for the controllers in the peripheral buses. So here is the summary for F4/F7/H7
1) DMA Controllers have multiple streams
2) Streams have multiple channels
3) The stream channels are hard-wired to peripheral devices
– stream configuration selects one of the devices

Ollie wrote:Sorry, now I have foot in my mouth. The F1 and other STM MCUs share the same name for the controllers in the peripheral buses. So here is the summary for F4/F7/H7
1) DMA Controllers have multiple streams
2) Streams have multiple channels
3) The stream channels are hard-wired to peripheral devices
– stream configuration selects one of the devices

victor_pv wrote:From what I have found, they have not integrated any DMA functions in the normal SPI library.
There is another library, DmaSpi, which is the one that has DMA capabilities:

https://github.com/crteensy/DmaSpi

That one works similar to what Steve was working on, it queues transfers with a series of properties (buffer pointer, size, and a pin object to control CS), then services those transfers in order.

Not sure if all that is worth the effort, as Steve tested, for small transfers doesn’t improve performance due to all the overhead, and the libraries using it have to be heavily modified.

racemaniac wrote:victor_pv wrote:From what I have found, they have not integrated any DMA functions in the normal SPI library.
There is another library, DmaSpi, which is the one that has DMA capabilities:

https://github.com/crteensy/DmaSpi

That one works similar to what Steve was working on, it queues transfers with a series of properties (buffer pointer, size, and a pin object to control CS), then services those transfers in order.

Not sure if all that is worth the effort, as Steve tested, for small transfers doesn’t improve performance due to all the overhead, and the libraries using it have to be heavily modified.

victor_pv wrote:racemaniac wrote:victor_pv wrote:From what I have found, they have not integrated any DMA functions in the normal SPI library.
There is another library, DmaSpi, which is the one that has DMA capabilities:

https://github.com/crteensy/DmaSpi

That one works similar to what Steve was working on, it queues transfers with a series of properties (buffer pointer, size, and a pin object to control CS), then services those transfers in order.

Not sure if all that is worth the effort, as Steve tested, for small transfers doesn’t improve performance due to all the overhead, and the libraries using it have to be heavily modified.

It seems like I have a working version now that allows to:
-Set callback functions that will be called when a DMA transfer completed. In case the callbacks are set dmaSend and dmaTransfer are non-blocking.
-Allow to set all the DMA related settings with one function (enable DMA controller, set transfer address, destination, data size, etc), and then a second function to reload the DMA transfer size, which needs to be reloaded before enabling the channel again, since the value is not kept at the end of a transmission. So if the buffer address, data side, are reused, only the second function needs to be called repeatedly.

I have tested the callback with the sdfat library and with an ILI spi display. Now the weird thing:
When using sdfat at spi div/2 speed, and using callbacks, somethings the DMA RX never completed, and leaves either 1 or 2 bytes pending.
So let’s say I want to receive 512bytes. For that the DMA RX is set to 512 bytes, the DR is read if RXNE is set, RX DMA enabled, and next set and enable TX DMA for 512 bytes.
After each byte goes out, one will come in, and the DMA controller reads it from DR, stores it in the RX buffer, and decrements the count of the RX DMA pending requests.

All works fine if I do not use callbacks, and just block until RX is completed. It also works fine if I set the port to 18Mb (DIV/4) while using callbacks.
But if I use callbacks and set the port to DIV/2 speed, then some times the RX never completes. The TX buffer is all sent, and 1 or 2 bytes still pending in RX. Since TX is completed, it’s not producing a clock any more, and RX will never get the last bytes in.

I have run it thru the debugger, and some times it completes several transmissions correctly before one fails, but is a different number of transmissions each time. Some times goes for longer, some for shorter.
I have tried setting the RX DMA priority to very high and the TX to medium, in case the DMA controller was servicing a TX while an RX was pending, which would overwrite the DR register and get the RX byte lost, but that did not help.

Other than setting a callback for Transfer complete event, the DMA setup is exactly the same whether blocking or not, so I can’t figure out what is happening, other than perhaps even when blocking the transfers are not always completing, but since there is a timeout check, some times the transfer is just being terminated on timeout and not because RX actually completed.

I need to test that theory by removing the timeout, but has anyone experienced any issue when receiving data from with the sdfat library and the max spi port speed when using DMA, or noticed any corruption in the data read?

I think it is normal that Rx bytes are still to be received when Tx ready, since TXE comes earlier than RXNE. Meaning that TXE is set before the last byte is received.
Do you wait in the Tx end callback function for not TXE and BSY?

stevestrong wrote:I think it is normal that Rx bytes are still to be received when Tx ready, since TXE comes earlier than RXNE. Meaning that TXE is set before the last byte is received.
Do you wait in the Tx end callback function for not TXE and BSY?

Ok, so you run both SPI ports in DMA mode…
You could try to let only the Rx part in DMA mode, the Tx part in “normal” mode to see if Rx bytes are still lost.

stevestrong wrote:Ok, so you run both SPI ports in DMA mode…
You could try to let only the Rx part in DMA mode, the Tx part in “normal” mode to see if Rx bytes are still lost.

I think the callback should not have any influence on the test result if you wait for TXE and BSY flags within the Tx end ISR.
When checking figures 246 and 247 overlapped, the one bit (2 APB1 clock) gap between RXNE and TXE should give priority to Rx channel if is set to highest priority (11), please double check that you have set the priorities correctly. DMA for Tx can be set to lowest priority (00).

Furthermore, instead of timeout, you could check the DMA_CNDTRx register value to determine whether there are still bytes to be received when Tx finished.

Does it fail when using only SPI_1 both Tx and Rx DMA with 36MHz clock (SPI2 not working)?

If not, then it is clear a race-condition/limitation of the hardware (bus matrix, AHB system bus and the two bridges to APB1 and APB2 peripheral buses as indicated in figure 2) which cannot handle so many data (DMA and CPU<->RAM) transfer requests within the short time period of one byte transfer at 36MHz, as SPI_1 is served on APB2 and SPI_2 on APB1.

Alternatively you could check what happens when setting the flash wait states down to 1 (CPU at 72MHz) or up to 3.

In addition, you could monitor the MODF and OVR error flags of SPI (enable these IRQs?) and/or TEIF of DMA.

EDIT
Can you please specify more details about how exactly do you test? Do you read blocks of 512 bytes on SPI 1 from SD card and read blocks (how many bytes?) from ILI display repeatedly? Do you use the older SdFat lib or the newer one, SdFat beta? I could maybe test in parallel if you share the testing code.

stevestrong wrote:I think the callback should not have any influence on the test result if you wait for TXE and BSY flags within the Tx end ISR.
When checking figures 246 and 247 overlapped, the one bit (2 APB1 clock) gap between RXNE and TXE should give priority to Rx channel if is set to highest priority (11), please double check that you have set the priorities correctly. DMA for Tx can be set to lowest priority (00).

Furthermore, instead of timeout, you could check the DMA_CNDTRx register value to determine whether there are still bytes to be received when Tx finished.

Does it fail when using only SPI_1 both Tx and Rx DMA with 36MHz clock (SPI2 not working)?

If not, then it is clear a race-condition/limitation of the hardware (bus matrix, AHB system bus and the two bridges to APB1 and APB2 peripheral buses as indicated in figure 2) which cannot handle so many data (DMA and CPU<->RAM) transfer requests within the short time period of one byte transfer at 36MHz, as SPI_1 is served on APB2 and SPI_2 on APB1.

Alternatively you could check what happens when setting the flash wait states down to 1 (CPU at 72MHz) or up to 3.

In addition, you could monitor the MODF and OVR error flags of SPI (enable these IRQs?) and/or TEIF of DMA.

EDIT
Can you please specify more details about how exactly do you test? Do you read blocks of 512 bytes on SPI 1 from SD card and read blocks (how many bytes?) from ILI display repeatedly? Do you use the older SdFat lib or the newer one, SdFat beta? I could maybe test in parallel if you share the testing code.

I don’t think that SPI_DR register for Tx and Rx would mix up, the hardware should handle the correct way of data flow of Tx and Rx data to and from SPI_DR. I think it is rather a matter of limitation on internal data buses.

Still, you did not try to run Tx and Rx DMA on SPI 1 with 36MHz and let SPI 2 disabled/inactive. This would give us more information.

stevestrong wrote:I don’t think that SPI_DR register for Tx and Rx would mix up, the hardware should handle the correct way of data flow of Tx and Rx data to and from SPI_DR. I think it is rather a matter of limitation on internal data buses.

Still, you did not try to run Tx and Rx DMA on SPI 1 with 36MHz and let SPI 2 disabled/inactive. This would give us more information.

Victor, how did you adapt the SdFat lib in order to cope with the callback functionality?
Because as far as I know, the SdFat uses currently SPI transfers in a blocking way.

stevestrong wrote:Victor, how did you adapt the SdFat lib in order to cope with the callback functionality?
Because as far as I know, the SdFat uses currently SPI transfers in a blocking way.

I have done a few tests today with a different board, with an F103RCT6 chip.
I connected MISO to MOSI in SPI1, and repeatedly sent and receive with DMA to 2 buffers. Then compare the content.
All was going well when I was only using SPI1, with or without callback.
Then I started using SPI2 also (spi2 without DMA), and the problems start. Some bits are not received correctly, and changes a 0 for 1 in the last bit of some transferred bytes.

The sketch run transfers in a loop and compares the result at the end. If there is an error, it stops and wait for user input, then can repeat. If there is not error it repeats the loop without stopping. When errors happen they happen every few passes, but not every single one. The errors also happen on different bytes, not always the same.

EDIT: I have repeated the same test using SPI2 with DMA, and also setting SPI1 to Div4 speed. These are the results:

DIV2 Speed on spi1:
SPI1 Alone Without callback: OK
SPI1 alone with callback: OK
SPI1 with callback, in parallel SPI2 without DMA: errors
SPI1 with callback, in parallel SPI2 with DMA: errors

DIV4 speed on spi1:
SPI1 with callback, in parallel SPI2 without DMA: OK
SPI1 with callback, in parallel SPI2 with DMA: OK

Seems so far that errors only happen when SPI1 and SPI2 are working at the same time, and SPI1 is operating at 36Mhz (over specs).
I’ll try to post all my code to github. I’ll stop with this tests here, since it started having issues when I started writting the callback stuff, and didn’t know if the problem was on that. I’m convinced now the callback code doesn’t have any problem, but the only problem was due to using both SPI ports completely in parallel, which was only allowed by using callbacks to signal the end of transfer.

Conclusion for me: spi1 is not reliable when operating at 36Mhz, specially when using another spi port at the same time. Probably ok for only sending data, or when used alone, or non critical reception.

maybe you could try adjusting the relative priority of the interrupts?

Rick Kimball wrote:maybe you could try adjusting the relative priority of the interrupts?

Resurrecting this thread to add the code.

I tested the functions extensively and the only problem I ever found was when using SPI1 at 36Mbit with DMA, as posted above. I verified problem would happen whether using callback or not.
Now, the version I am posting is not the one I tested. I had to redo it to add the latest changes from Roger, and need to retest it, but I am posting it so more people can test it.
This version does not use the dmatubes, although I have another version that does, which I wrote for the F4 support, but Steve is the one working the most in the F4 and his core doesn’t use the dma tubes, so I see no point using that.
I’ll try to test it as soon as I can and post back, if someone finds any problem let me know.

(10.42 KiB) Downloaded 14 times