USB Composite library -> sdreader example SDHC windows 7

Hello,
Have someone successfully used the “USB composite” library example “sdreader” with SdFatEX (also tried SdFat) and SDHC cards (like 32GB) under Windows?
I tried out two cards: 2GB standard and 32GB SDHC (both cards formatted with the official SD tool):
Both cards are working good with all test examples (like benchmark) so it’s not a card/reader or connection problem.
Under OSX both cards are available and correct shown, but slow (another problem)
Under Windows 7 (same machine -> macbook pro 2009) ONLY the 2GB card is recognized (and much faster than on OSX), but the 32GB isn’t correct shown (only 848 MB) and Windows cries for reformatting (but can’t do that)
I also tried lowering/rising the SPI speed.
This is the – more or less – standard example code: I guess there must be something more configured for using it with windows and SDHC cards?
// This uses the greiman sdfat library.
// To use SdFatEX, set ENABLE_EXTENDED_TRANSFER_CLASS to 1 in the library's
// src/SdFatConfig.h
#include <USBComposite.h>
#include <SPI.h>
#include "SdFat.h"

#define LED_PIN 33
#define PRODUCT_ID 0x29

SdFatEX sd;

///const uint32_t speed = SD_SCK_MHZ(18) ;
const uint8_t SD_CHIP_SELECT = 20;
bool enabled = false;
uint32 cardSize;

bool write(uint32_t memoryOffset, const uint8_t *writebuff, uint16_t transferLength) {
return sd.card()->writeBlocks(memoryOffset/512, writebuff, transferLength/512);
}

bool read(uint32_t memoryOffset, uint8_t *readbuff, uint16_t transferLength) {
return sd.card()->readBlocks(memoryOffset/512, readbuff, transferLength/512);
}

void setup() {
USBComposite.setProductId(PRODUCT_ID);
pinMode(LED_PIN,OUTPUT);
digitalWrite(LED_PIN,1);
}

void initReader() {
digitalWrite(LED_PIN,0);
cardSize = sd.card()->cardSize();
MassStorage.setDrive(0, cardSize*512, read, write);
MassStorage.registerComponent();
CompositeSerial.registerComponent();
USBComposite.begin();
enabled=true;
}

void loop() {
if (!enabled) {
if (sd.begin(SD_CHIP_SELECT,SPI_CLOCK_DIV2)) {
initReader();
}
else {
delay(50);
}
}
else {
MassStorage.loop();
}
}


Ok, maybe it’s also a problem with the format:
SD formatter is using FAT16 for cards <= 2GB and FAT32 for bigger ones, so my
2GB card = FAT16
32GB card = FAT32
Connecting the 32GB directly to Windows 7 it’s shown up correct.

Looking at the mass storage code, it addresses data using uint32 byte offsets (rather than sector offsets), and hence cannot work for more than 4gb. This was in the code I adapted to use with the library, but it would be well worth changing.

Thank you, you hit the point!
I changed in USBMassStorage.h and USBMassStorage.cpp

void setDrive(uint32 driveNumber, uint32 byteSize, MassStorageReader reader,
MassStorageWriter writer = NULL, MassStorageStatuser = NULL, MassStorageInitializer = NULL);

Only drawback is the write speed: copying an MP3 folder from Windows @ 146KB/s (highest SPI speed, SdfatEX, DMA enabled, SPI port 1 (standard)) but on OSX it is slower.
Edit on my Windows 10 Home PC I got 177KB/s…

I wonder if it’s better to do uint64 byte addressing or to use sector numbers. An advantage of uint64 is that uint32 sector numbers will have a 2TB limit.

**Edit**: Nevermind. The SCSI commands that are implemented have a 2TB limit anyway.

You’ll need to change a bunch more stuff if you want to read beyond the first 4GB of data, though.

OK, I see:
I took about 12GB on the SD card and after a 1/3 of the subfolders are not shown correctly.
I think more than 2TB isn’t really useful for embedded devices (because lack of RAM), but <4GB is far too less for practical things (storing/playing music, videos, data logger)

4GB isn’t really very small, but then on stm32f103cb we’ve (only) got 20k sram
nevertheless, i think it is still a worthwhile effort to ‘break’ the 4GB ‘barrier’, but it may cost more of that 20k sram

I think it’s an important step, caused by the SD cards you can buy: Cards less than 4GB will disappear from the market quickly (expect for special needs and they will become more expensive).
I don’t think, that it will costs really more RAM, but I’m not the expert.
I’m investigating the whole library and dependencies, many things are unclear for me (especially all dependencies)
Meanwhile I got:

• USBMassStorage.h
  USBMassStorage.cpp
  usb_mass_mal.h
  usb_mass_mal.c
  usb_mass.h (?)
  usb_mass.c (?)
  usb_scsi.h (?)
  usb_scsi.c (?)
  all *mass examples

well, i’m not too sure if this may partly be due to some of the implementation details
out of curiosity i take a look at the scsi command reference
https://www.seagate.com/files/staticfil … 93068h.pdf
from the contents page it starts at p.89 READ(6), READ (10), READ (12) and READ (16)
READ (6) use only 16 bits for the LBA address , this gives at most 32 megs, hence we are unlikely to be using this
READ (10) use a 32 bits LBA address but a 16 bits transfer length, i’d guess this should be adequate
if i work 2^32 * 512 that gives 2TB, 16 bits transfer length i’d guess means a max transfer length of 32 megs for that transaction.
i’m not too sure if this has any implications, but i’d guess it’d still work if the reads are 1 or a few sectors of data.
READ (12) use 32 bits LBA and a 32 bits transfer length
while READ (16) use 64 bits LBA and a 32 bits transfer length

usb-mass storage is basically SCSI over usb, and this article shows an interesting dependency on READ(10)
https://blogs.msdn.microsoft.com/usbcor … ompliance/
which is most likely used by usb-mass storage on windows as well as other os (e.g. linux)
the other part which is somewhat more opaque is the commands to read from / write to the sd card
the closest is from the mmc cards the rest can hardly be found on the internet
http://elm-chan.org/docs/mmc/mmc_e.html

this issue may be ‘difficult to debug’ as it would seem that with 32 bits LBA we’d be able to address any of those 2TB blocks (512 bytes sectors) on the sd card. i’m not too sure what could be causing the desktop os to deem that the sd card is only 4GB large

oh yes, we are not talking about sd-fat here right? be sure to separate the two. usb-mass storage don’t use much of the sd-fat except for perhaps the parts that read/write sectors directly on the sd-card.

(i think as apruss puts it, i’ve not dug into the codes yet, it is memory addressing rather than sector addressing which caused the limits. 2^23 * 512 = 4GB, i think using sector addressing may save some memory compared to using int64 if it gets passed around and is duplicated in structures as int64 takes 8 bytes for that rather than 4. but if it is sd-fat’s codes that are used, it may mean having to revise sd-fat parts for the sd-access instead)

sdfat(EX) do everything right, even with cards over 4GB (I tested the folders and files via serial).
I think building everything to sectors will break the SD cards FAT16 or?

I changed the mass storage code to work in terms of sectors, thereby in theory allowing up to 2TB. I haven’t actually tested anything large — please do.

The read and write callbacks need to be changed as now they are in terms of sectors. See the revised mass and sdreader examples. I also changed the order of the arguments in the callbacks so that the compiler will warn you if you compile old code with the new library.

Thanks, Arpruss!
Tested it with two cards:
2GB SD (Fat16)
32GB SDHC (Fat32, with files over 12GB)
Now both cards are working!
Windows 10: Strangely the 2GB is recognized immediately, the 32GB after some seconds with a warning, that the card has to be checked. But it works without problem.
Writing speed SDHC about 177kB/s – as I read in other forums, that’s the “normal” speed (read posts in several forums (Keil, HAL, CUBE…)) using SPI. SDIO should boost it to 2.5x. (Not tested, only my STM32F407 board has a SDIO soldered SD-card reader)
Reading speed SDHC flickers between 200 and 560 kB/s. (So average is about 400 kB/s, not too bad)

i think the other bottleneck is actually working with the usb 2.0 full speed parts usb 2.0 full speed in theory give 10 Mbps that gives something closer to 1 MBps at most.

I think working on more speed is a real expert thing.
Some offtopic: I’m trying to disable and enable the USB mass storage, so that Windows (or any other OS) would recognized switching cards, doing “USBComposite.end()” works as expected. The card would disappear, but USBComposite.begin();wouldn’t show up the new card (but the same card!) again. Only way I found out is a code based reset. _> nvic_sys_reset();
Maybe this is caused by the SDfat library, cause there is nothing like “sd.end()” or Windows needs some further code.

Here is a working code for changing SD cards via a button, I had removed the USB serial class, because I don’t need that.
// This uses the greiman sdfat library.
// To use SdFatEX, set ENABLE_EXTENDED_TRANSFER_CLASS to 1 in the library's
// src/SdFatConfig.h
#include <USBComposite.h>
#include <SPI.h>
#include "SdFat.h"

USBMassStorage MassStorage;

#define LED_PIN 33
#define BUTTON_PIN 32
#define PRODUCT_ID 0x29

SdFatEX sd;
const uint32_t speed = SPI_CLOCK_DIV2 ;
const uint8_t SD_CHIP_SELECT = 20;
bool enabled = false;
uint32 cardSize;

bool write(const uint8_t *writebuff, uint32_t startSector, uint16_t numSectors) {
return sd.card()->writeBlocks(startSector, writebuff, numSectors);
}

bool read(uint8_t *readbuff, uint32_t startSector, uint16_t numSectors) {
return sd.card()->readBlocks(startSector, readbuff, numSectors);
}

void setup() {
USBComposite.setProductId(PRODUCT_ID);
pinMode(LED_PIN, OUTPUT);
pinMode(BUTTON_PIN, INPUT);
digitalWrite(LED_PIN, 1);
}

void initReader() {
digitalWrite(LED_PIN, 0);
cardSize = sd.card()->cardSize();
MassStorage.setDriveData(0, cardSize, read, write);
MassStorage.registerComponent();
USBComposite.begin();
//USBComposite.end();
enabled = true;
}

void loop() {
if (!enabled) {
if (sd.begin(SD_CHIP_SELECT, SD_SCK_MHZ(50))) {
initReader();
}
else {
delay(50);
}
}
else {
MassStorage.loop();
if (digitalRead(BUTTON_PIN)) {
MassStorage.clearDrives();
MassStorage.end();
USBComposite.end();
digitalWrite(LED_PIN, 1);
delay(1500);
enabled = false;
}
}
}

it would seem to me that change of removable media is normally done from the host os side

there is something rather ‘ancient’ deemed as ufi or floppy interface, but it seemed even that does not provide a means to detect a floppy change
http://www.usb.org/developers/docs/devc … -ufi10.pdf

more usually, the host need to ‘eject’ or ‘unmount’ the device

usb has something called usb-reset, i.e. single ended zero (i.e. pull both the d+ d- lines low) for 10ms
http://www.usbmadesimple.co.uk/ums_3.htm
this is supposed to get the host side to re-do enumeration. but that’d also mean the device should do a reset as well
it could be a workaround for media change, but i’m not sure if better methods are available

ag123: A few seconds too late
I edited my post with a working code!
MassStorage.clearDrives();
MassStorage.end();
USBComposite.end();