Advice on simulating SD card with SPI data protocol

[tawjaw]

Hi @urish
I am planning to simulate an SD card to store data from simulated sensors.
The goal is to have the actual storage be in a database for each student (which is the storage of the SD card) and then they'll be able to load that data in a python environment to do data analysis. I would like to have a proper use of the SD.h and SPI.h libraries.

I just started looking into this but if anyone has any advice or comments I'd appreciate it.
For example can I use SPI communication protocol with the AVR8js?

SS – digital 10. You can use other digital pins, but 10 is generally the default as it is next to the other SPI pins;
MOSI – digital 11;
MISO – digital 12;
SCK – digital 13;

[urish]

Hi Tawfiq, there's an open issue for SPI implementation (#33). From what I remember it should be pretty straightforward as the AVR SPI interface is very simple. Once it is in place, I think it shouldn't be too hard to simulate an SD card. The FAT filesystem could be a little challenging, but there's at least one JavaScript implementation, so that should probably help

[tawjaw]

Thanks Uri, that sounds promising! I am worried about the FAT filesystem, thanks for sharing that library.
I will take a look at the datasheet and try to understand it for the SPI interface.

My quick solution for my students will be to give them an easy way to store the serial print output into a file that they download.
But I definitely would like in the future to simulate an SD card!

[urish]

Yes, I'd love to see this happening as well :)

[tawjaw]

Correct me if I’m wrong:

The SPI communication with the AVR chip does not require modification to the existing AVR simulation.

From what I understood it requires digital and pwm pins to work. Which are complete in AVR8js.

The part that needs to be simulated is how to process the values of the pins based on the clock pin of the communication.

In other words is implementing the SPI communication will be part of the AVR8js library or a separate thing that processes the AVR pins outputs?

[urish]

Hardware SPI does require some little modification, that's what #33 is about. It doesn't use the PWM or GPIO modules, rather it is a separate peripheral (similar to how I2C and USART are implemented).

However, if you check the part of the datasheet that talks about the Hardware SPI (page 176 / section 19.5), it should be pretty simple to implement, as it has far less options that I2C or USART.

[tawjaw]

Yea I saw that after I made my comment.

I’m following your steps in implementing AVRUSART to implement AVRSPI

I’ll let you know when I have something working.
Like you said, the logic of the registers is straight forward.

[urish]

Hi @tawjaw, AVRSPI has now been implemented, just released in avr8js v0.11.0.

I created a quick demo for you, showing the new SPI peripheral in action:

https://stackblitz.com/edit/avr8js-spi?file=index.ts

I'd love to see if you can come up with some code to simulate SD card based on the new SPI peripheral!

[tawjaw]

Hi @urish that's great! sorry the past month has been busy.
I will start playing with the SD card simulation sometime this month after I am done with the deadlines before the new academic semester starts.

How does this map to the actual pins on the Arduino board? if we want to actually "connect" wire to the pin on the Arduino

I think when the wire value is updated, we check for the the SPI pins if SPI is enabled, we use the SPI registers. otherwise we use the value of the pin.
Does that make sense?

[urish]

Yes, that makes total sense. As far as I remember, SPI overrides some of the pins (I think the MOSI pin), and the other pins are only used if they are set to the appropriate mode (OUTPUT for SCK, INPUT for MOSI)

[urish]

Anything I can do to help here?

[tawjaw]

Hi @urish

I did some thinking about the SD card before.

I didn't start implementing anything though.

But the main thing that I was still not sure of, is it better to emulate the memory and the filesystem instead of simulating a 1to1 FAT filesystem memory.

Maybe you can help with the brainstorming.

I was thinking of a JSON based virtual memory with the FAT communication emulated.

This way I can:

• not reserve a full memory if I'm not using it.
• be able to easily store the memory in a database.. if it's all going to be just a JSON object

What do you think?

[urish]

That sounds like a useful approach.

I probably wouldn't go with JSON for the actual file storage, rather with ArrayBuffer, but for the metadata (e.g. directories, file names, etc) that could be a javascript'ish way to go.

One thing that could be challenging is if the simulated code holds any cache of the file system data. Also, have you looked into the FAT filesystem structures?

Also, this package may be helpful: https://github.com/natevw/fatfs

[tawjaw]

Sorry for the late reply. Getting busy with the end of semester now.
I've checked the FAT filesystem.

The structure doesn't seem complicated. It can be done easily for sure.

But what I am not sure of is the performance. Right now I have students using the simulations with basic laptops while on zoom calls and sharing their screens. The simulation in those cases work at 25% performance.

Do you think an emulated SD card (memory + reader device) be better on performance than having a 1 to 1 simulation?

For example, if the SD card is empty I would want no overhead at all.
If the SD card is used to store one file with 100 characters, the memory cost would be proportional to the 100 characters with some simple meta data of the file without an overhead cost.

[urish]

End of semester probably means a lot of work for you!

Yes, FAT isn't too hard. If I remember correctly, filenames are stored in an awkward way, due to backward compatibility concerns. Other than that it should be a pretty simple filesystem.

Ideally, you could only store used clusters. You'll still incur some overhead (I think a few kilobytes of memory), and there's also cluster size overhead (so one 100 characters file will take a whole cluster, that is 512 bytes or a multiple of 512, depending on the total size of FS). I think that this memory overhead is small enough not to be a concern.

In terms of performance, 25% is indeed very slow for a modern laptop. A few questions come to mind:

• Are they seeing different numbers when running the same simulation without zoom?
• Are you using the latest version of avr8js (or a version > 0.14.x)? We had a bunch of performance-related improvements over the last few months.
• Which web browser do they use? Chrome usually works best, Firefox is a bit behind, and Safari sometimes decides to "punish" the browser tab for being too aggressive with that CPU (though there are several workarounds for this)

In general, I have some ideas how to squeeze extra performance, and some are pretty low-hanging fruit. I can share some details if you like.

[tawjaw]

I definitely need to upgrade. I'm using version 0.9.0 on the built site.

The performance is correlated directly with sharing their screen while on zoom call.

I recommend to them using Chrome. With Safari I encounter some weird performance issues sometimes after running the simulation multiple times on the same page. Although I made sure I'm stopping the simulation properly before rerunning.

I had a student using IE (which I was judging lol) and was having a lot of performance issues while on zoom call. After switching to chrome things worked well.

For the SD I will look at the Fat filesystem in more details and let you know of any updates. What you're mentioning is reasonable overhead cost. I was worried I'll need to have a full empty memory size allocated.

[urish]

I definitely need to upgrade. I'm using version 0.9.0 on the built site.

I would love to hear if the upgrade improves the performance during the zoom calls.

Also, I recently switched to requestAnimationFrame() (simplified code and complete working example). This seemed to make Safari much happier and I haven't heard any user complaints on Safari performance since.

I'm surprised to hear that it even worked in IE :)

For the SD I will look at the Fat filesystem in more details and let you know of any updates. What you're mentioning is reasonable overhead cost. I was worried I'll need to have a full empty memory size allocated.

Awesome!

[urish]

Closing this, as I've started working on SD card simulation. See wokwi/wokwi-features#183 for more details.