GDS Uplink and Downlink

[rawinza555]

I have read the user guide and found that the uplink file is limited to 32 MB. Is there a reason why it is limited to only this size and if it is possible to expand the file size limit. I agree that I don't think I would need to upload a file that is bigger than 32 MB but I'd never know if I would need to.

Also, is there a file size limit on the downlink side?

[Joshua-Anderson]

I saw that comment in the documentation as well, but I'm unsure of where that comment is coming from. I think it's possibly a fprime-gds limitation, not a FileUplink limitation, or maybe it doesn't apply anymore. Doing a quick scan of the FileUplink code, I'm not seeing anything obviously limiting the size of files to 32 MB. The file uplink packets have a byte offset field of size U32, so I would expect the max file uplink size should be more like 4.3 GB.

[LeStarch]

The uplink process is two parts: computer through browser to GDS server and GDS server through uplink to flight software. The 32 MB limit comes from that first step and is the limit to how large of a file you can send through the browser.

If more than 32MB is needed, one could chunk the file and upload by parts.

[rawinza555]

Does this imply that if I were to swap out F' GDS to OpenMCT or COSMOS, I could potentially increase the size limit?

[Joshua-Anderson]

Yes, but you'd probably have to work on the OpenMCT/Cosmos integrations a good amount to get them working properly. The 32 MB limit in the F' GDS could also probably be removed if we redesigned the file uplink logic.

[sheershak-agarwal]

I tried uplinking and downlinking large files. Downlink ran fine but Uplink threw segmantion fault at

fprime/Svc/FramingProtocol/FprimeProtocol.cpp

Line 114 in eacdbd9

FW_ASSERT(buffer.getSize() >= size);

It works fine with really small files. I see the comment "adjust size". So should I just increase the size to like 140? While downlink I saw each filePacket is max 140 bytes. I don't know if gds also send 140 bytes max.

[timcanham]

There aren't limits per se on the number of arguments for events, but they all have to pack into a ComBuffer with a finite size:

https://github.com/nasa/fprime/blob/devel/config/FpConfig.hpp#L221

The ComBuffer is tuned on a per-project basis based on the max size of packets the flight to ground system can support.

With headers in the packet, you are really limited to this size for events:

https://github.com/nasa/fprime/blob/devel/config/FpConfig.hpp#L246

[timcanham]

The issue with the assert is that if you are uplinking over a network interface, you may be quickly running out of buffers before they have a chance to write the data and get recycled. You can increase the number of buffers held in the file uplink pool by increasing the buffers here:

https://github.com/nasa/fprime/blob/devel/Ref/Top/instances.fpp#L305

Questions for @LeStarch:

1. We should probably change the assert to a throttled WARNING event, since this is an external input, not a software error.
2. Should the STORE_SIZE match the biggest uplink packet?

[sheershak-agarwal]

Awesome, changing the value of FW_COM_BUFFER_MAX_SIZE solved the event. I can see the bytes forming but on the gds gui, I can only see the partial event string(10 args out of 90). Is the GDS not parsing them or it cant display a big string?

Also what does throttling exactly mean in terms of events?

[timcanham]

I don't know if we've ever tested one with that many arguments before. It may be that the GUI is clipping it. Take a look at <deployment directory>/logs/<date>/event.log and see if it is dumping it to the text file okay.

[sheershak-agarwal]

never mind, my dictionary wasn't updated on gds. Thanks

[sheershak-agarwal]

Alright I will check the logs. Also, I changed the following values but still get error in file uplink.
UPLINK_BUFFER_STORE_SIZE = 6000,
UPLINK_BUFFER_QUEUE_SIZE = 6000.

[timcanham]

How big is the file? In principle, you should allocate max file size/CHUNK_SIZE(256)

[sheershak-agarwal]

I am trying with a 1MB image right now. I want to upload at least 5 MB later.

[sheershak-agarwal]

both store size and queue size? shouldn't queue size be enough?

[timcanham]

STORE_SIZE doesn't really matter over 256, since it stores each 256 byte chunk as it uploads, so it is hugely underutilizing the buffer size if it is 6000. You should do 5*1024*1024 / 256 = 20.5K buffers if you want to guarantee no overruns. Slower radios transfer slowly enough that the software can keep up with a shallower queue, but if you are using a network you should probably allocate the max queue depth.

[sheershak-agarwal]

#1265

[timcanham]

What kind of processor is it?

[sheershak-agarwal]

our current test setup is rpi4

[sheershak-agarwal]

@timcanham I found the issue with this. The uplink also works. I was directly transferring to an external storage device which takes a lot of time to write to it. So I temporarily store the file on the ram and then transfer to the external storage device. I don't know if I should resolve the issue I created?

[aggelosmots]

@sheershak-agarwal What did you do to resolve it? I use a RPI Zero W