Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding download all messages feature #1628

Merged
merged 5 commits into from
Jan 16, 2024

Conversation

AlexisSouquiere
Copy link
Collaborator

@AlexisSouquiere AlexisSouquiere commented Dec 8, 2023

Fix #1487

  • Follow Micronaut guidelines about sending large amount of data (StreamedFile + PipedInputStream/PipedOutputStream for custom datasource)
  • Add button to download all the messages next to the filters
    image
  • Add a warning message to warn people about the amount of data
    image
  • Take into account the filters set by the users

Tested locally with a topic of 600K records, downloaded a 220Mb file in less than 1min

Downloaded file format is the following:
image
With Avro messages
image
With Tombstone
image

Feedback is appreciated 😄

@AlexisSouquiere AlexisSouquiere marked this pull request as draft December 8, 2023 14:22
@AlexisSouquiere AlexisSouquiere marked this pull request as ready for review December 11, 2023 09:46
@AlexisSouquiere
Copy link
Collaborator Author

AlexisSouquiere commented Dec 11, 2023

I will have to take into account the endTimestamp filter in the new download endpoint if #1629 is merged before

@tchiotludo
Copy link
Owner

tchiotludo commented Dec 17, 2023

@AlexisSouquiere your PR #1629 is merged before, sorry 😅

@tchiotludo
Copy link
Owner

@AlexisSouquiere you want to add extra commit on that one to handle endTimestamp?

@AlexisSouquiere
Copy link
Collaborator Author

It's fine for me I took into account the new filter on the download endpoint :)
PR is ready

@tchiotludo tchiotludo merged commit c89db95 into tchiotludo:dev Jan 16, 2024
4 checks passed
@arindampatra33
Copy link
Contributor

@tchiotludo @AlexisSouquiere This commit added a @JsonIgnore to Topic field of Record , this is breaking the tail functionality

@AlexisSouquiere
Copy link
Collaborator Author

@arindampatra33 thanks for the alert. I will check and fix it

@arindampatra33
Copy link
Contributor

arindampatra33 commented Feb 15, 2024

@AlexisSouquiere its failing with below error in the UI when its trying to read topic.name
I guess you have to revert the JsonIgnore topic is an important field

Tail.jsx:398 Uncaught TypeError: Cannot read properties of undefined (reading 'name')
at rowId (Tail.jsx:398:33)
at Table.jsx:519:17
at Array.forEach ()
at o.value (Table.jsx:517:10)
at Wa (react-dom.production.min.js:182:192)
at Ba (react-dom.production.min.js:181:224)
at Ss (react-dom.production.min.js:263:490)
at vc (react-dom.production.min.js:246:265)
at mc (react-dom.production.min.js:246:194)

@arindampatra33
Copy link
Contributor

@AlexisSouquiere If you added it just to have lesser number of columns in the downloaded data , you can create a new model and populate. from Record data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Ability to download all of a topic's messages
3 participants