You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Bulk imports that continually add files to a tablet that already has a lot of files will eventually cause multiple problems.
Describe the solution you'd like
#5104 adds limits that can prevent this situation from happening, but those default to unlimited. There is also an existing property table.bulk.max.tablets that was introduced in 2.1.0 that defaults to unlimited. After #5104 is merged could change those defaults to a limit and also change the existing property table.bulk.max.tablets to also have a default limit. Could set the limit of all these properties to 100 to start with. When a user encounters the limit they can adjust as needed. This will make the default behavior of the system more sustainable.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
The text was updated successfully, but these errors were encountered:
#5117 (comment) this is an example of one problem that could happen w/ too many files in tablet. Most Accumulo code assumes that all tablet metadata can be read into memory. When a tablet has 25K files and many threads reading that tablets metadata into memory it can cause a lot of memory pressure. There are other problems too many files can cause for merge, split, scan, and compaction also.
Is your feature request related to a problem? Please describe.
Bulk imports that continually add files to a tablet that already has a lot of files will eventually cause multiple problems.
Describe the solution you'd like
#5104 adds limits that can prevent this situation from happening, but those default to unlimited. There is also an existing property table.bulk.max.tablets that was introduced in 2.1.0 that defaults to unlimited. After #5104 is merged could change those defaults to a limit and also change the existing property table.bulk.max.tablets to also have a default limit. Could set the limit of all these properties to 100 to start with. When a user encounters the limit they can adjust as needed. This will make the default behavior of the system more sustainable.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
The text was updated successfully, but these errors were encountered: