[ISSUE #2310]🚀Implement MappedFile isLoaded method #2311
Conversation
WalkthroughThe pull request introduces platform-specific memory loading checks for the Changes
Assessment against linked issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (1)
⏰ Context from checks skipped due to timeout of 90000ms (9)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
|
🔊@mxsm 🚀Thanks for your contribution🎉！ 💡CodeRabbit(AI) will review your code first🔥！ Note 🚨The code review suggestions from CodeRabbit are to be used as a reference only, and the PR submitter can decide whether to make changes based on their own judgment. Ultimately, the project management personnel will conduct the final code review💥. |
rocketmq-rust-bot
added
AI review first
rocketmq-rust-bot
requested review from
rocketmq-rust-bot,
SpaceXCN and
TeslaRustor
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
rocketmq-store/src/log_file/mapped_file/default_mapped_file_impl.rs (1)
726-774: Improve error handling and add logging.Both implementations should log errors to help with debugging memory-related issues.
Add error logging:
#[cfg(target_os = "linux")] fn is_loaded(&self, position: i64, size: usize) -> bool { use libc::c_void; use libc::mincore; use libc::EINVAL; let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize }; let page_count = (size + page_size - 1) / page_size; let mut vec = vec![0u8; page_count]; let address = self.mmapped_file.as_ptr().wrapping_add(position as usize); let ret = unsafe { mincore(address as *mut c_void, size, vec.as_mut_ptr()) }; if ret == -1 { + error!("mincore failed with errno: {}", unsafe { *libc::__errno_location() }); return false; }#[cfg(target_os = "windows")] fn is_loaded(&self, position: i64, size: usize) -> bool { // ... previous code ... if result == 0 { + error!("VirtualQuery failed with error: {}", std::io::Error::last_os_error()); return false; }🧰 Tools
🪛 GitHub Actions: CI
[error] 728-730: Missing dependency: The 'libc' crate is not included in dependencies but is being used in imports
[error] 732-732: Undefined variable: Cannot find value 'length' in scope
[error] 735-735: Undefined variables: Cannot find values 'address' and 'length' in scope
[error] 731-731: Unresolved module: Failed to resolve 'libc' crate in sysconf call
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (2)
rocketmq-store/Cargo.toml(1 hunks)rocketmq-store/src/log_file/mapped_file/default_mapped_file_impl.rs(1 hunks)
🧰 Additional context used
🪛 GitHub Actions: CI
rocketmq-store/src/log_file/mapped_file/default_mapped_file_impl.rs
[error] 728-730: Missing dependency: The 'libc' crate is not included in dependencies but is being used in imports
[error] 732-732: Undefined variable: Cannot find value 'length' in scope
[error] 735-735: Undefined variables: Cannot find values 'address' and 'length' in scope
[error] 731-731: Unresolved module: Failed to resolve 'libc' crate in sysconf call
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: test
- GitHub Check: auto-approve
🔇 Additional comments (1)
rocketmq-store/Cargo.toml (1)
52-56: LGTM! Dependencies are correctly configured.The target-specific dependencies are properly defined with appropriate versions:
- Linux: libc crate for memory management
- Windows: windows crate with necessary memory management features
| #[cfg(target_os = "linux")] | ||
| fn is_loaded(&self, position: i64, size: usize) -> bool { | ||
| use libc::c_void; | ||
| use libc::mincore; | ||
| use libc::EINVAL; | ||
| let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize }; | ||
| let page_count = (length + page_size - 1) / page_size; | ||
|
|
||
| let mut vec = vec![0u8; page_count]; | ||
| let ret = unsafe { mincore(address as *mut c_void, length, vec.as_mut_ptr()) }; | ||
|
|
||
| if ret == -1 { | ||
| return false; | ||
| } | ||
|
|
||
| !vec.iter().any(|&byte| byte & 1 == 0) | ||
| } | ||
|
|
||
| #[inline] | ||
| #[cfg(target_os = "windows")] | ||
| fn is_loaded(&self, position: i64, size: usize) -> bool { | ||
| /*use windows::Win32::Foundation::{BOOL, HANDLE}; | ||
| use windows::Win32::System::Memory::{VirtualQuery, MEMORY_BASIC_INFORMATION, MEM_COMMIT}; | ||
|
|
||
| let address = self.mmapped_file.as_ptr().wrapping_add(position as usize); | ||
| let mut info: MEMORY_BASIC_INFORMATION = unsafe { std::mem::zeroed() }; | ||
| let mut offset = 0; | ||
|
|
||
| while offset < length { | ||
| let result = unsafe { | ||
| VirtualQuery( | ||
| address.add(offset) as *const _, | ||
| &mut info, | ||
| std::mem::size_of::<MEMORY_BASIC_INFORMATION>(), | ||
| ) | ||
| }; | ||
|
|
||
| if result == 0 { | ||
| return Err(std::io::Error::last_os_error()); | ||
| } | ||
|
|
||
| if info.State != MEM_COMMIT { | ||
| return Ok(false); | ||
| } | ||
|
|
||
| offset += info.RegionSize; | ||
| }*/ | ||
|
|
||
| true |
There was a problem hiding this comment.
Add bounds checking for memory safety.
The implementations need bounds checking to prevent buffer overflows and ensure memory safety.
Add bounds checking at the start of both implementations:
#[cfg(target_os = "linux")]
fn is_loaded(&self, position: i64, size: usize) -> bool {
+ if position < 0 || position as usize + size > self.file_size as usize {
+ error!("Invalid position or size: position={}, size={}, file_size={}",
+ position, size, self.file_size);
+ return false;
+ }
// ... rest of the implementation
}
#[cfg(target_os = "windows")]
fn is_loaded(&self, position: i64, size: usize) -> bool {
+ if position < 0 || position as usize + size > self.file_size as usize {
+ error!("Invalid position or size: position={}, size={}, file_size={}",
+ position, size, self.file_size);
+ return false;
+ }
// ... rest of the implementation
}Committable suggestion skipped: line range outside the PR's diff.
🧰 Tools
🪛 GitHub Actions: CI
[error] 728-730: Missing dependency: The 'libc' crate is not included in dependencies but is being used in imports
[error] 732-732: Undefined variable: Cannot find value 'length' in scope
[error] 735-735: Undefined variables: Cannot find values 'address' and 'length' in scope
[error] 731-731: Unresolved module: Failed to resolve 'libc' crate in sysconf call
| #[cfg(target_os = "linux")] | ||
| fn is_loaded(&self, position: i64, size: usize) -> bool { | ||
| use libc::c_void; | ||
| use libc::mincore; | ||
| use libc::EINVAL; | ||
| let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize }; | ||
| let page_count = (length + page_size - 1) / page_size; | ||
|
|
||
| let mut vec = vec![0u8; page_count]; | ||
| let ret = unsafe { mincore(address as *mut c_void, length, vec.as_mut_ptr()) }; | ||
|
|
||
| if ret == -1 { | ||
| return false; | ||
| } | ||
|
|
||
| !vec.iter().any(|&byte| byte & 1 == 0) | ||
| } |
There was a problem hiding this comment.
Fix critical issues in Linux implementation of is_loaded.
The implementation has several issues:
- Unused parameters: position and size
- Undefined variables: length and address
- Incorrect memory page calculation
Apply this diff to fix the implementation:
#[cfg(target_os = "linux")]
fn is_loaded(&self, position: i64, size: usize) -> bool {
use libc::c_void;
use libc::mincore;
use libc::EINVAL;
let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize };
- let page_count = (length + page_size - 1) / page_size;
+ let page_count = (size + page_size - 1) / page_size;
let mut vec = vec![0u8; page_count];
- let ret = unsafe { mincore(address as *mut c_void, length, vec.as_mut_ptr()) };
+ let address = self.mmapped_file.as_ptr().wrapping_add(position as usize);
+ let ret = unsafe { mincore(address as *mut c_void, size, vec.as_mut_ptr()) };
if ret == -1 {
return false;
}
!vec.iter().any(|&byte| byte & 1 == 0)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| #[cfg(target_os = "linux")] | |
| fn is_loaded(&self, position: i64, size: usize) -> bool { | |
| use libc::c_void; | |
| use libc::mincore; | |
| use libc::EINVAL; | |
| let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize }; | |
| let page_count = (length + page_size - 1) / page_size; | |
| let mut vec = vec![0u8; page_count]; | |
| let ret = unsafe { mincore(address as *mut c_void, length, vec.as_mut_ptr()) }; | |
| if ret == -1 { | |
| return false; | |
| } | |
| !vec.iter().any(|&byte| byte & 1 == 0) | |
| } | |
| #[cfg(target_os = "linux")] | |
| fn is_loaded(&self, position: i64, size: usize) -> bool { | |
| use libc::c_void; | |
| use libc::mincore; | |
| use libc::EINVAL; | |
| let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) as usize }; | |
| let page_count = (size + page_size - 1) / page_size; | |
| let mut vec = vec![0u8; page_count]; | |
| let address = self.mmapped_file.as_ptr().wrapping_add(position as usize); | |
| let ret = unsafe { mincore(address as *mut c_void, size, vec.as_mut_ptr()) }; | |
| if ret == -1 { | |
| return false; | |
| } | |
| !vec.iter().any(|&byte| byte & 1 == 0) | |
| } |
🧰 Tools
🪛 GitHub Actions: CI
[error] 728-730: Missing dependency: The 'libc' crate is not included in dependencies but is being used in imports
[error] 732-732: Undefined variable: Cannot find value 'length' in scope
[error] 735-735: Undefined variables: Cannot find values 'address' and 'length' in scope
[error] 731-731: Unresolved module: Failed to resolve 'libc' crate in sysconf call
| #[cfg(target_os = "windows")] | ||
| fn is_loaded(&self, position: i64, size: usize) -> bool { | ||
| /*use windows::Win32::Foundation::{BOOL, HANDLE}; | ||
| use windows::Win32::System::Memory::{VirtualQuery, MEMORY_BASIC_INFORMATION, MEM_COMMIT}; | ||
|
|
||
| let address = self.mmapped_file.as_ptr().wrapping_add(position as usize); | ||
| let mut info: MEMORY_BASIC_INFORMATION = unsafe { std::mem::zeroed() }; | ||
| let mut offset = 0; | ||
|
|
||
| while offset < length { | ||
| let result = unsafe { | ||
| VirtualQuery( | ||
| address.add(offset) as *const _, | ||
| &mut info, | ||
| std::mem::size_of::<MEMORY_BASIC_INFORMATION>(), | ||
| ) | ||
| }; | ||
|
|
||
| if result == 0 { | ||
| return Err(std::io::Error::last_os_error()); | ||
| } | ||
|
|
||
| if info.State != MEM_COMMIT { | ||
| return Ok(false); | ||
| } | ||
|
|
||
| offset += info.RegionSize; | ||
| }*/ | ||
|
|
||
| true |
There was a problem hiding this comment.
Fix Windows implementation returning incorrect result.
The current implementation always returns true without performing any memory checks, which could lead to incorrect behavior.
Uncomment and fix the Windows implementation:
#[cfg(target_os = "windows")]
fn is_loaded(&self, position: i64, size: usize) -> bool {
- /*use windows::Win32::Foundation::{BOOL, HANDLE};
- use windows::Win32::System::Memory::{VirtualQuery, MEMORY_BASIC_INFORMATION, MEM_COMMIT};
-
- let address = self.mmapped_file.as_ptr().wrapping_add(position as usize);
- let mut info: MEMORY_BASIC_INFORMATION = unsafe { std::mem::zeroed() };
- let mut offset = 0;
-
- while offset < length {
- let result = unsafe {
- VirtualQuery(
- address.add(offset) as *const _,
- &mut info,
- std::mem::size_of::<MEMORY_BASIC_INFORMATION>(),
- )
- };
-
- if result == 0 {
- return Err(std::io::Error::last_os_error());
- }
-
- if info.State != MEM_COMMIT {
- return Ok(false);
- }
-
- offset += info.RegionSize;
- }*/
-
- true
+ use windows::Win32::Foundation::BOOL;
+ use windows::Win32::System::Memory::{VirtualQuery, MEMORY_BASIC_INFORMATION, MEM_COMMIT};
+
+ let address = self.mmapped_file.as_ptr().wrapping_add(position as usize);
+ let mut info: MEMORY_BASIC_INFORMATION = unsafe { std::mem::zeroed() };
+ let mut offset = 0;
+
+ while offset < size {
+ let result = unsafe {
+ VirtualQuery(
+ address.wrapping_add(offset) as *const _,
+ &mut info,
+ std::mem::size_of::<MEMORY_BASIC_INFORMATION>(),
+ )
+ };
+
+ if result == 0 {
+ return false;
+ }
+
+ if info.State != MEM_COMMIT {
+ return false;
+ }
+
+ offset += info.RegionSize;
+ }
+
+ true
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| #[cfg(target_os = "windows")] | |
| fn is_loaded(&self, position: i64, size: usize) -> bool { | |
| /*use windows::Win32::Foundation::{BOOL, HANDLE}; | |
| use windows::Win32::System::Memory::{VirtualQuery, MEMORY_BASIC_INFORMATION, MEM_COMMIT}; | |
| let address = self.mmapped_file.as_ptr().wrapping_add(position as usize); | |
| let mut info: MEMORY_BASIC_INFORMATION = unsafe { std::mem::zeroed() }; | |
| let mut offset = 0; | |
| while offset < length { | |
| let result = unsafe { | |
| VirtualQuery( | |
| address.add(offset) as *const _, | |
| &mut info, | |
| std::mem::size_of::<MEMORY_BASIC_INFORMATION>(), | |
| ) | |
| }; | |
| if result == 0 { | |
| return Err(std::io::Error::last_os_error()); | |
| } | |
| if info.State != MEM_COMMIT { | |
| return Ok(false); | |
| } | |
| offset += info.RegionSize; | |
| }*/ | |
| true | |
| #[cfg(target_os = "windows")] | |
| fn is_loaded(&self, position: i64, size: usize) -> bool { | |
| use windows::Win32::Foundation::BOOL; | |
| use windows::Win32::System::Memory::{VirtualQuery, MEMORY_BASIC_INFORMATION, MEM_COMMIT}; | |
| let address = self.mmapped_file.as_ptr().wrapping_add(position as usize); | |
| let mut info: MEMORY_BASIC_INFORMATION = unsafe { std::mem::zeroed() }; | |
| let mut offset = 0; | |
| while offset < size { | |
| let result = unsafe { | |
| VirtualQuery( | |
| address.wrapping_add(offset) as *const _, | |
| &mut info, | |
| std::mem::size_of::<MEMORY_BASIC_INFORMATION>(), | |
| ) | |
| }; | |
| if result == 0 { | |
| return false; | |
| } | |
| if info.State != MEM_COMMIT { | |
| return false; | |
| } | |
| offset += info.RegionSize; | |
| } | |
| true | |
| } |
|
🔊@mxsm 🚀Thanks for your contribution🎉！ 💡CodeRabbit(AI) will review your code first🔥！ Note 🚨The code review suggestions from CodeRabbit are to be used as a reference only, and the PR submitter can decide whether to make changes based on their own judgment. Ultimately, the project management personnel will conduct the final code review💥. |
rocketmq-rust-bot
added
approved
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2311 +/- ##
==========================================
- Coverage 28.21% 28.20% -0.01%
==========================================
Files 504 504
Lines 72456 72470 +14
==========================================
Hits 20440 20440
- Misses 52016 52030 +14 ☔ View full report in Codecov by Sentry. |
Which Issue(s) This PR Fixes(Closes)
Fixes #2310
Brief Description
How Did You Test This Change?
Summary by CodeRabbit
New Features
Chores