Skip to content

Replace windows to windows-sys #6356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
youknowone merged 3 commits into from
Dec 8, 2025
Merged

Replace windows to windows-sys #6356

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
91d347d
Bump windows from 0.52.0 to 0.62.2
dependabot[bot] Oct 12, 2025
338170a
Update windows
youknowone Dec 8, 2025
52aa978
additional fix
youknowone Dec 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
additional fix
  • Loading branch information
@youknowone
youknowone committed Dec 8, 2025
commit 52aa978dba14d18b7df1a79f6291fcb1bb6bf05e
33 changes: 24 additions & 9 deletions crates/vm/src/stdlib/winapi.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,14 +291,23 @@ mod _winapi {
}

#[pyfunction]
fn OpenProcess(desired_access: u32, inherit_handle: bool, process_id: u32) -> isize {
unsafe {
fn OpenProcess(
desired_access: u32,
inherit_handle: bool,
process_id: u32,
vm: &VirtualMachine,
) -> PyResult<WinHandle> {
let handle = unsafe {
windows_sys::Win32::System::Threading::OpenProcess(
desired_access,
i32::from(inherit_handle),
process_id,
) as _
)
};
if handle.is_null() {
return Err(errno_err(vm));
}
Ok(WinHandle(handle))
}

#[pyfunction]
Expand Down Expand Up @@ -432,7 +441,7 @@ mod _winapi {
0,
(2 & 0xffff) | 0x20000, // PROC_THREAD_ATTRIBUTE_HANDLE_LIST
handlelist.as_mut_ptr() as _,
(handlelist.len() * std::mem::size_of::<isize>()) as _,
(handlelist.len() * std::mem::size_of::<usize>()) as _,
std::ptr::null_mut(),
std::ptr::null(),
)
Expand Down Expand Up @@ -506,17 +515,23 @@ mod _winapi {
}

#[pyfunction]
fn OpenMutexW(desired_access: u32, inherit_handle: bool, name: u16) -> PyResult<isize> {
fn OpenMutexW(
desired_access: u32,
inherit_handle: bool,
name: PyStrRef,
vm: &VirtualMachine,
) -> PyResult<isize> {
let name_wide = name.as_str().to_wide_with_nul();
let handle = unsafe {
windows_sys::Win32::System::Threading::OpenMutexW(
desired_access,
i32::from(inherit_handle),
windows_sys::core::PCWSTR::from(name as _),
name_wide.as_ptr(),
)
};
// if handle.is_invalid() {
// return Err(errno_err(vm));
// }
if handle == INVALID_HANDLE_VALUE {
return Err(errno_err(vm));
}
Ok(handle as _)
}
Comment on lines +518 to 536
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

Win32 OpenMutexW API return value failure NULL INVALID_HANDLE_VALUE

💡 Result:

Short answer: OpenMutexW returns NULL on failure — it does NOT return INVALID_HANDLE_VALUE. On success it returns a valid HANDLE; on failure check GetLastError() for the reason (e.g., ERROR_FILE_NOT_FOUND if the named mutex doesn't exist, access denied if security blocks the requested access). [1]

Common causes when OpenMutexW returns NULL

  • The named mutex doesn't exist. GetLastError() == ERROR_FILE_NOT_FOUND (2). [1]
  • You asked for access that the mutex's security descriptor disallows. [1]
  • Name mismatch (case sensitive) or wrong namespace (use "Global\name" vs "Local\name" appropriately across sessions). [1]
  • Another object with the same name but different type exists (CreateMutex/OpenMutex can fail with ERROR_INVALID_HANDLE when a non-mutex object already uses that name). [3]
  • Unicode/ANSI mismatch or incorrect lpName string (OpenMutexW expects LPCWSTR). [3]
  • Race conditions between create/open across processes. [1][3]

What to do

  • Check the return against NULL, not INVALID_HANDLE_VALUE. Then call GetLastError() to diagnose. [1][2]
  • Ensure you pass a wide string to OpenMutexW (L"Name" or use the generic OpenMutex with proper TCHAR usage).
  • Verify namespace and spelling; try tools like WinObj to inspect kernel object names if needed.
  • If you need create-or-open semantics, use CreateMutex (it opens if exists, creates if not) and check GetLastError() for ERROR_ALREADY_EXISTS. [1][3]

References

  • Microsoft Docs: OpenMutexW (return value and errors). [1]
  • Discussion: checking handles vs INVALID_HANDLE_VALUE (use 0/NULL for many objects). [2]
  • Community notes on CreateMutex/OpenMutex failures (name collisions, Unicode issues). [3]

OpenMutexW failure check uses the wrong sentinel, causing silent errors

The current implementation checks:

if handle == INVALID_HANDLE_VALUE {
    return Err(errno_err(vm));
}

However, OpenMutexW returns NULL on failure, not INVALID_HANDLE_VALUE. With this code:

  • A failing OpenMutexW call will return NULL, which passes the check
  • Such failures will slip through as Ok(0isize), and the error will only surface later when using the bogus handle

This is a correctness bug and should be fixed to treat NULL as failure, similar to OpenProcess:

     let handle = unsafe {
         windows_sys::Win32::System::Threading::OpenMutexW(
             desired_access,
             i32::from(inherit_handle),
             name_wide.as_ptr(),
         )
     };
-    if handle == INVALID_HANDLE_VALUE {
-        return Err(errno_err(vm));
-    }
-    Ok(handle as _)
+    if handle.is_null() {
+        return Err(errno_err(vm));
+    }
+    Ok(handle as isize)
🤖 Prompt for AI Agents 
In crates/vm/src/stdlib/winapi.rs around lines 518 to 536, the OpenMutexW
failure check is using INVALID_HANDLE_VALUE but OpenMutexW returns NULL (0) on
failure; change the check to test for a zero handle (if handle == 0) and return
Err(errno_err(vm)) in that case, keeping the existing cast/Ok(handle as _) for
the success path.


Expand Down
Loading