Skip to content

Updated windows advanced addon to replace ai Request with ocr endpoints.#364

Open
ManojTestsigma wants to merge 2 commits intodevfrom
windows-advanced-actions
Open

Updated windows advanced addon to replace ai Request with ocr endpoints.#364
ManojTestsigma wants to merge 2 commits intodevfrom
windows-advanced-actions

Conversation

@ManojTestsigma
Copy link
Copy Markdown
Contributor

@ManojTestsigma ManojTestsigma commented Mar 25, 2026

please review this addon and publish as PUBLIC

Addon name : windows_advanced_actions
Addon accont: https://jarvis.testsigma.com/ui/tenants/2817/addons

updated windows advanced addon with OCR endpoints Instead Of Ai request

there are total 30 NLP's ( 1 action use Ai , Other actions use Robots and staging endpoints.

Summary by CodeRabbit

  • New Features
    • Added Windows automation actions for clicking on images, text, and screen coordinates
    • Introduced OCR-based text detection and verification capabilities
    • Enhanced keyboard input automation with character-level typing and modifier key combinations
    • Added file and clipboard data manipulation operations
    • Added image and text presence detection with wait timeouts and relative positioning options

@ManojTestsigma ManojTestsigma requested review from rajesh-testsigma and removed request for rajesh-testsigma March 25, 2026 12:02
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Mar 25, 2026

📝 Walkthrough

Walkthrough

This pull request significantly extends the Windows Advanced and Windows Lite addons by reorganizing utility classes, introducing OCR-based text detection, and adding dozens of new action implementations for keyboard input, image/text clicking, data verification, and file operations. The artifact version is downgraded from 1.0.23 to 1.0.1.

Changes

Cohort / File(s) Summary
Version & Configuration
windows_advanced_actions/pom.xml
Maven artifact version downgraded from 1.0.23 to 1.0.1.
Utility Classes - Package Reorganization & New Utils
windows_advanced_actions/src/main/java/com/testsigma/addons/util/Constants.java, KeyboardUtils.java, ScreenshotUtils.java, StringCompareUtil.java
Migrated and enhanced utilities from windowsAdvanced.utils to util package. Added Constants class with Visual Server endpoint URLs and API token. Introduced typeCharacter() method in KeyboardUtils for US keyboard character input with special key handling. Refactored ScreenshotUtils with 1-second delay before screenshot capture and disabled several public helper methods. Added StringCompareUtil for string comparison operations (equals, contains, case-sensitive/insensitive variants).
OCR Data Models & Integration
windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRResponse.java, OCRTextPoint.java, OCRUtils.java, ResponseObjectForFindImage.java
Introduced OCR response models: OCRResponse (error + text points), OCRTextPoint (text with bounding box + center calculations). Implemented OCRUtils with text extraction via HTTP multipart POST to visual server, multi-pass text matching (exact → case-insensitive → substring), sentence-level matching with word boundary logic, and camelCase fallback. Added ResponseObjectForFindImage model for image detection results.
Windows Advanced - File & Clipboard Operations
windowsAdvanced/AddDataToClipboard.java, AddDataToFile.java, CopyAndPasteTestDataOnScreen.java, EraseDataInFile.java
Added four new actions: AddDataToClipboard (copy to system clipboard with verification), AddDataToFile (append/create file with directory creation), CopyAndPasteTestDataOnScreen (paste via Ctrl+V with clipboard restoration), EraseDataInFile (truncate file to zero length with validation).
Windows Advanced - Coordinate & Image Clicking
windowsAdvanced/ClickOnFixedCoordinates.java, ClickOnRelativeCoordinates.java, ClickOnImage.java, ClickOnImageWithFindImage.java, ClickOnPositionRelativeToImage.java
Added coordinate-based and image-based clicking actions. ClickOnFixedCoordinates and ClickOnRelativeCoordinates validate and click at screen coordinates. ClickOnImage, ClickOnImageWithFindImage, and ClickOnPositionRelativeToImage perform HTTP visual server API calls to locate target images and compute click points based on position type (Left/Right/Top/Bottom/Center) with pixel offsets.
Windows Advanced - Text-Based Interactions
windowsAdvanced/ClickOnTextWithWait.java, ClickOnTextWithSpacesAndWait.java, ClickOnPositionRelativeToText.java
Replaced AI-based text detection with OCR in ClickOnTextWithWait. Added ClickOnTextWithSpacesAndWait for sentence-level OCR matching with timeout polling. Introduced ClickOnPositionRelativeToText for position-relative clicks on OCR-detected text with configurable wait timeout.
Windows Advanced - Keyboard Input
windowsAdvanced/EnterDataUsingKeyboard.java, EnterDataUsingKeyboardAndPressEnter.java, PerformMultipleActions.java
Updated EnterDataUsingKeyboard to use character-by-character KeyboardUtils.typeCharacter() instead of clipboard-based input. Added EnterDataUsingKeyboardAndPressEnter for keyboard entry followed by Enter key press. Introduced PerformMultipleActions supporting complex action sequences: {WAIT(n)} for delays, {KEY(...)} for character typing, and special key names (TAB, ENTER, arrows, function keys).
Windows Advanced - Key Modifiers
windowsAdvanced/PressFunctionKey.java, PressModifierKey.java, PressModifierKeyWithBasicKeyCombination.java, PressModifierKeyWithGivenKey.java, PressTwoModifierKeys.java, PressTwoModifierKeysWithBasicKey.java
Updated six key-pressing actions to use consolidated util package utilities (ScreenshotUtils, KeyboardUtils). Updated action display names to user-friendly format (e.g., "PressFunctionKey" → "Press a Function Key"). No logic changes; utility imports consolidated.
Windows Advanced - Text Verification & Extraction
windowsAdvanced/VerifyTextInApplication.java, VerifyTextInApplicationWithAI.java, StorePresenceOfTextInRuntimeVariable.java, ExtractDataFromScreen.java, StringCompareWindowsAdvanced.java
Updated VerifyTextInApplication from AI to OCR-based detection with runtime variable storage. Added VerifyTextInApplicationWithAI (OCR-based fallback), StorePresenceOfTextInRuntimeVariable (OCR with runtime variable), ExtractDataFromScreen (AI-based data extraction from screenshots), StringCompareWindowsAdvanced (string comparison via StringCompareUtil).
Windows Advanced - Image & Text Waiting
windowsAdvanced/WaitUntilImagePresent.java, WaitUntilImagePresentWithFindImage.java, WaitUntilTextPresentInScreen.java
WaitUntilImagePresent receives minor formatting update. Added WaitUntilImagePresentWithFindImage with visual server API polling for image detection with threshold validation. Updated WaitUntilTextPresentInScreen from AI to OCR-based polling with configurable timeout and interval.
Windows Lite - Image & Text Clicking
windowsLite/ClickOnImage.java, ClickOnPositionRelativeToImage.java, ClickOnPositionRelativeToText.java, ClickOnTextWithWait.java, RightClick.java
Added five Windows Lite actions using injected @OCR component for image/text detection. ClickOnImage, ClickOnPositionRelativeToImage, ClickOnPositionRelativeToText locate visual elements via OCR; ClickOnTextWithWait polls OCR until timeout; RightClick performs right-click at fixed coordinates.
Windows Lite - Keyboard Input
windowsLite/CopyAndPasteTestDataOnScreen.java, EnterDataUsingKeyboard.java, EnterDataUsingKeyboardAndPressEnter.java, PressFunctionKey.java, PressModifierKey.java, PressModifierKeyWithBasicKeyCombination.java, PressModifierKeyWithGivenKey.java, PressTwoModifierKeys.java, PressTwoModifierKeysWithBasicKey.java
Added nine Windows Lite keyboard actions mirroring Windows Advanced functionality: clipboard paste, character-by-character keyboard input with Enter, function key pressing, modifier key combinations. Uses KeyboardUtils.typeCharacter() and utility helpers.
Windows Lite - Text Verification & Waiting
windowsLite/StorePresenceOfTextInRuntimeVariable.java, VerifyTextInApplication.java, WaitUntilImagePresent.java, WaitUntilTextPresentInScreen.java
Added four Windows Lite actions: StorePresenceOfTextInRuntimeVariable (AI-based text detection with runtime storage), VerifyTextInApplication (AI-based verification), WaitUntilImagePresent (OCR-based image polling via injected component), WaitUntilTextPresentInScreen (AI-based text polling).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • vigneshtestsigma
  • Ganesh-Testsigma

Poem

🐰 From utils scattered wide, we gather near,
OCR eyes see text so crystal clear,
Keyboards dance with each keystroke and key,
New actions bloom on every tree! 🌳✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 51.16% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title clearly and specifically describes the main change: replacing AI request functionality with OCR endpoints in the Windows Advanced addon.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch windows-advanced-actions

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Note

Due to the large number of review comments, Critical severity comments were prioritized as inline comments.

🟠 Major comments (28)
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/AddDataToFile.java-43-55 (1)

43-55: ⚠️ Potential issue | 🟠 Major

Null check occurs after potential NPE.

If getValue() returns null, calling .toString() on lines 43-44 will throw NullPointerException before the null check on line 50 is reached. Both fileContents and filePath values should be validated for null before conversion.

🐛 Proposed fix with proper null validation
     try {
-        String content = fileContents.getValue().toString();
-        String path = filePath.getValue().toString();
-        
-        logger.info("Adding data to file: " + path);
-        logger.info("Data to add: " + content);
-        
-        // Validate file path
-        if (path == null || path.trim().isEmpty()) {
+        Object contentValue = fileContents.getValue();
+        Object pathValue = filePath.getValue();
+
+        // Validate inputs before conversion
+        if (pathValue == null || pathValue.toString().trim().isEmpty()) {
             setErrorMessage("File path cannot be empty");
             ScreenshotUtils.captureAndUploadScreenshot(testStepResult, 
                     "add_data_to_file_failure_screenshot", logger);
             return Result.FAILED;
         }
+
+        String path = pathValue.toString();
+        String content = contentValue != null ? contentValue.toString() : "";
+
+        logger.info("Adding data to file: " + path);
+        logger.info("Data to add: " + content);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/AddDataToFile.java`
around lines 43 - 55, The code calls fileContents.getValue().toString() and
filePath.getValue().toString() before validating for null; update AddDataToFile
so you first validate fileContents and filePath and their getValue() results for
null/empty and only then call toString() on them. Specifically, check
fileContents == null or fileContents.getValue() == null (and similarly for
filePath) and on failure call setErrorMessage(...) and
ScreenshotUtils.captureAndUploadScreenshot(testStepResult,
"add_data_to_file_failure_screenshot", logger) and return Result.FAILED; only
after these checks convert values to String and proceed, keeping
logger.info(...) and other logic unchanged.
windows_advanced_actions/src/main/java/com/testsigma/addons/util/StringCompareUtil.java-10-54 (1)

10-54: ⚠️ Potential issue | 🟠 Major

Null pointer exception risk if either input string is null.

If string1 or string2 is null, calling .equals(), .equalsIgnoreCase(), .contains(), or .toLowerCase() will throw a NullPointerException. Add null checks at the start of the method.

🛡️ Proposed fix to add null validation
     public boolean performOperation(String string1, String string2, String operation){
+        if (string1 == null || string2 == null) {
+            this.errorMessage = "Input strings cannot be null. string1: " + string1 + ", string2: " + string2;
+            return false;
+        }
         boolean equalsCheck = false;
         boolean containsCheck = false;
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/util/StringCompareUtil.java`
around lines 10 - 54, performOperation risks NullPointerException when string1
or string2 is null; add early null validation in performOperation to check both
inputs before the switch, set this.errorMessage (e.g., "Input string1 is null"
or "Input string2 is null" or a combined message) and return false if either is
null so no .equals/.contains/.toLowerCase calls run on null; update handling for
all branches (equals, equals ignore-case, contains, contains ignore-case) to
rely on the validated non-null inputs and preserve setting
this.successMessage/this.errorMessage and the boolean result logic.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ExtractDataFromScreen.java-125-141 (1)

125-141: ⚠️ Potential issue | 🟠 Major

Potential NullPointerException when JSON fields are missing.

JsonNode.get() returns null when the field doesn't exist. Calling asBoolean() or asText() on a null reference will throw NullPointerException. Use path() instead, which returns a MissingNode that safely handles default values.

🐛 Proposed fix using path()
     private AIResponse parseAIResponse(String aiResponse) {
         try {
             ObjectMapper mapper = new ObjectMapper();
             JsonNode jsonNode = mapper.readTree(aiResponse);

             AIResponse response = new AIResponse();
-            response.isQueryRelated = jsonNode.get("isQueryRelated").asBoolean();
-            response.extractedText = jsonNode.get("extracted Text").asText("");
-            response.additionalData = jsonNode.get("additional data").asText("");
+            response.isQueryRelated = jsonNode.path("isQueryRelated").asBoolean(false);
+            response.extractedText = jsonNode.path("extracted Text").asText("");
+            response.additionalData = jsonNode.path("additional data").asText("");

             return response;
         } catch (Exception e) {
             logger.warn("Failed to parse AI response JSON: " + e.getMessage());
             logger.warn("Raw AI response: " + aiResponse);
             return null;
         }
     }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ExtractDataFromScreen.java`
around lines 125 - 141, The parseAIResponse method can NPE when fields are
absent because jsonNode.get(...) may return null; change jsonNode.get(...) calls
to jsonNode.path(...) in parseAIResponse so MissingNode handles defaults (e.g.,
response.isQueryRelated = jsonNode.path("isQueryRelated").asBoolean();
response.extractedText = jsonNode.path("extracted Text").asText("");
response.additionalData = jsonNode.path("additional data").asText("");), keep
the same AIResponse field assignments and existing exception handling/logging
with logger unchanged.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnImage.java-58-61 (1)

58-61: ⚠️ Potential issue | 🟠 Major

Don’t log the screenshot upload URL.

testStepResult.getScreenshotUrl() is typically a presigned upload URL. Writing it at info level exposes temporary upload credentials in the step logs.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnImage.java`
around lines 58 - 61, The code logs a presigned upload URL
(testStepResult.getScreenshotUrl()) which exposes temporary credentials; in
ClickOnImage (where logger.info is called around
testStepResult.getScreenshotUrl() and before ocr.uploadFile) remove or redact
those logger.info calls so the full URL is not written to logs (either delete
the two logger.info lines that reference getScreenshotUrl() or replace them with
a non-sensitive message like "Uploading screenshot to storage" without including
the URL); ensure no other logging in the ClickOnImage flow outputs
testStepResult.getScreenshotUrl().
windows_advanced_actions/src/main/java/com/testsigma/addons/util/KeyboardUtils.java-60-100 (1)

60-100: ⚠️ Potential issue | 🟠 Major

Handle unmappable characters and supplementary code points before calling Robot.

This method accepts characters via Character.isLetter() that cannot be typed on any keyboard layout. Characters like é, à, ñ, and CJK ideographs pass isLetter() but KeyEvent.getExtendedKeyCodeForChar() returns VK_UNDEFINED when they lack a key mapping, leading to undefined Robot behavior. Additionally, supplementary Unicode code points (U+10000 and above) are represented as surrogate pairs in Java's char type; passing them one character at a time causes each surrogate to fail validation and throw IllegalArgumentException mid-input, leaving Robot in an inconsistent state.

Add early validation to reject unmappable characters or accept only BMP code points. For full Unicode support, accept String input and handle surrogate pairs correctly using Character.toCodePoint() and Character.codePointAt().

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/util/KeyboardUtils.java`
around lines 60 - 100, typeCharacter(Robot robot, char character) currently
assumes every Character.isLetter/isDigit has a valid keycode and that
single-char input covers all Unicode; add early validation to reject unmappable
or surrogate chars before touching the Robot: in typeCharacter check if
Character.isSurrogate(character) (or Character.isHighSurrogate/isLowSurrogate)
and throw IllegalArgumentException for supplementary code points, and after
computing keyCode via KeyEvent.getExtendedKeyCodeForChar(...) ensure keyCode !=
KeyEvent.VK_UNDEFINED and throw before calling robot.keyPress; apply the same
keyCode != VK_UNDEFINED guard for letter, digit and SPECIAL_CHAR_MAP branches
(SPECIAL_CHAR_MAP entries should still be used when present), or alternatively
change the API to accept a String and process code points with
Character.codePointAt/Character.toChars and Character.toCodePoint for full
Unicode support.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/EraseDataInFile.java-39-49 (1)

39-49: ⚠️ Potential issue | 🟠 Major

Same null-safety issue – check getValue() before calling toString().

If filePath.getValue() returns null, line 39 throws an NPE before the validation on line 44 is reached.

🐛 Proposed fix
         try {
-            String path = filePath.getValue().toString();
-            
-            logger.info("Erasing data from file: " + path);
-            
-            // Validate file path
-            if (path == null || path.trim().isEmpty()) {
+            Object value = filePath.getValue();
+            if (value == null || value.toString().trim().isEmpty()) {
                 setErrorMessage("File path cannot be empty");
                 ScreenshotUtils.captureAndUploadScreenshot(testStepResult, 
                         "erase_data_in_file_failure_screenshot", logger);
                 return Result.FAILED;
             }
+            
+            String path = value.toString();
+            logger.info("Erasing data from file: " + path);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/EraseDataInFile.java`
around lines 39 - 49, The code calls filePath.getValue().toString() which can
NPE if getValue() is null; first retrieve the value into a local (e.g., valueObj
= filePath.getValue()), check valueObj for null or empty before calling
toString(), and only then assign path = valueObj.toString(); if null/empty call
setErrorMessage and ScreenshotUtils.captureAndUploadScreenshot(testStepResult,
"erase_data_in_file_failure_screenshot", logger) and return Result.FAILED to
preserve the existing error handling flow.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/AddDataToClipboard.java-37-47 (1)

37-47: ⚠️ Potential issue | 🟠 Major

Null check is ineffective – toString() will throw NPE before reaching it.

On line 37, dataToCopy.getValue().toString() is called. If getValue() returns null, this throws a NullPointerException before the null check on line 42 is reached.

🐛 Proposed fix to check for null before toString()
         try {
-            String data = dataToCopy.getValue().toString();
-            
-            logger.info("Adding data to clipboard: " + data);
-            
-            // Validate data
-            if (data == null) {
+            Object value = dataToCopy.getValue();
+            if (value == null) {
                 setErrorMessage("Data to copy cannot be null");
                 ScreenshotUtils.captureAndUploadScreenshot(testStepResult, 
                         "add_data_to_clipboard_failure_screenshot", logger);
                 return Result.FAILED;
             }
+            
+            String data = value.toString();
+            logger.info("Adding data to clipboard: " + data);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/AddDataToClipboard.java`
around lines 37 - 47, The null check is after calling
dataToCopy.getValue().toString(), which can NPE; fix by checking
dataToCopy.getValue() for null before invoking toString(): first retrieve Object
val = dataToCopy.getValue(), if val is null call setErrorMessage(...) and
ScreenshotUtils.captureAndUploadScreenshot(testStepResult,
"add_data_to_clipboard_failure_screenshot", logger) and return Result.FAILED;
otherwise set String data = val.toString() and proceed with logger.info("Adding
data to clipboard: " + data). Ensure you reference dataToCopy.getValue(), data,
setErrorMessage, ScreenshotUtils.captureAndUploadScreenshot, logger,
testStepResult and Result.FAILED in the updated flow.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnImage.java-249-429 (1)

249-429: 🛠️ Refactor suggestion | 🟠 Major

Remove large commented-out code block.

Lines 249-429 contain a large block of commented-out code (the previous OCR-based implementation). This should be removed to improve code clarity. If you need to reference the old implementation, it can be found in version control history.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnImage.java`
around lines 249 - 429, Remove the large commented-out legacy OCR implementation
inside the ClickOnImage class (the block that contains the old
urlToFileConverter(...) and cleanupFile(...) implementations and the commented
execute/closing braces); delete that entire commented section so only the active
class code (current execute and any real helper methods) remains, and then clean
up any now-unused imports or leftover comment markers to keep ClickOnImage and
its real methods consistent.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnImage.java-174-177 (1)

174-177: ⚠️ Potential issue | 🟠 Major

Format string placeholders without arguments.

The error messages contain format placeholders (%s) but no arguments are provided to String.format(). This will print literal %s in the error message.

Suggested fix
         } catch (IOException e) {
-            logger.info("Exception occurred while performing visual test at %s : %s" + ExceptionUtils.getStackTrace(e));
-            setErrorMessage("Unable to perform visual testing for : <b> %s</b>");
-            throw new RuntimeException("Error occurred while performing visual test at  ");
+            logger.info("Exception occurred while performing visual test: " + ExceptionUtils.getStackTrace(e));
+            setErrorMessage("Unable to perform visual testing");
+            throw new RuntimeException("Error occurred while performing visual test", e);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnImage.java`
around lines 174 - 177, The log and messages in ClickOnImage (catch block around
the IOException) use "%s" placeholders but aren't passing arguments; update the
logger.info, setErrorMessage, and the RuntimeException to use proper
String.format(...) with the intended arguments (e.g., the image path or context
and ExceptionUtils.getStackTrace(e) or include the exception as a parameter), or
remove the "%s" placeholders and concatenate properly; ensure logger.info
includes the exception details and setErrorMessage contains the target
identifier, and throw a RuntimeException with a clear formatted message (use the
same identifiers used in the original lines).
windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRUtils.java-544-561 (1)

544-561: ⚠️ Potential issue | 🟠 Major

OkHttp Response is not closed—potential resource leak.

The Response object from client.newCall(request).execute() must be closed to release the underlying connection. Even when response.body().string() is called, the response should be closed in all code paths.

🐛 Proposed fix using try-with-resources
             logger.info("Making OCR API call");
-            Response response = client.newCall(request).execute();
-
-            if (response.isSuccessful() && response.body() != null) {
-                String responseBody = response.body().string();
-                logger.info("OCR Response body: " + responseBody);
-
-                OCRResponse ocrResponse = mapper.readValue(responseBody, OCRResponse.class);
-
-                if (ocrResponse.hasError()) {
-                    throw new RuntimeException("OCR API returned error: " + ocrResponse.getError());
-                }
-                if (!ocrResponse.hasText()) {
-                    throw new RuntimeException("No text found in the image");
+            try (Response response = client.newCall(request).execute()) {
+                if (response.isSuccessful() && response.body() != null) {
+                    String responseBody = response.body().string();
+                    logger.info("OCR API call successful");
+
+                    OCRResponse ocrResponse = mapper.readValue(responseBody, OCRResponse.class);
+
+                    if (ocrResponse.hasError()) {
+                        throw new RuntimeException("OCR API returned error: " + ocrResponse.getError());
+                    }
+                    if (!ocrResponse.hasText()) {
+                        throw new RuntimeException("No text found in the image");
+                    }
+                    return ocrResponse.getText();
+                } else {
+                    throw new RuntimeException("OCR API call failed with status: " + response.code());
                 }
-                return ocrResponse.getText();
-            } else {
-                throw new RuntimeException("OCR API call failed with status: " + (response.code()));
             }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRUtils.java`
around lines 544 - 561, The Response returned by
client.newCall(request).execute() in OCRUtils (the block using Response response
= client.newCall(request).execute()) is never closed, causing a resource leak;
wrap the execute() call in a try-with-resources (try (Response response =
client.newCall(request).execute()) { ... }) so the Response (and its body) is
always closed on all code paths, keep the existing logic that reads
response.body().string(), handle null body as before, and rethrow the same
RuntimeExceptions inside the try block.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnPositionRelativeToText.java-222-247 (1)

222-247: ⚠️ Potential issue | 🟠 Major

OkHttp Response not closed - potential resource leak.

The Response object from OkHttp must be closed to release the underlying connection. Currently, if the response is successful, the connection may not be properly released.

🐛 Proposed fix using try-with-resources
             logger.info("Making OCR API call");
-            Response response = client.newCall(request).execute();
-
-            if (response.isSuccessful()) {
-                logger.info("OCR API response is successful");
-                if (response.body() != null) {
-                    String responseBody = response.body().string();
+            try (Response response = client.newCall(request).execute()) {
+                if (response.isSuccessful()) {
+                    logger.info("OCR API response is successful");
+                    if (response.body() != null) {
+                        String responseBody = response.body().string();
                     // ... rest of processing
+                    } else {
+                        throw new RuntimeException("OCR API returned null response body");
+                    }
+                } else {
+                    throw new RuntimeException("OCR API call failed with status: " + response.code());
+                }
+            }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnPositionRelativeToText.java`
around lines 222 - 247, The Response from OkHttp (variable Response response
returned by client.newCall(request).execute() in ClickOnPositionRelativeToText)
is not closed and can leak connections; wrap the execute call in a
try-with-resources (e.g., try (Response response =
client.newCall(request).execute()) { ... }) or explicitly close the
response/body after reading, then read response.body().string() inside that
scope, deserialize with mapper into ocrResponse, and keep existing error/status
handling unchanged so the response is always closed even on exceptions.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnPositionRelativeToText.java-52-53 (1)

52-53: 🛠️ Refactor suggestion | 🟠 Major

Instance fields should be local variables.

ObjectMapper and OCRResponse are created as instance fields but used only within execute() and extractTextPoints(). The ocrResponse field is particularly problematic as it gets overwritten on each OCR call, which could cause issues if this action is reused.

♻️ Suggested refactor
-    ObjectMapper mapper = new ObjectMapper();
-    OCRResponse ocrResponse = new OCRResponse();
+    private static final ObjectMapper mapper = new ObjectMapper();

And make ocrResponse a local variable inside extractTextPoints().

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnPositionRelativeToText.java`
around lines 52 - 53, ObjectMapper and OCRResponse are declared as instance
fields but only used inside execute() and extractTextPoints(), and ocrResponse
is overwritten on each call; move their declarations into the methods that use
them: create a local ObjectMapper inside execute() (or inside
extractTextPoints() if only used there) and make ocrResponse a local variable
inside extractTextPoints(), updating references in
ClickOnPositionRelativeToText.execute() and
ClickOnPositionRelativeToText.extractTextPoints() accordingly so no shared
mutable state remains.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnTextWithWait.java-67-71 (1)

67-71: ⚠️ Potential issue | 🟠 Major

Handle the same null OCR case before logging it.

getTextPointFromText() already treats null as a normal "not found yet" case, but printAllCoordinates(textPoints) runs first and will throw on that same input. That turns a normal poll miss into an immediate failure.

Also applies to: 137-153

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnTextWithWait.java`
around lines 67 - 71, printAllCoordinates(textPoints) can NPE when OCR returns
null, turning a normal "not found yet" poll into a hard failure; change the
sequence in ClickOnTextWithWait so you call getTextPointFromText(textPoints) (or
check textPoints for null/empty) before calling printAllCoordinates, and only
invoke printAllCoordinates when textPoints is non-null and non-empty; apply the
same defensive null/empty check-update to the other similar block that uses
extractTextFromImage/getTextPointFromText/printAllCoordinates so both polling
paths handle a null OCR result gracefully.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java-70-74 (1)

70-74: ⚠️ Potential issue | 🟠 Major

Every poll leaks a temporary screenshot.

screenshotFile is created on each iteration, but there is no cleanup on miss, success, or exceptions. Longer waits will keep leaving PNGs behind in the temp directory.

Also applies to: 78-105, 123-135

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java`
around lines 70 - 74, The loop in ClickOnTextWithSpacesAndWait creates a
temporary screenshot via saveScreenshotToFile on every poll but never deletes
it, leaking PNGs; update the polling logic (the method in class
ClickOnTextWithSpacesAndWait that calls saveScreenshotToFile and
extractTextPoints) to ensure each screenshotFile is deleted after use in all
paths (miss, success, and exception) by moving deletion into a finally block or
equivalent try-with-resources cleanup; apply the same cleanup to the other
screenshot usages mentioned around the extractTextPoints call (the blocks
covering lines ~78-105 and ~123-135) so every temporary file is removed
regardless of outcome.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilTextPresentInScreen.java-81-88 (1)

81-88: ⚠️ Potential issue | 🟠 Major

Don't stop polling before the declared timeout.

When remainingTime drops below POLLING_INTERVAL_MS, this breaks immediately instead of waiting out the remainder and doing one last OCR check. That makes the action fail up to 1.5 seconds early for texts that appear near the deadline.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilTextPresentInScreen.java`
around lines 81 - 88, The loop exits early when remainingTime <
POLLING_INTERVAL_MS, causing a missed final OCR attempt; in
WaitUntilTextPresentInScreen replace the else { break } behavior by sleeping the
remainingTime (if >0) and then performing one last check instead of breaking —
use remainingTime = endTime - System.currentTimeMillis(); if (remainingTime > 0)
Thread.sleep(remainingTime) (handle InterruptedException) and then proceed to
run the final OCR check (or let the loop iterate once more) rather than breaking
out; reference POLLING_INTERVAL_MS, remainingTime, endTime, Thread.sleep and the
WaitUntilTextPresentInScreen polling loop.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnTextWithWait.java-90-98 (1)

90-98: ⚠️ Potential issue | 🟠 Major

The last sub-1.5s window is skipped.

As soon as remainingTime <= POLLING_INTERVAL_MS, this breaks instead of waiting out the remainder and retrying once more. That shortens the requested wait and makes the action flaky near the deadline.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnTextWithWait.java`
around lines 90 - 98, The loop in ClickOnTextWithWait uses endTime and
POLLING_INTERVAL_MS and breaks when remainingTime <= POLLING_INTERVAL_MS, which
skips the final partial wait; instead, compute remainingTime = endTime -
System.currentTimeMillis() and if remainingTime > 0 sleep for
Math.min(POLLING_INTERVAL_MS, remainingTime) (or sleep remainingTime if smaller)
and then retry once more; only break when remainingTime <= 0 so the full
requested timeout is honored.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilImagePresentWithFindImage.java-65-76 (1)

65-76: ⚠️ Potential issue | 🟠 Major

The polling loop never cleans up its temporary files.

Each iteration writes a new screenshot PNG, and urlToFileConverter() may also create a temp copy for remote image-url inputs. None of those files are deleted on success, timeout, or exception, so repeated waits will keep growing the temp directory. Be careful to only clean up files created by this action, not caller-supplied local paths.

Also applies to: 79-90, 173-207

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilImagePresentWithFindImage.java`
around lines 65 - 76, The polling loop creates temporary screenshot files and
urlToFileConverter("target_image", imageUrlValue) may create a temp copy for
remote inputs but none are deleted; update WaitUntilImagePresentWithFindImage to
track which files it creates (e.g., the per-iteration screenshotFile and a
boolean/marker for the searchImageFile returned by urlToFileConverter) and
delete those files on success, timeout, and on any exception (use try/finally
around the loop and ensure inside-loop cleanup of each screenshotFile after
processing each iteration). Ensure you only delete files that this action
created (do not delete caller-supplied local paths) by using the marker or by
detecting remote-generated temp files from urlToFileConverter before removing
them. Also apply the same cleanup pattern to the other affected blocks
referenced (lines ~79-90 and ~173-207) where temp files are created.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/VerifyTextInApplication.java-17-31 (1)

17-31: ⚠️ Potential issue | 🟠 Major

The result runtime variable is declared but never set.

The action text says it stores the outcome in result-variable-name, but testData1 is never assigned in either branch. Later steps cannot consume the verification result this action advertises.

Also applies to: 59-66

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/VerifyTextInApplication.java`
around lines 17 - 31, The runtime variable field testData1 (in class
VerifyTextInApplication) is never assigned; update the action's execution flow
(inside the execute method or the branches that determine text presence) to set
the runtime variable represented by testData1 to the boolean result (e.g.,
"true"/"false") after the OCR check. Use the TestData runtime API on testData1
(e.g., call the TestData runtime setter provided by the Testsigma SDK such as
setValue/setRuntimeValue or the project’s runtime variable helper) to persist
the outcome so later steps can read the variable.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/VerifyTextInApplication.java-24-37 (1)

24-37: ⚠️ Potential issue | 🟠 Major

The advertised runtime variable is never populated.

The action text says the result is stored in test-data1, but testData1 is never written before either return path. Downstream steps will always see that runtime variable unset.

Also applies to: 93-101

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/VerifyTextInApplication.java`
around lines 24 - 37, The runtime variable test-data1 is never assigned before
the method returns; update the VerifyTextInApplication execution flow (e.g., in
the execute() or equivalent method) to set testData1 with the result before
every return path — for example call testData1.setValue(String.valueOf(found))
or testData1.setValue(foundText) depending on expected value and then proceed
with the existing result handling; apply the same fix to the similar block
mentioned around lines 93-101 so all exit paths write to testData1.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java-107-116 (1)

107-116: ⚠️ Potential issue | 🟠 Major

This wait can exit up to one poll early.

When remainingTime <= POLLING_INTERVAL_MS, the loop breaks instead of waiting out the remainder and retrying once more. That shortens the advertised timeout and makes late-appearing text flaky.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java`
around lines 107 - 116, The loop can exit up to one poll early because it breaks
when remainingTime <= POLLING_INTERVAL_MS; instead, compute a sleep duration as
the lesser of POLLING_INTERVAL_MS and remainingTime and sleep that amount if
remainingTime > 0, only breaking when remainingTime <= 0. Update the block in
ClickOnTextWithSpacesAndWait to use remainingTime/endTime and
POLLING_INTERVAL_MS to derive nextSleep = Math.min(POLLING_INTERVAL_MS,
remainingTime), log the actual sleep seconds, call Thread.sleep(nextSleep) when
remainingTime > 0, and only break when remainingTime <= 0 so the full timeout is
honored.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java-173-179 (1)

173-179: ⚠️ Potential issue | 🟠 Major

Treat "no text in this frame" as a miss, not a failure.

In a wait action, an OCR response with no text should just mean "keep polling." Throwing here aborts the step on the first blank frame instead of waiting until timeout.

🛠️ Suggested change
-                    if (!ocrResponse.hasText()) {
-                        throw new RuntimeException("No text found in the image");
-                    }
+                    if (!ocrResponse.hasText()) {
+                        logger.info("No OCR text detected in this poll attempt");
+                        return java.util.Collections.emptyList();
+                    }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java`
around lines 173 - 179, The current logic in ClickOnTextWithSpacesAndWait
erroneously throws on ocrResponse.hasText() == false, causing the wait action to
abort on the first blank frame; change the behavior so only
ocrResponse.hasError() throws, while a missing text result is treated as a
non-fatal miss (continue polling until timeout) by returning to the polling loop
(or skipping this iteration) instead of throwing—update the code paths in
ClickOnTextWithSpacesAndWait where ocrResponse is checked so that
!ocrResponse.hasText() does not raise an exception but allows the wait loop to
keep retrying.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java-73-73 (1)

73-73: ⚠️ Potential issue | 🟠 Major

Avoid logging raw OCR text from the entire screen.

printAllCoordinates() emits every recognized string and coordinate at info, which can leak unrelated UI content into step logs. Log the match target or element count instead.

Also applies to: 217-220

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java`
at line 73, The call to printAllCoordinates(textPoints) in
ClickOnPositionRelativeToText logs all OCR-recognized strings and coordinates
and may leak unrelated UI content; replace these info-level dumps with minimal
safe logs that record only the search target and counts (e.g., "Found N matches
for '<targetText>'") and/or the chosen coordinate used for the click. Update
both occurrences that call printAllCoordinates (the one shown and the other call
around the 217-220 region) to stop printing raw OCR text and instead log the
target string and textPoints.size() and the final selected Point.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/VerifyTextInApplicationWithAI.java-45-53 (1)

45-53: ⚠️ Potential issue | 🟠 Major

Clean up the temp screenshot after upload.

saveScreenshotToFile() creates a persistent temp PNG, but neither the success path nor the catch path removes it. Repeated steps will leave stale screenshots on the agent.

Also applies to: 65-69, 73-83

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/VerifyTextInApplicationWithAI.java`
around lines 45 - 53, The temp screenshot file created by saveScreenshotToFile
in VerifyTextInApplicationWithAI is never deleted; after calling
ScreenshotUtils.uploadScreenshotToS3 (and in any catch/failure branches) ensure
the File returned by saveScreenshotToFile is deleted (or deletedOnExit) to avoid
accumulating files. Update the method containing the snippet to wrap screenshot
creation/processing in a try/finally (or ensure cleanup in both success and
exception handlers) and call screenshotFile.delete() (or Files.deleteIfExists)
after upload and after any early returns so that saveScreenshotToFile,
OCRUtils.extractTextPoints, OCRUtils.searchForText, and
ScreenshotUtils.uploadScreenshotToS3 all run but the temporary file is always
removed afterward.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java-157-158 (1)

157-158: ⚠️ Potential issue | 🟠 Major

Remove raw OCR text from logs.

Logging the full OCR response body and all detected strings dumps full-screen text into step logs, which is a privacy/compliance risk. Log counts or the matched string only.

Also applies to: 218-221

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java`
around lines 157 - 158, The code currently logs the full OCR response body and
all detected strings (see logger.info("OCR Response body: " + responseBody) in
ClickOnTextWithWait and the similar logging at the later block around lines
218-221); remove these raw text dumps and replace them with privacy-safe logs
that only record summary info such as the number of detected text blocks and, if
needed, the single matched string (not the entire response). Update the logging
calls that use responseBody and any collections of detected strings to log
counts (e.g., detectedList.size()) and/or the matchedText variable only, and
ensure no full OCR payload or full list of detected strings is written to logs.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java-50-118 (1)

50-118: ⚠️ Potential issue | 🟠 Major

Honor wait-time-in-seconds in the execution path.

maxWaitSeconds is only used in log/error text here. The action does a single OCR pass and then fails immediately, so any text that appears after the first capture is missed.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java`
around lines 50 - 118, The current implementation in
ClickOnPositionRelativeToText does a single OCR pass and ignores maxWaitSeconds;
change the flow to poll until the timeout by wrapping the
screen-capture→saveScreenshotToFile→ocr.extractTextFromImage→getTextPointFromText
sequence in a loop that checks elapsed time against maxWaitSeconds (use
System.currentTimeMillis() or similar), sleeping a short interval between
attempts, and breaking when OCR returns a non-null OCRTextPoint; ensure
calculateClickPosition, robot actions and the temporary screenshotFile cleanup
happen only after a successful find, and always delete any temporary files
created in each iteration to avoid leaks; keep existing log messages but move
the timeout log/error and Result.FAILED return to after the polling loop so the
action honors wait-time-in-seconds.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java-71-72 (1)

71-72: ⚠️ Potential issue | 🟠 Major

Delete the temp screenshot on every polling iteration.

Each pass through the wait loop creates a new temp PNG and none of them are removed on retry, success, or exception. A long timeout will leak many files on the agent.

Also applies to: 97-119

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java`
around lines 71 - 72, The per-iteration temp PNG created via
saveScreenshotToFile(screenCapture, "click_text_screenshot") is never removed;
modify the polling loop in ClickOnTextWithWait so each iteration deletes the
temporary screenshot after use (including on success and on exceptions). Wrap
the per-iteration screenshot creation/processing in a try/finally (or ensure a
Files.deleteIfExists/screenshotFile.delete() is called) so the File returned
from saveScreenshotToFile is removed after processing; apply the same cleanup
for the other occurrences in the wait loop (the block covering the later
saveScreenshotToFile calls around lines 97-119).
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java-67-67 (1)

67-67: ⚠️ Potential issue | 🟠 Major

Delete the temp screenshot from a finally block.

The current delete() calls only run on the normal success/not-found branches. Any OCR or click failure after saveScreenshotToFile(...) leaks the PNG on the local agent.

Also applies to: 98-109

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java`
at line 67, Declare and assign the File returned by saveScreenshotToFile(...)
(the screenshotFile variable) outside/above the try block that performs OCR and
clicking, and then move the deletion logic into a finally block so the temporary
PNG is always removed even if OCR or click throws; specifically, ensure in the
finally you check screenshotFile != null && screenshotFile.exists() and attempt
screenshotFile.delete() (with optional logging on failure), and apply the same
change to the other similar block referenced around lines 98-109 so both
saveScreenshotToFile usages always clean up their temp files.
windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java-163-169 (1)

163-169: ⚠️ Potential issue | 🟠 Major

Return an empty OCR result when nothing is detected.

Throwing here makes a blank/loading screen fail the step immediately, even though this action is supposed to keep polling until the wait time expires.

♻️ Suggested change
-                    if (!ocrResponse.hasText()) {
-                        throw new RuntimeException("No text found in the image");
-                    }
-
-                    return ocrResponse.getText();
+                    if (!ocrResponse.hasText()) {
+                        return java.util.Collections.emptyList();
+                    }
+
+                    return ocrResponse.getText();
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java`
around lines 163 - 169, The code currently throws a RuntimeException when
ocrResponse.hasText() is false, causing the step to fail immediately; modify
ClickOnTextWithWait so that instead of throwing for missing text it returns an
empty/blank OCR result (an OCRResponse instance with no text or a clearly empty
marker) so the polling loop can continue until the wait expires; keep the
existing throw for ocrResponse.hasError() as-is, and replace the `throw new
RuntimeException("No text found in the image")` branch with a
return/empty-response path that the surrounding polling logic in
ClickOnTextWithWait can recognize and retry.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6675d7a8-875a-4144-905e-9af6f52f58ca

📥 Commits

Reviewing files that changed from the base of the PR and between 5e7c165 and 2ce9ef5.

📒 Files selected for processing (56)
  • windows_advanced_actions/pom.xml
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/Constants.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/KeyboardUtils.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRResponse.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRTextPoint.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRUtils.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/ResponseObjectForFindImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/ScreenshotUtils.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/util/StringCompareUtil.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/AddDataToClipboard.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/AddDataToFile.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnFixedCoordinates.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnImageWithFindImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnPositionRelativeToImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnPositionRelativeToText.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnRelativeCoordinates.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithSpacesAndWait.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ClickOnTextWithWait.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/CopyAndPasteTestDataOnScreen.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/EnterDataUsingKeyboard.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/EnterDataUsingKeyboardAndPressEnter.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/EraseDataInFile.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/ExtractDataFromScreen.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PerformMultipleActions.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PressFunctionKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PressModifierKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PressModifierKeyWithBasicKeyCombination.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PressModifierKeyWithGivenKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PressTwoModifierKeys.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/PressTwoModifierKeysWithBasicKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/StorePresenceOfTextInRuntimeVariable.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/StringCompareWindowsAdvanced.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/VerifyTextInApplication.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/VerifyTextInApplicationWithAI.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilImagePresent.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilImagePresentWithFindImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/WaitUntilTextPresentInScreen.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToImage.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToText.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnTextWithWait.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/CopyAndPasteTestDataOnScreen.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/EnterDataUsingKeyboard.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/EnterDataUsingKeyboardAndPressEnter.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/PressFunctionKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/PressModifierKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/PressModifierKeyWithBasicKeyCombination.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/PressModifierKeyWithGivenKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/PressTwoModifierKeys.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/PressTwoModifierKeysWithBasicKey.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/RightClick.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/StorePresenceOfTextInRuntimeVariable.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/VerifyTextInApplication.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/WaitUntilImagePresent.java
  • windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/WaitUntilTextPresentInScreen.java

Comment on lines +4 to +10
public static String VISUAL_SERVER_API_END_POINT = "https://visualtesting-staging.testsigma.com/image_analysis_with_files";
public static String VISUAL_SERVER_FIND_IMAGE_ENDPOINT = "https://visualtesting-staging.testsigma.com/find_image_with_files";
public static String VISUAL_SERVER_OCR_TEXT_ENDPOINT = "https://visualtesting-staging.testsigma.com/ocr-text-points-with-files";



public static String API_TOKEN = "DHRNEYFTDCDGOEDDCOEICVYOEEUY";
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Move the visual-server config out of source control.

API_TOKEN is embedded in a public addon, and the endpoints are hard-wired to a staging host. That leaks a credential and makes rotation/promotion unsafe; load both from runtime config or a secret store instead.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/util/Constants.java`
around lines 4 - 10, Constants currently hard-code VISUAL_SERVER_API_END_POINT,
VISUAL_SERVER_FIND_IMAGE_ENDPOINT, VISUAL_SERVER_OCR_TEXT_ENDPOINT and API_TOKEN
in class Constants; remove these secrets/hosts from source and instead load them
at runtime (e.g., via environment variables, a configuration provider or secret
manager). Update Constants to read values from System.getenv()/a config helper
or inject a Config/Secrets service and fail fast if missing, referencing
VISUAL_SERVER_API_END_POINT, VISUAL_SERVER_FIND_IMAGE_ENDPOINT,
VISUAL_SERVER_OCR_TEXT_ENDPOINT and API_TOKEN so callers keep the same
identifiers while the actual values are sourced securely at runtime.

Comment on lines +45 to +53
if (operationResult) {
logger.info("Operation success: " + getSuccessMessage());
setSuccessMessage(getSuccessMessage());
result = com.testsigma.sdk.Result.SUCCESS;
} else {
logger.info("Operation failed: " + getErrorMessage());
setErrorMessage(getErrorMessage());
result = com.testsigma.sdk.Result.FAILED;
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical bug: Success/error messages from utility are never used.

The code calls getSuccessMessage() and getErrorMessage() on this (the action class), not on the util object that computed the messages. This results in null or default messages being set instead of the meaningful messages generated by StringCompareUtil.

🐛 Proposed fix to use utility's messages
         if (operationResult) {
-            logger.info("Operation success: " + getSuccessMessage());
-            setSuccessMessage(getSuccessMessage());
+            logger.info("Operation success: " + util.getSuccessMessage());
+            setSuccessMessage(util.getSuccessMessage());
             result = com.testsigma.sdk.Result.SUCCESS;
         } else {
-            logger.info("Operation failed: " + getErrorMessage());
-            setErrorMessage(getErrorMessage());
+            logger.info("Operation failed: " + util.getErrorMessage());
+            setErrorMessage(util.getErrorMessage());
             result = com.testsigma.sdk.Result.FAILED;
         }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (operationResult) {
logger.info("Operation success: " + getSuccessMessage());
setSuccessMessage(getSuccessMessage());
result = com.testsigma.sdk.Result.SUCCESS;
} else {
logger.info("Operation failed: " + getErrorMessage());
setErrorMessage(getErrorMessage());
result = com.testsigma.sdk.Result.FAILED;
}
if (operationResult) {
logger.info("Operation success: " + util.getSuccessMessage());
setSuccessMessage(util.getSuccessMessage());
result = com.testsigma.sdk.Result.SUCCESS;
} else {
logger.info("Operation failed: " + util.getErrorMessage());
setErrorMessage(util.getErrorMessage());
result = com.testsigma.sdk.Result.FAILED;
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsAdvanced/StringCompareWindowsAdvanced.java`
around lines 45 - 53, The success/error messages are being read from the action
instance instead of the StringCompareUtil that computed them; update the logging
and message setters in StringCompareWindowsAdvanced (the branch where
operationResult is true/false) to use util.getSuccessMessage() and
util.getErrorMessage() (and pass those values to setSuccessMessage and
setErrorMessage and to logger.info) so the actual utility-generated messages are
logged and stored.

Comment on lines +94 to +95
cleanupFile(targetImageFile);
cleanupFile(baseImageFile);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Only clean up temp files you created.

For local inputs, urlToFileConverter() returns new File(url), and the success path always passes that file to cleanupFile(). A successful run therefore deletes the caller’s original image, while failure paths leak temp files because cleanup never runs from a finally block.

Also applies to: 163-166

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnImage.java`
around lines 94 - 95, The current cleanup calls (cleanupFile(targetImageFile);
cleanupFile(baseImageFile);) can delete caller-owned local files because
urlToFileConverter() returns new File(url) for local inputs; instead, track
which files you actually create (e.g., set boolean flags like targetImageIsTemp
and baseImageIsTemp when you download or create temp files) and move all cleanup
into a finally block; in that finally only call cleanupFile(...) for files where
the corresponding isTemp flag is true. Apply the same pattern to the second
cleanup site (the other cleanup calls around the same logic) so only temp files
created by ClickOnImage are deleted and all temp cleanup runs on both success
and failure.

Comment on lines +52 to +53
File targetImageFile = urlToFileConverter("target_image", imageUrlValue);
logger.info("Target image file prepared: " + targetImageFile.getAbsolutePath());
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Don't delete caller-supplied local image files.

For local image-url values, urlToFileConverter() returns new File(url), and both success/failure paths pass that file to cleanupFile(). That will remove the user's source image from disk. The converted file is also not used by ocr.findImage(), which still receives the original string.

Also applies to: 71-71, 102-113, 195-229

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/ClickOnPositionRelativeToImage.java`
around lines 52 - 53, The code currently calls
urlToFileConverter("target_image", imageUrlValue) and later always invokes
cleanupFile(...) which deletes caller-supplied local files because
urlToFileConverter returns new File(url) for local paths; change the cleanup
logic to only delete files that were actually created/downloaded by
urlToFileConverter (temporary files). Implement this by modifying
urlToFileConverter to return or expose a flag (or return a small holder)
indicating whether the File is temporary, or detect remote vs local input (e.g.,
URL scheme or temp directory), then call cleanupFile(targetImageFile) only when
that flag indicates ownership; ensure ocr.findImage(...) still receives the
correct original string where required and apply the same guarded cleanup to all
other usages noted (the blocks around targetImageFile at lines ~71, ~102-113,
and ~195-229).

Comment on lines +41 to +51
StringSelection stringSelection = new StringSelection(testData.getValue().toString());
Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
Transferable data = clipboard.getContents(null);
clipboard.setContents(stringSelection, null);
robot.keyPress(KeyEvent.VK_CONTROL);
robot.keyPress(KeyEvent.VK_V);
KeyboardUtils.sleep(100);
robot.keyRelease(KeyEvent.VK_V);
robot.keyRelease(KeyEvent.VK_CONTROL);
Thread.sleep(500);
clipboard.setContents(data, null);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Restore the clipboard in a finally block and handle a null return from getContents().

Clipboard.getContents(null) can return null, and any exception after modifying the clipboard leaves test data in the system clipboard. This pollutes the machine's clipboard and can cause subsequent paste operations to fail or expose temporary data. Clipboard restoration must occur in a finally block.

📋 Proposed fix
-            Transferable data = clipboard.getContents(null);
-            clipboard.setContents(stringSelection, null);
-            robot.keyPress(KeyEvent.VK_CONTROL);
-            robot.keyPress(KeyEvent.VK_V);
-            KeyboardUtils.sleep(100);
-            robot.keyRelease(KeyEvent.VK_V);
-            robot.keyRelease(KeyEvent.VK_CONTROL);
-            Thread.sleep(500);
-            clipboard.setContents(data, null);
+            Transferable originalContents = clipboard.getContents(null);
+            clipboard.setContents(stringSelection, null);
+            try {
+                robot.keyPress(KeyEvent.VK_CONTROL);
+                robot.keyPress(KeyEvent.VK_V);
+                KeyboardUtils.sleep(100);
+                robot.keyRelease(KeyEvent.VK_V);
+                robot.keyRelease(KeyEvent.VK_CONTROL);
+                Thread.sleep(500);
+            } finally {
+                clipboard.setContents(
+                        originalContents != null ? originalContents : new StringSelection(""),
+                        null
+                );
+            }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
StringSelection stringSelection = new StringSelection(testData.getValue().toString());
Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
Transferable data = clipboard.getContents(null);
clipboard.setContents(stringSelection, null);
robot.keyPress(KeyEvent.VK_CONTROL);
robot.keyPress(KeyEvent.VK_V);
KeyboardUtils.sleep(100);
robot.keyRelease(KeyEvent.VK_V);
robot.keyRelease(KeyEvent.VK_CONTROL);
Thread.sleep(500);
clipboard.setContents(data, null);
StringSelection stringSelection = new StringSelection(testData.getValue().toString());
Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
Transferable originalContents = clipboard.getContents(null);
clipboard.setContents(stringSelection, null);
try {
robot.keyPress(KeyEvent.VK_CONTROL);
robot.keyPress(KeyEvent.VK_V);
KeyboardUtils.sleep(100);
robot.keyRelease(KeyEvent.VK_V);
robot.keyRelease(KeyEvent.VK_CONTROL);
Thread.sleep(500);
} finally {
clipboard.setContents(
originalContents != null ? originalContents : new StringSelection(""),
null
);
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/CopyAndPasteTestDataOnScreen.java`
around lines 41 - 51, The clipboard manipulation in CopyAndPasteTestDataOnScreen
uses Clipboard.getContents(null) and then overwrites the clipboard without
guaranteeing restoration; wrap the clipboard replace + Robot key events
(StringSelection creation, clipboard.setContents(...), robot.keyPress/Release,
Thread.sleep) in a try block and move the clipboard.setContents(originalData,
null) into a finally block so the original Transferable (which may be null) is
restored after the operation; explicitly handle a null return from getContents()
when restoring (only call setContents if original is non-null or restore to an
empty/known safe value) and catch/log any IllegalStateException when calling
clipboard.setContents to avoid leaving the clipboard polluted.

Comment on lines +141 to +165
private boolean parseAIResponse(String aiResponse) {
if (aiResponse == null || aiResponse.trim().isEmpty()) {
logger.debug("AI response is null or empty");
return false;
}

String response = aiResponse.trim().toUpperCase();
logger.info("Parsing AI response: " + response);

// Check for various positive responses
if (response.contains("YES") || response.contains("TRUE") || response.contains("FOUND") ||
response.contains("PRESENT") || response.contains("EXISTS")) {
return true;
}

// Check for various negative responses
if (response.contains("NO") || response.contains("FALSE") || response.contains("NOT FOUND") ||
response.contains("ABSENT") || response.contains("NOT PRESENT")) {
return false;
}

// If response is unclear, log it and return false
logger.debug("Unclear AI response: " + aiResponse + ". Treating as 'not found'.");
return false;
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical bug: Positive keywords match before negative keywords are checked.

The parsing logic checks for positive keywords first (lines 151-152), but several negative phrases contain positive substrings. For example:

  • "NOT FOUND" contains "FOUND" → incorrectly returns true
  • "NOT PRESENT" contains "PRESENT" → incorrectly returns true
  • "ABSENT" does not contain positive keywords, but the pattern is error-prone

The negative check (lines 157-158) is never reached when the response contains these phrases.

🐛 Proposed fix: Check negative phrases first
     private boolean parseAIResponse(String aiResponse) {
         if (aiResponse == null || aiResponse.trim().isEmpty()) {
             logger.debug("AI response is null or empty");
             return false;
         }
 
         String response = aiResponse.trim().toUpperCase();
         logger.info("Parsing AI response: " + response);
 
-        // Check for various positive responses
-        if (response.contains("YES") || response.contains("TRUE") || response.contains("FOUND") ||
-                response.contains("PRESENT") || response.contains("EXISTS")) {
-            return true;
-        }
-
         // Check for various negative responses
-        if (response.contains("NO") || response.contains("FALSE") || response.contains("NOT FOUND") ||
+        if (response.contains("NOT FOUND") || response.contains("NOT PRESENT") ||
                 response.contains("ABSENT") || response.contains("NOT PRESENT")) {
             return false;
         }
+        
+        // Check for various positive responses
+        if (response.contains("YES") || response.contains("FOUND") ||
+                response.contains("PRESENT") || response.contains("EXISTS")) {
+            return true;
+        }
+        
+        // Check simple NO/FALSE last to avoid matching "NOT" prefix issues
+        if (response.contains("NO") || response.contains("FALSE")) {
+            return false;
+        }
 
         // If response is unclear, log it and return false
         logger.debug("Unclear AI response: " + aiResponse + ". Treating as 'not found'.");
         return false;
     }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/StorePresenceOfTextInRuntimeVariable.java`
around lines 141 - 165, In parseAIResponse, the positive keyword check runs
before negatives which causes phrases like "NOT FOUND"/"NOT PRESENT" to match
"FOUND"/"PRESENT" and return true; fix by evaluating negative phrases first
(check for "NO", "FALSE", "NOT FOUND", "NOT PRESENT", "ABSENT" against the
uppercased trimmed aiResponse) and returning false if any match, then evaluate
positive phrases and return true; keep existing null/empty guard (aiResponse ==
null || aiResponse.trim().isEmpty()) and logging (logger.debug/logger.info)
intact.

Comment on lines +147 to +156
// Check for various positive responses
if (response.contains("YES") || response.contains("TRUE") || response.contains("FOUND") ||
response.contains("PRESENT") || response.contains("EXISTS")) {
return true;
}

// Check for various negative responses
if (response.contains("NO") || response.contains("FALSE") || response.contains("NOT FOUND") ||
response.contains("ABSENT") || response.contains("NOT PRESENT")) {
return false;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Negative AI answers can be parsed as success.

"NOT FOUND" and "NOT PRESENT" both contain "FOUND"/"PRESENT", and the positive branch runs first. A response like No, the text was not found will incorrectly pass this step.

🐛 Suggested parser fix
-        // Check for various positive responses
-        if (response.contains("YES") || response.contains("TRUE") || response.contains("FOUND") || 
-            response.contains("PRESENT") || response.contains("EXISTS")) {
-            return true;
-        }
-
-        // Check for various negative responses
-        if (response.contains("NO") || response.contains("FALSE") || response.contains("NOT FOUND") || 
-            response.contains("ABSENT") || response.contains("NOT PRESENT")) {
+        if (response.contains("NOT FOUND") || response.contains("NOT PRESENT")
+                || response.contains("ABSENT") || response.startsWith("NO")
+                || response.startsWith("FALSE")) {
             return false;
         }
+
+        if (response.startsWith("YES") || response.startsWith("TRUE")
+                || response.contains("FOUND") || response.contains("PRESENT")
+                || response.contains("EXISTS")) {
+            return true;
+        }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Check for various positive responses
if (response.contains("YES") || response.contains("TRUE") || response.contains("FOUND") ||
response.contains("PRESENT") || response.contains("EXISTS")) {
return true;
}
// Check for various negative responses
if (response.contains("NO") || response.contains("FALSE") || response.contains("NOT FOUND") ||
response.contains("ABSENT") || response.contains("NOT PRESENT")) {
return false;
if (response.contains("NOT FOUND") || response.contains("NOT PRESENT")
|| response.contains("ABSENT") || response.startsWith("NO")
|| response.startsWith("FALSE")) {
return false;
}
if (response.startsWith("YES") || response.startsWith("TRUE")
|| response.contains("FOUND") || response.contains("PRESENT")
|| response.contains("EXISTS")) {
return true;
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/VerifyTextInApplication.java`
around lines 147 - 156, The current contains-based checks in
VerifyTextInApplication treat "NOT FOUND"/"NOT PRESENT" as positives because
"FOUND"/"PRESENT" are matched first; modify the logic to either test negative
patterns before positives or, better, replace response.contains(...) checks with
whole-word/word-boundary matching (e.g., use Pattern/Matcher with regex like
"\bNOT FOUND\b", "\bNOT PRESENT\b", "\bFOUND\b", "\bPRESENT\b" against the
uppercased response variable) so "No, the text was not found" matches the
negative branch; update the if-blocks referencing response.contains(...)
accordingly.

Comment on lines +67 to +123
while (System.currentTimeMillis() < endTime) {
try {
// Fetch the Details of the Screen Size
Rectangle screenSize = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());

// Take the Snapshot of the Screen
BufferedImage tmp = robot.createScreenCapture(screenSize);

// Provide the destination details to copy the screenshot
String tempDir = System.getProperty("java.io.tmpdir");
String filename = "screenshot"+System.currentTimeMillis()+".jpg";
String path = tempDir + filename;

// To copy source image in to destination path
ImageIO.write(tmp, "jpg",new File(path));
int width = tmp.getWidth();
int height = tmp.getHeight();
logger.info("Width of image: " + width);
logger.info("Height of image: " + height);

File baseImageFile = new File(path);
String url = testStepResult.getScreenshotUrl();
ocr.uploadFile(url, baseImageFile);
logger.info("url: "+ testStepResult.getScreenshotUrl());
FindImageResponse responseObject = ocr.findImage(imageUrl.getValue().toString(),
Float.valueOf(threshold.getValue().toString()));
if (responseObject.getIsFound()){
boolean isFound = responseObject.getIsFound();
int x1 = responseObject.getX1();
int y1 = responseObject.getY1();
int x2 = responseObject.getX2();
int y2 = responseObject.getY2();

int clickLocationX = (x1 + x2) / 2;
int clickLocationY = (y1 + y2) / 2;

logger.info("Click Location X: " + clickLocationX);
logger.info("Click Location Y: " + clickLocationY);


setSuccessMessage("Image Found :" + isFound +
" Image coordinates :" + "x1-" + x1 + ", x2-" + x2 + ", y1-" + y1 + ", y2-" + y2);
Thread.sleep(1000);
return Result.SUCCESS;
} else {
setErrorMessage("Unable to fetch the coordinates");
result = Result.FAILED;
return result;
}
}
catch (Exception e){
logger.info("Exception: "+ ExceptionUtils.getStackTrace(e));
setErrorMessage("Exception occurred while performing click action");
result = Result.FAILED;
return result;
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Polling logic is broken—loop exits on first OCR failure instead of retrying.

The loop immediately returns Result.FAILED (lines 111-114) when responseObject.getIsFound() is false, instead of sleeping and retrying until timeout. This defeats the purpose of the wait-until action.

Additionally:

  1. POLLING_INTERVAL_MS (line 47) is never used—there's no sleep between iterations
  2. The searchImageFile (line 62) is created but never used; imageUrl.getValue().toString() is passed to findImage() instead
🐛 Proposed fix for polling logic
                     if (responseObject.getIsFound()){
                         boolean isFound = responseObject.getIsFound();
                         int x1 = responseObject.getX1();
                         int y1 = responseObject.getY1();
                         int x2 = responseObject.getX2();
                         int y2 = responseObject.getY2();

                         int clickLocationX = (x1 + x2) / 2;
                         int clickLocationY = (y1 + y2) / 2;

                         logger.info("Click Location X: " + clickLocationX);
                         logger.info("Click Location Y: " + clickLocationY);


                         setSuccessMessage("Image Found :" + isFound +
                                 "    Image coordinates :" + "x1-" + x1 + ", x2-" + x2 + ", y1-" + y1 + ", y2-" + y2);
                         Thread.sleep(1000);
                         return Result.SUCCESS;
-                    } else {
-                        setErrorMessage("Unable to fetch the coordinates");
-                        result = Result.FAILED;
-                        return result;
                     }
+                    // Image not found yet - continue polling
+                    logger.info("Image not found yet, will retry...");
                 }
                 catch (Exception e){
                     logger.info("Exception: "+ ExceptionUtils.getStackTrace(e));
-                    setErrorMessage("Exception occurred while performing click action");
-                    result = Result.FAILED;
-                    return result;
+                    // Log and continue polling on transient errors
+                    logger.info("Transient error during polling, will retry...");
                 }
+                
+                // Sleep before next polling attempt
+                long remainingTime = endTime - System.currentTimeMillis();
+                if (remainingTime > 0) {
+                    Thread.sleep(Math.min(POLLING_INTERVAL_MS, remainingTime));
+                }
             }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/windowsLite/WaitUntilImagePresent.java`
around lines 67 - 123, The loop in WaitUntilImagePresent immediately returns
Result.FAILED when responseObject.getIsFound() is false instead of retrying;
change the logic so the method only returns SUCCESS when found or returns FAILED
after the timeout. Specifically, inside the while(System.currentTimeMillis() <
endTime) loop (the block that creates tmp, writes path, uploads via
ocr.uploadFile and calls ocr.findImage), remove the immediate
return/result-setting on a non-match; instead use
Thread.sleep(POLLING_INTERVAL_MS) between attempts (use the existing
POLLING_INTERVAL_MS constant) and continue looping. Also fix the unused
searchImageFile by passing the correct file/URL to ocr.findImage (use the
uploaded testStepResult.getScreenshotUrl() or the created searchImageFile path
consistently) and only set error messages/result = Result.FAILED after the loop
times out (set an appropriate timeout message). Ensure exception handling still
returns FAILED on unexpected errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant