Skip to content

Updated windows advanced addon to replace ai Request with ocr endpoints. #364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ManojTestsigma wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Updated windows advanced addon to replace ai Request with ocr endpoints. #364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
657d70f
Updated windows advanced addon to replace ai Request with ocr endpoints.
ManojTestsigma Mar 25, 2026
2ce9ef5
removed unnecessary files.
ManojTestsigma Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion windows_advanced_actions/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.testsigma.addons</groupId>
<artifactId>windows_advanced_actions</artifactId>
<version>1.0.23</version>
<version>1.0.1</version>
<packaging>jar</packaging>

<properties>
Expand Down
12 changes: 12 additions & 0 deletions windows_advanced_actions/src/main/java/com/testsigma/addons/util/Constants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package com.testsigma.addons.util;

public class Constants {
public static String VISUAL_SERVER_API_END_POINT = "https://visualtesting-staging.testsigma.com/image_analysis_with_files";
public static String VISUAL_SERVER_FIND_IMAGE_ENDPOINT = "https://visualtesting-staging.testsigma.com/find_image_with_files";
public static String VISUAL_SERVER_OCR_TEXT_ENDPOINT = "https://visualtesting-staging.testsigma.com/ocr-text-points-with-files";



public static String API_TOKEN = "DHRNEYFTDCDGOEDDCOEICVYOEEUY";
Comment on lines +4 to +10
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Move the visual-server config out of source control.

API_TOKEN is embedded in a public addon, and the endpoints are hard-wired to a staging host. That leaks a credential and makes rotation/promotion unsafe; load both from runtime config or a secret store instead.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In
`@windows_advanced_actions/src/main/java/com/testsigma/addons/util/Constants.java`
around lines 4 - 10, Constants currently hard-code VISUAL_SERVER_API_END_POINT,
VISUAL_SERVER_FIND_IMAGE_ENDPOINT, VISUAL_SERVER_OCR_TEXT_ENDPOINT and API_TOKEN
in class Constants; remove these secrets/hosts from source and instead load them
at runtime (e.g., via environment variables, a configuration provider or secret
manager). Update Constants to read values from System.getenv()/a config helper
or inject a Config/Secrets service and fail fast if missing, referencing
VISUAL_SERVER_API_END_POINT, VISUAL_SERVER_FIND_IMAGE_ENDPOINT,
VISUAL_SERVER_OCR_TEXT_ENDPOINT and API_TOKEN so callers keep the same
identifiers while the actual values are sourced securely at runtime.


}
95 changes: 94 additions & 1 deletion .../windowsAdvanced/utils/KeyboardUtils.java → .../testsigma/addons/util/KeyboardUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,105 @@
package com.testsigma.addons.windowsAdvanced.utils;
package com.testsigma.addons.util;

import java.awt.*;
import java.awt.event.KeyEvent;
import java.util.HashMap;
import java.util.Map;

/**
* Utility class for keyboard operations and key code mappings
*/
public class KeyboardUtils {

private static final Map<Character, int[]> SPECIAL_CHAR_MAP = new HashMap<>();

static {
// Shifted characters: maps character -> [physicalKeyCode, needsShift (1=yes)]
SPECIAL_CHAR_MAP.put('!', new int[]{KeyEvent.VK_1, 1});
SPECIAL_CHAR_MAP.put('@', new int[]{KeyEvent.VK_2, 1});
SPECIAL_CHAR_MAP.put('#', new int[]{KeyEvent.VK_3, 1});
SPECIAL_CHAR_MAP.put('$', new int[]{KeyEvent.VK_4, 1});
SPECIAL_CHAR_MAP.put('%', new int[]{KeyEvent.VK_5, 1});
SPECIAL_CHAR_MAP.put('^', new int[]{KeyEvent.VK_6, 1});
SPECIAL_CHAR_MAP.put('&', new int[]{KeyEvent.VK_7, 1});
SPECIAL_CHAR_MAP.put('*', new int[]{KeyEvent.VK_8, 1});
SPECIAL_CHAR_MAP.put('(', new int[]{KeyEvent.VK_9, 1});
SPECIAL_CHAR_MAP.put(')', new int[]{KeyEvent.VK_0, 1});
SPECIAL_CHAR_MAP.put('_', new int[]{KeyEvent.VK_MINUS, 1});
SPECIAL_CHAR_MAP.put('+', new int[]{KeyEvent.VK_EQUALS, 1});
SPECIAL_CHAR_MAP.put('{', new int[]{KeyEvent.VK_OPEN_BRACKET, 1});
SPECIAL_CHAR_MAP.put('}', new int[]{KeyEvent.VK_CLOSE_BRACKET, 1});
SPECIAL_CHAR_MAP.put('|', new int[]{KeyEvent.VK_BACK_SLASH, 1});
SPECIAL_CHAR_MAP.put(':', new int[]{KeyEvent.VK_SEMICOLON, 1});
SPECIAL_CHAR_MAP.put('"', new int[]{KeyEvent.VK_QUOTE, 1});
SPECIAL_CHAR_MAP.put('<', new int[]{KeyEvent.VK_COMMA, 1});
SPECIAL_CHAR_MAP.put('>', new int[]{KeyEvent.VK_PERIOD, 1});
SPECIAL_CHAR_MAP.put('?', new int[]{KeyEvent.VK_SLASH, 1});
SPECIAL_CHAR_MAP.put('~', new int[]{KeyEvent.VK_BACK_QUOTE, 1});

// Unshifted special characters
SPECIAL_CHAR_MAP.put('-', new int[]{KeyEvent.VK_MINUS, 0});
SPECIAL_CHAR_MAP.put('=', new int[]{KeyEvent.VK_EQUALS, 0});
SPECIAL_CHAR_MAP.put('[', new int[]{KeyEvent.VK_OPEN_BRACKET, 0});
SPECIAL_CHAR_MAP.put(']', new int[]{KeyEvent.VK_CLOSE_BRACKET, 0});
SPECIAL_CHAR_MAP.put('\\', new int[]{KeyEvent.VK_BACK_SLASH, 0});
SPECIAL_CHAR_MAP.put(';', new int[]{KeyEvent.VK_SEMICOLON, 0});
SPECIAL_CHAR_MAP.put('\'', new int[]{KeyEvent.VK_QUOTE, 0});
SPECIAL_CHAR_MAP.put(',', new int[]{KeyEvent.VK_COMMA, 0});
SPECIAL_CHAR_MAP.put('.', new int[]{KeyEvent.VK_PERIOD, 0});
SPECIAL_CHAR_MAP.put('/', new int[]{KeyEvent.VK_SLASH, 0});
SPECIAL_CHAR_MAP.put('`', new int[]{KeyEvent.VK_BACK_QUOTE, 0});
SPECIAL_CHAR_MAP.put(' ', new int[]{KeyEvent.VK_SPACE, 0});
SPECIAL_CHAR_MAP.put('\t', new int[]{KeyEvent.VK_TAB, 0});
SPECIAL_CHAR_MAP.put('\n', new int[]{KeyEvent.VK_ENTER, 0});
}

/**
* Types a single character using the Robot class, correctly handling
* uppercase letters, digits, and all special characters (US keyboard layout).
*/
public static void typeCharacter(Robot robot, char character) {
if (Character.isLetter(character)) {
int keyCode = KeyEvent.getExtendedKeyCodeForChar(Character.toUpperCase(character));
boolean upperCase = Character.isUpperCase(character);
if (upperCase) {
robot.keyPress(KeyEvent.VK_SHIFT);
}
robot.keyPress(keyCode);
sleep(10);
robot.keyRelease(keyCode);
if (upperCase) {
robot.keyRelease(KeyEvent.VK_SHIFT);
}
return;
}

if (Character.isDigit(character)) {
int keyCode = KeyEvent.getExtendedKeyCodeForChar(character);
robot.keyPress(keyCode);
sleep(10);
robot.keyRelease(keyCode);
return;
}

int[] mapping = SPECIAL_CHAR_MAP.get(character);
if (mapping != null) {
int keyCode = mapping[0];
boolean needsShift = mapping[1] == 1;
if (needsShift) {
robot.keyPress(KeyEvent.VK_SHIFT);
}
robot.keyPress(keyCode);
sleep(10);
robot.keyRelease(keyCode);
if (needsShift) {
robot.keyRelease(KeyEvent.VK_SHIFT);
}
return;
}

throw new IllegalArgumentException("Cannot type character: " + character);
}

// Allowed values for modifier keys
public static final String[] MODIFIER_KEYS = {"Alt", "BackSpace", "CapsLock", "Ctrl", "Delete", "Down", "Enter",
"Esc", "Left", "Right", "Shift", "Tab", "Up", "WINDOW"};
Expand Down
29 changes: 29 additions & 0 deletions windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package com.testsigma.addons.util;

import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;

import java.util.List;

@Data
public class OCRResponse {
@JsonProperty("error")
private String error;

@JsonProperty("text")
private List<OCRTextPoint> text;

// Manual getter methods since Lombok annotation processor is not working
public String getError() { return error; }
public List<OCRTextPoint> getText() { return text; }

// Helper method to check if there are any errors
public boolean hasError() {
return error != null && !error.trim().isEmpty();
}

// Helper method to check if text was found
public boolean hasText() {
return text != null && !text.isEmpty();
}
}
45 changes: 45 additions & 0 deletions windows_advanced_actions/src/main/java/com/testsigma/addons/util/OCRTextPoint.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
package com.testsigma.addons.util;

import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;

@Data
public class OCRTextPoint {
@JsonProperty("text")
private String text;

@JsonProperty("x1")
private double x1;

@JsonProperty("x2")
private double x2;

@JsonProperty("y1")
private double y1;

@JsonProperty("y2")
private double y2;

// Manual getter methods since Lombok annotation processor is not working
public String getText() { return text; }
public double getX1() { return x1; }
public double getX2() { return x2; }
public double getY1() { return y1; }
public double getY2() { return y2; }

// Manual setter methods
public void setText(String text) { this.text = text; }
public void setX1(double x1) { this.x1 = x1; }
public void setX2(double x2) { this.x2 = x2; }
public void setY1(double y1) { this.y1 = y1; }
public void setY2(double y2) { this.y2 = y2; }

// Helper method to get center coordinates for clicking
public double getCenterX() {
return (x1 + x2) / 2.0;
}

public double getCenterY() {
return (y1 + y2) / 2.0;
}
}
Loading