Fix VS Code request#10
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR implements comprehensive test coverage for the YAML parser module and addresses path resolution issues in the VS Code request handling. It also includes updates to test files to use updated instruction markers and package naming fixes.
- Adds complete unit test coverage for YAML parser path resolution and guideline detection
- Fixes VS Code request format by switching from file:// schemes to proper file URIs
- Updates example files to use distinct instruction markers per language type
Reviewed Changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tests/test_yaml_parser.py | New comprehensive test suite for YAML parser functionality |
| src/tests/test_signatures.py | Remove 'outcome' field validation from QuerySignature test |
| src/tests/test_models.py | Update VS Code model tests to handle absolute paths and new preread format |
| src/tests/test_cli.py | Enhanced CLI tests with LLM judge validation and prompt dumping |
| src/diagnostics/dspy_diag.py | Fix package name capitalization |
| src/bbeval/yaml_parser.py | Enhanced path resolution with verbose logging and absolute path handling |
| src/bbeval/models.py | Update VS Code preread block format and file URI generation |
| src/bbeval/cli.py | Add prompt dumping functionality and package name fixes |
| src/bbeval/init.py | Fix package name capitalization |
| pyproject.toml | Bump version to 0.1.8 |
| docs/examples/simple/prompts/python.instructions.md | Update instruction marker to language-specific version |
| docs/examples/simple/prompts/javascript.instructions.md | New JavaScript instruction file with unique marker |
| docs/examples/simple/evals/example.test.yaml | Update test expectations to use language-specific markers |
| docs/examples/simple/.gitignore | Add BbEval-specific ignore patterns |
| AGENTS.md | Add reference to base instructions |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| mock_judge_instance.assert_called_once_with( | ||
| expected_outcome=test_case.outcome, | ||
| task_requirements=test_case.task, | ||
| request=test_case.task, |
There was a problem hiding this comment.
The parameter name 'request' in the mock judge call should match the signature field name. Based on the context, this should likely be 'task_requirements' to match the QualityGrader signature expectations.
| request=test_case.task, | |
| task_requirements=test_case.task, |
| # Show all attempted paths for better debugging | ||
| attempted = "\n ".join(str(p) for p in attempted_paths) | ||
| print(f"\033[33mWarning: File not found: {file_path_display}") | ||
| print(f" Tried:\n {attempted}\033[0m") |
There was a problem hiding this comment.
The ANSI color codes are hardcoded and repeated throughout the codebase. Consider extracting these into constants or a utility function for consistent warning message formatting.
| ) | ||
|
|
||
| return f"## 1. Mandatory Pre-Read\n\n{consolidated_instruction}\n\n" | ||
| return f"[[ ## mandatory_pre_read ## ]]\n\n{consolidated_instruction}\n\n" |
There was a problem hiding this comment.
The section headers like '[[ ## mandatory_pre_read ## ]]' appear to be magic strings used for VS Code formatting. These should be defined as constants to avoid duplication and make their purpose clearer.
| sanitized = re.sub(r'[^A-Za-z0-9._-]+', '_', value) | ||
| return sanitized or "prompt" |
There was a problem hiding this comment.
The fallback value 'prompt' is a magic string. Consider defining this as a constant like DEFAULT_FILENAME_PREFIX = 'prompt' to make the intent clearer and allow for easier changes.
No description provided.