Skip to content

Update time management of python test generation #1893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tamarinvs19 merged 5 commits into from
Mar 9, 2023
Merged

Update time management of python test generation #1893

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b002d99
Refactor time management
tamarinvs19 Mar 3, 2023
dd863d1
Add iteration counter to baseline algorithm
tamarinvs19 Mar 6, 2023
f56ffaf
Merge branch 'main' into tamarinvs19/timeout-generation
tamarinvs19 Mar 6, 2023
b0cbb92
Add initial annotation handling
tamarinvs19 Mar 6, 2023
310c71b
Update fuzzing process: add duplicate ignoring, fix feedback returning
tamarinvs19 Mar 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add iteration counter to baseline algorithm
  • Loading branch information
@tamarinvs19
tamarinvs19 committed Mar 6, 2023
commit dd863d109769e73f36bcd2c454dd6cca592008da
6 changes: 3 additions & 3 deletions utbot-python/src/main/kotlin/org/utbot/python/PythonTestCaseGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,11 +267,11 @@ class PythonTestCaseGenerator(
return@breaking
}

algo.run(hintCollector.result, typeInferenceCancellation, annotationHandler)
val iterationNumber = algo.run(hintCollector.result, typeInferenceCancellation, annotationHandler)

val existsAnnotation = method.definition.type
if (existsAnnotation.arguments.all { it.pythonTypeName() != "typing.Any" }) {
if (iterationNumber == 1) {
limitManager.mode = TimeoutMode
val existsAnnotation = method.definition.type
annotationHandler(existsAnnotation)
Copy link
Copy Markdown
Member

@tochilinak tochilinak Mar 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially this code was needed to run fuzzing with initial types (even when those are not full). Will we run fuzzing with list[Any] for the following function?

def f(x: list): ...

Copy link
Copy Markdown
Collaborator Author

@tamarinvs19 tamarinvs19 Mar 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe. But only if all arguments have not-typing.Any annotations. And if all annotations are full we run fuzzing twice

Copy link
Copy Markdown
Member

@tochilinak tochilinak Mar 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure? Type inference does not run annotationHandler on initial signature.

Copy link
Copy Markdown
Collaborator Author

@tamarinvs19 tamarinvs19 Mar 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I run fuzzing with initial annotations only after type inference) But I can change program and run fuzzing twice with initial annotations, but I think it isn't a good solution

Copy link
Copy Markdown
Collaborator Author

@tamarinvs19 tamarinvs19 Mar 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added handling of an initial annotation after first expansion in BaselineAlgorithm.

  • if annotation is full we cannot expand it and stop type inference and then start fuzzing;
  • if annotation is not full we try to expand and first of all try to fuzz initial annotation (for example, if it is list[typing.Any] we can generate empty list), after that we continue type inference

}
}
Expand Down
2 changes: 1 addition & 1 deletion utbot-python/src/main/kotlin/org/utbot/python/newtyping/inference/TypeInferenceAPI.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ abstract class TypeInferenceAlgorithm {
hintCollectorResult: HintCollectorResult,
isCancelled: () -> Boolean,
annotationHandler: suspend (Type) -> InferredTypeFeedback,
)
): Int
}

sealed interface InferredTypeFeedback
Expand Down
6 changes: 5 additions & 1 deletion ...python/src/main/kotlin/org/utbot/python/newtyping/inference/baseline/BaselineAlgorithm.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,17 +39,20 @@ class BaselineAlgorithm(
hintCollectorResult: HintCollectorResult,
isCancelled: () -> Boolean,
annotationHandler: suspend (Type) -> InferredTypeFeedback,
) {
): Int {
val generalRating = createGeneralTypeRating(hintCollectorResult, storage)
val initialState = getInitialState(hintCollectorResult, generalRating)
val states: MutableList<BaselineAlgorithmState> = mutableListOf(initialState)
val fileForMypyRuns = TemporaryFileManager.assignTemporaryFile(tag = "mypy.py")
var iterationCounter = 0

run breaking@ {
while (states.isNotEmpty()) {
if (isCancelled())
return@breaking
logger.debug("State number: ${states.size}")
iterationCounter++

val state = chooseState(states)
val newState = expandState(state, storage)
if (newState != null) {
Expand All @@ -71,6 +74,7 @@ class BaselineAlgorithm(
}
}
}
return iterationCounter
}

private fun checkSignature(signature: FunctionType, fileForMypyRuns: File, configFile: File): Boolean {
Expand Down