Skip to content

Python: Fix import of refined variable#12244

Merged
tausbn merged 27 commits into
github:mainfrom
RasmusWL:import-refined
Mar 24, 2023
Merged

Python: Fix import of refined variable#12244
tausbn merged 27 commits into
github:mainfrom
RasmusWL:import-refined

Conversation

@RasmusWL
Copy link
Copy Markdown
Member

@RasmusWL RasmusWL commented Feb 17, 2023

Such as importing Foo from the following file

class Foo: pass
Foo.bar = 42

The original approach wasn't quite on point, which I discovered with the tests now added in fb425b7 -- so a force push and a lot of rework of the approach later, I think we're at a good enough place now (if performance is reasonable 😳)

(there could still be more work on import resolution, but I'm going to refrain from digging more into it now)

tausbn
tausbn previously approved these changes Feb 17, 2023
Copy link
Copy Markdown
Contributor

@tausbn tausbn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This one might have an impact on performance, so I'll eagerly await the results. 🙂

However, as illustrated by the `CWE-327-InsecureProtocol` test, this fix
is NOT good enough, since now even the `secure_context` is considered to
be insecure (for both versions). Ouch.

Will fix this in a later commit, since it was only discoverd late on.
We need some recursive unwinding to get all of these right
For `from <pkg> import <attr>` we would use to treat the `<pkg>`
(ImportExpr) as a definition of the name `<attr>`.

Since this removes bad import-flow, and nothing broke, I'm guessing this
was never intentional.
Notice that `has_defined_all_indirection` all have both
`all_defined_bar_copy` and `all_defined_foo_copy` marked as exported,
even though only `all_defined_foo_copy` is available.
…> import *`

However, we can see that `from <pkg> import *` and `import pkg` are
handled differently. Would have liked `has_defined_all_indirection` to
behave in the same way no matter how the import was made.
I guess we could have done this at the very start of introducing this
test in this PR, but I think the last commit was mostly inspired from
looking at all the things that evidently was re-exported from the trace
import, even when I knew they were not available because of the
`__all__` definition.
It's not very useful to look at, and it's a mess when you change any
tests to see all the changes lines in the expected output that you
really do not care about!
Just like the one added for `py/insecure-protocol` in fb425b7, but
instead added in the import-resolution tests, such that we don't have to
remember it's in a completely different directory.
It's nice that it fixes the `InsecureProtocol` test-case (which maybe
should have been a test-case for the import resolution library in the
first place?)

But it's not quite right:

1. it adds spurious flow for `clashing_attr`
2. it runs into huge problems for typetracking_imports/tracked.expected
3. it runs into the problem for
   github#10176 with an `from <pkg>
   import *` blocking flow from previously defined variable, that is NOT
   overridden. (simplistic_reexport.bar_attr)
That one line was an afterthought, and certainly did not work as
intended.
@RasmusWL RasmusWL requested a review from a team as a code owner February 23, 2023 09:30
@RasmusWL RasmusWL requested review from asgerf and removed request for a team February 23, 2023 09:30
@calumgrant calumgrant removed the request for review from asgerf February 27, 2023 10:20
@RasmusWL
Copy link
Copy Markdown
Member Author

Tests should pass after a rerun, were only failing due to an unupdated .expected file on main

tausbn
tausbn previously approved these changes Mar 20, 2023
Copy link
Copy Markdown
Contributor

@tausbn tausbn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. 👍

Since we can analyze operator.py from Python3, but not in Python 2
(since it's implemented in C), we get a difference for the index tests.

note: `operator.length_hint` is only available in Python 3.4 and later,
so would always fail under Python 2.
I could not for the life of me figure out why the tests were failing,
when they were working for me locally 🤦
@RasmusWL RasmusWL requested a review from tausbn March 22, 2023 09:02
@RasmusWL
Copy link
Copy Markdown
Member Author

Performance is fine ✔️ new results looks ok from what I can tell.

Copy link
Copy Markdown
Contributor

@tausbn tausbn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! 👍

@tausbn tausbn merged commit c0eb611 into github:main Mar 24, 2023
@RasmusWL RasmusWL deleted the import-refined branch March 24, 2023 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants