importing approvaltests triggers http requests against raw.githubusercontent.com creating potential security issues there #248

I noticed some real slowdowns in my python unittest-based test runner recently, just during test discovery. I noticed some really odd network traffic coming from it against an IP address of 133.108.199.188 when I attached to the process with strace while it appeared to be hung. I was very nervous that I was watching a supply chain exploit in action as this IP address appeared to be somewhat nefarious using reverse lookups, and deep diving into the problem led me to the approvaltests module that one of our engineers had recently added to our build and tests, and just importing approvaltests led to these network requests.

The internals/logs/log_commons.py module gets the download_script_from_repo_if_needed() function called every time my tests run to download both remove_abandoned_files.py and approve_all.py which are then saved to a local directory created in my source repository if that request succeeds (which isn't guaranteed). This has the potential (and has) slowed our test runs down a lot when this raw.githubusercontent.com domain isn't running fast or is undergoing IP address changes, as we aren't caching these files all the time for fresh CI runs and for new.

Apart from the annoying startup performance penalty I have to pay here for scripts that don't appear to be used at all when my tests are run, I'm very concerned with this behavior as this code is now downloading files blindly, with no verification of their contents, marking those files as executable and potentially running them on my system. In this case it appeared that an old IP address was getting used by my DNS resolver and that IP address could have been used to serve up malicious content that could have hijacked my system.

I know that we're all trusting DNS at some level, but could these scripts be included in a git submodule, or in another pypi package I could install, so that we can validate this library and not have to worry about it adding different source code dynamically at run time which we haven't validated? Is there a way that I can supply these scripts without having to rely on this magic .approval_tests_temp directory being in only one possible place? How do I know if the scripts I might pre-cache are the correct ones on an ongoing basis? This pattern of dynamically downloading dependencies with no versioning check or content verification is concerning for a lot of obvious reasons.

This did not turn out to be a supply chain exploit thankfully, but I was experiencing very odd behavior against github for a while and these mysterious odd delays have happened many times over the last week. I'd love to know of any workarounds anyone might have, and would love to see some more rigor go into this dependency challenge. The test-focused engineers on our team love your library, and I don't mean to insult anyone here over this issue. If I've understood things wrong, I would love to be corrected. I'd also love to help test any changes to allow library consumers to customize this behavior for performance improvements and for security improvements.

Thanks!