Skip to content
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
This repository was archived by the owner on Nov 6, 2023. It is now read-only.

fetch-test.sh incorrectly reports an error because it doesn't follow a non-trivial rewrite rule #14365

Description

@jeremyn

Type: code issue

There are three domains that behave like this:

URL result
http://armytimes.com 301 http://www.armytimes.com/
https://armytimes.com Connection refused
http://www.armytimes.com 301 https://www.armytimes.com/
https://www.armytimes.com 200
http://link.armytimes.com 302 http://armytimes.com
https://link.armytimes.com 302 http://armytimes.com

If you run fetch-test.sh (d5c54c4) against the following ruleset, which is a minimal version of ArmyTimes.com.xml:

<ruleset name="ArmyTimes.com (minimal)">
	<target host="armytimes.com" />
	<target host="www.armytimes.com" />
	<target host="link.armytimes.com" />

	<rule from="^http://armytimes\.com/"
		to="https://www.armytimes.com/" />

	<rule from="^http:" to="https:" />
</ruleset>

and change the logging level to debug in test/rules/manual.checker.config, you get:

DEBUG Loaded certificate platforms: cacert,chromium,default,firefox
DEBUG Parsing ArmyTimes-minimal.com.xml
DEBUG Checking coverage for 'ArmyTimes.com (minimal)'.
DEBUG Checking target validity for 'ArmyTimes.com (minimal)'.
DEBUG =**= Start http://armytimes.com/ => https://www.armytimes.com/ ****
DEBUG Fetching transformed page https://www.armytimes.com/
DEBUG Fetching plain page http://armytimes.com/
DEBUG Following redirect http://armytimes.com/ => http://www.armytimes.com/
DEBUG Following redirect http://www.armytimes.com/ => https://www.armytimes.com/
DEBUG ==== D: 0.0000; http://armytimes.com/ (140074) -> https://www.armytimes.com/ (140074) =====
INFO Finished comparing http://armytimes.com/ -> https://www.armytimes.com/. Rulefile: ArmyTimes-minimal.com.xml.
DEBUG =**= Start http://www.armytimes.com/ => https://www.armytimes.com/ ****
DEBUG Fetching transformed page https://www.armytimes.com/
DEBUG Fetching plain page http://www.armytimes.com/
DEBUG Following redirect http://www.armytimes.com/ => https://www.armytimes.com/
DEBUG ==== D: 0.0000; http://www.armytimes.com/ (140074) -> https://www.armytimes.com/ (140074) =====
INFO Finished comparing http://www.armytimes.com/ -> https://www.armytimes.com/. Rulefile: ArmyTimes-minimal.com.xml.
DEBUG =**= Start http://link.armytimes.com/ => https://link.armytimes.com/ ****
DEBUG Fetching transformed page https://link.armytimes.com/
DEBUG Following redirect https://link.armytimes.com/ => http://armytimes.com
DEBUG Redirect rewritten: http://armytimes.com => https://armytimes.com
DEBUG =**= Start http://link.armytimes.com/ => https://link.armytimes.com/ ****
DEBUG Fetching transformed page https://link.armytimes.com/
DEBUG Following redirect https://link.armytimes.com/ => http://armytimes.com
DEBUG Redirect rewritten: http://armytimes.com => https://armytimes.com
DEBUG Fetch error: http://link.armytimes.com/ => https://link.armytimes.com/: (7, 'Failed to connect to armytimes.com port 443: Connection refused')
INFO Finished comparing http://link.armytimes.com/ -> https://link.armytimes.com/. Rulefile: ArmyTimes-minimal.com.xml.
ERROR ArmyTimes-minimal.com.xml: Fetch error: http://link.armytimes.com/ => https://link.armytimes.com/: (7, 'Failed to connect to armytimes.com port 443: Connection refused')
INFO Finished in 2.32 seconds. Loaded rulesets: 1, URL pairs: 3.

The error is invalid because the ruleset checker should use the included extra rule to rewrite http://armytimes.com to https://www.armytimes.com instead of rewriting it to https://armytimes.com .

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions