I have tested most of it, but as I don't have a Linux machine easily accessible I can't make any promises until it runs on a real server.

You can flip the switch (enable testing) for your fork on your Travis account page. Then Travis will run the tests for you.

You can flip the switch (enable testing) for your fork on your Travis account page. Then Travis will run the tests for you.

I know, I was just lazy and mentally ignored the option 😉 I'll go do that and see what happens.

Well... good news is it ran. Bad news is we have some missing alts and hrefs. Looks like I have some work to do to get our site up to par 😒

https://travis-ci.org/WasabiFan/ev3dev.github.io/builds/99493798

@dlech Should I continue and fix these reported issues to get a clean build? I want to make sure I don't put much more effort into it if you don't want the changes.

If so, we'll also need to fix the BOMs, some of which you have already fixed in your branch. That means that it would create merge conflicts, so we need to sync up on that.

1039 errors. That's not so bad is it?

Seriously though, wait for me to merge #86 and then try again before fixing anything.

1039 errors. That's not so bad is it?

The majority of them are duplicated on pretty much every page because they are from the header, so it should actually be pretty easy. Although some of them might require some path fiddling...

Seriously though, wait for me to merge #86 and then try again before fixing anything.

Will-do. And honestly, I don't really have any other choice at the moment 😉

Yeah, it looks like if you add an alt="logo" to the logo and alt="screenshot" to the screenshot include, it will take care of a big chunk.

OK, most of the issues are now fixed. I expect that once I fix relative link resolution it'll be happy.

https://travis-ci.org/WasabiFan/ev3dev.github.io/builds/99527585

@dlech Any ideas how I can get the URLs to resolve properly? This is an issue we've been needing to address for a while, so we should get a permanent solution if possible.

The issue is that all our paths reference site.github.url for the root. On dev machines where site.github doesn't exist, it just returns an empty string; so {{ site.github.url }}/foo/bar resolves to just /foo/bar and is treated as a relative path. We need a way to instead control the value that we use so that we can intelligently combine config, site.github.url, and other factors into a root path that makes sense for the environment.

If we put that logic into the page layout, it should allow us to use the variable that we declare in most places that it is needed. But we can't use it in JavaScript files, templates, etc. What do we want to do about this? Have a predefined config line that we copy into every file that it's needed in?

Can we make our own plugin that provides site.github variables? Since GitHub pages blacklists plugins, it would only run on dev machines, but not when you push to GitHub.

Can we make our own plugin that provides site.github variables?

This is a tough question to answer because there are different answers depending on how you meant the question.

GitHub already publishes a plugin called GitHub-metadata that makes the same information that is available on GH Pages servers available on dev machines.

We could write a plugin to put our own data there to fake whatever we need, but that prevents us from using their official plugin to supply other repo metadata, which is something that I would rather not do.

We might be able to write a plugin (maybe fork theirs?) that sets just the URL and forwards the other GitHub data, or maybe it would overwrite just the URL; those are both options, but do require a significant effort.

So what exactly about GitHub-metadata would need to be changed? It sounds like exactly what we need. The readme says it has PAGES_PAGES_HOSTNAME to set the url.

If we enable the real metadata, because we use site.github.url for links, all links on local dev servers will point to www.ev3dev.org... that would make it very hard to work with. As far as I can tell, their gem doesn't let you override values without modification, so I can't think of a way around it. Am I missing something?

I think I might have a viable solution. I will try tomorrow and see.

As far as I can tell, the only item left to be done on this is to merge a finished version of #91 and add the requisite option to that new script to set the base URL for the CI server.

OK, you should be able to modify this using something like this

BASENAME="@TMP@" ./publish.rb --test 'my_test_script'

This should be run with ev3dev.github.io as the current working directory. You must run jekyll before this so that _site exists. This will copy _site to a temporary directory (@TMP@ is a placeholder for this), fixing it up along the way. my_test_script can be a script (or bash function maybe?) that will be run in the temporary directory where fixed up files reside.

OK, I should be able to get it to work using this. Will try later today. Thanks!

@dlech It doesn't look like the publish script is executable. Can you add execute permissions or explain what I'm doing wrong?

It is executable. You could run ruby publish.rb instead, I suppose.

Also, I don't think byte order marks are a problem unless they are in an include file which causes them to end up in the middle of a file instead of at the beginning.

Also, I don't think byte order marks are a problem unless they are in an include file which causes them to end up in the middle of a file instead of at the beginning.

According to the Jekyll docs, they have been known to cause issues -- I think that is reason enough to just remove all of them, regardless of our thoughts on the issue. There's no reason to have them, as far as I know. Plus, if we wanted to be selective, we would need to remove them from all content files, layouts that aren't page, include templates, etc. but not from the others.

@dlech Should links that previously pointed to ev3dev-kernel point to the lego-linux-drivers repo or ev3-kernel? I can find some of the same files in the driver repo, but not all of them. For example, where is ev3_input_port.c? I can try to figure it out if you tell me where to look, or you can do it if you want to.

Unfortunately, at this point I think I have done all I can to fix the broken external links. The rest are all kernel documentation and Ubuntu release notes, which I can't re-locate myself without adequate understanding. After that, HTML validation should be easy to fix by myself, if there are any issues at all.

I just enabled HTML verification and started fixing those issues. It found a few errors in various docs pages, a few in index pages, and some other misc stuff. Most of it was just typos and/or not understanding the types of elements that can go in <p>s. And yes, that last one was definitely caused by me 😉 The others were probably mostly me too but I haven't actually checked/don't care.

I am seeing parser errors on one of the pages -- I can't quire tell what the issue is, but the error is occurring on both a <small> tag and a <title> tag so I don't think it is an issue with the validator not supporting HTML5. I suspect it might either be a hidden character in the title or the ampersand, although I am not sure why that would cause the issues I'm seeing.

We're close!

P.S. After all this is done, you'll need to go through the diff and make sure that I didn't modify any pages that I wasn't supposed to (i.e. pages that were autogened). But it isn't time for that quite yet.

I just pushed another round of link fixes. One thing I learned you might find useful is that jekyll does not have an html escape function. However, I used the xml_escape function to fix htmlParseEntityRef: no name in page titles that contained an &. It might come in handy other places as well.

The most recent build just finished. Only a few more issues left (14 in total). A few of them are very easy to fix, but It'll be an hour or so before I can get to fixing any of the links or other errors.

OK, I'll give it another go.

http://processors.wiki.ti.com seems to be having server issues, so we will have to ignore those for now.

Is there a way to tell the link checker to ignore certain pages, like the example project page?

I've pushed more changes (and apparently caused another merge conflict).

Is there a way to tell the link checker to ignore certain pages, like the example project page?

Yes, there is! Although I think you were asking about ignoring pages on our site, I think the best option is to ignore specific destination URLs. Our test command currently looks like this:

htmlproof ./ --href-ignore /.*example\.com.*/ --check-html --ignore-script-embeds

That --href-ignore tag takes URLs/regexps that we want to ignore. We currently just ignore any URL containing "example.com", which fixes some other issues in the example project. Note that there are slashes and escaped periods before and after "example.com" -- the beginning and end slashes specify that it is a regex, and the escaped periods are regex for "0 or more of any character".

You should be able to add more items so that we also ignore the fake GitHub project on the example page and the ti.com ones. The repo for the lib that we use is here, so you can see what they support. But unfortunately most of the docs are for the Ruby interface, so you have to figure out how to translate what would normally be an array onto the CLI. It might just be [foo, bar, abc] (i.e. a Ruby array literal), it might be a comma-separated list, or maybe you specify the same argument with different values every time. They don't really say 😒.

Good to know. Hopefully the ti links are just temporary. I think they were working earlier today. I don't think we want to add them to the list because we might forget to remove it.

I don't think we're going to get this green today. I think we are close enough on the links for now other than ignoring the github link on the example project page. I can fix the rest later and hopefully the ti wiki will come back online.

Let me know when you are ready to merge.

I don't think we're going to get this green today.

I agree, although I can probably make some progress later today.

Let me know when you are ready to merge.

I'm not sure I understand... you're saying you want to merge but not enable Travis on the main repo? That would make sense. In that case, I don't think there is anything else that I feel I need to do. If, however, you want to then enable the CI server, I would recommend that you instead wait until the build passes so that people opening PRs aren't yelled at for no fault of their own.

Earlier, you said...

P.S. After all this is done, you'll need to go through the diff and make sure that I didn't modify any pages that I wasn't supposed to (i.e. pages that were autogened). But it isn't time for that quite yet.

So I got the impression that you had some more changes you wanted to make before I merge this. If not, I'll go ahead and merge no, otherwise I'll wait for you to say when.

So I got the impression that you had some more changes you wanted to make before I merge this.

I do have more changes that I want to make before we actually enable the CI server on the main repo. But I have no issue with you merging now, assuming that I haven't made any changes that would be lost when you update drivers. And given that I have fixed loads of links and HTML syntax, merging without enabling the server will still provide benefit.

I'm going to go ahead and merge then. It's not going to hurt if I enable CI is it? We know it's just going to fail, but so what?

By the way, thanks very much for this. It's going to be very useful when I do kernel releases to make sure I got all of my links right.

It's not going to hurt if I enable CI is it? We know it's just going to fail, but so what?

It will hurt if someone opens a PR against the repo. It will tell them that their code is broken, even though it isn't -- and they won't know why. And every time you push a commit it will send you an email telling you about it. On the flip side, why would we want to enable it? If you want to check to see if you broke anything between now and when we fix the rest of the issues, you can just run the cibuild.sh script locally (because you're on Linux). And hopefully it will only be a few days until everything works.

It's going to be very useful when I do kernel releases to make sure I got all of my links right.

That's my hope! And it already fixed a bunch of issues that we hadn't (and probably wouldn't have) noticed without it.