@erezrokah I've updated the code to address merge conflict, it's ready for merge now

@erezrokah the issue with timelines was caused by the merging one branch into the other (basically, last commit was the same) as it messed a bit with GH trying not to do the same work several times.

I've updated the branches and have them diverged a bit, it should help with the proper testing.

As for the fact that the AWS source job might take a bit longer than the wait -- it's all in the wait logs (we got the successful status from API, nothing else to wait). Maybe the time displayed for the job also includes some additional tine for artifacts/ something else?

@erezrokah see https://github.com/cloudquery/cloudquery/actions/runs/4649767311/jobs/8228345356?pr=9760#step:4:625 for the wait for the required workflows reasoning

Should this PR be ready? I see the AWS still running but wait for all completed:

As I mentioned, it might be easier to test this for a single plugin instead of all of them. Regardless I think we would like to introduce this change incrementally instead of all plugins as the same time.

As I mentioned, it might be easier to test this for a single plugin instead of all of them. Regardless I think we would like to introduce this change incrementally instead of all plugins as the same time.

@erezrokah seems there's a bug in the logic as we see in here that the following entries triggered the workflows to be marked ready:

{"name":"plugins/source/awspricing","conclusion":"success"},
{"name":"plugins/source/awspricing","conclusion":"success"}

So we saw a double entry for awspricing and no for aws, interesting...
I think fixing the logic for the wait-for-required-workflows may be out of scope of this PR.

I can do a separate PR introducing this change only to AWS source testing, and, if we're satisfied with the results, we can merge this and then do refactoring for the other plugin workflows, WDYT?

So we saw a double entry for awspricing and no for aws, interesting...
I think fixing the logic for the wait-for-required-workflows may be out of scope of this PR.

I haven't seen this issue before, so we should probably fix it before doing any other CI changes.
I'll try to reproduce in a separate PR, without this change and see if it does.

I can do a separate PR introducing this change only to AWS source testing, and, if we're satisfied with the results, we can merge this and then do refactoring for the other plugin workflows, WDYT?

That sounds good, but it will have to wait for the fix in the wait for all logic

This SDK PR #9661 doesn't seem to have this issue:

Maybe we can add the reusable workflow without the skip logic at first to simplify things?

The issue is that the reusable workflow will still require the name validate-release / validate-release to be added, and the skip logic is really minor.
The logic issue we saw when a couple of runs with the same name (awspricing) were shown is due to the paging, I presume.

And the fact that we basically check for the following:

• any failed builds present? -> fail
• the len of filtered completed runs == the amount of runs we wait for -> success

In the run we had issue with we had the same entry counted twice, so that's why we skipped the wait and just went with it.

I will create a separate issue to fix the workflow behavior in the following way (I think it's non-blocking for now):

• account for the id of the run we see (so we skip all the runs we already processed)
• move from array to map (name -> amount) so that we properly check the runs for a specific action

@erezrokah I've extracted simple (no cgo) valiate-release for AWS Source plugin into #9780.
I'll be following up with cgo-releated ones after that one is merged, and then I'll create PRs to replace all others

I will create a separate issue to fix the workflow behavior in the following way (I think it's non-blocking for now):

I think if we have an issue with the wait for all logic, we should fix it first before making other CI changes, otherwise we won't be able to verify the changes we make.

Also I haven't seen that issue on other PRs, so we'll need to investigate. We know what happens on this PR, but haven't confirmed the root cause, nor that it happens on all PRs

Also I haven't seen that issue on other PRs, so we'll need to investigate. We know what happens on this PR, but haven't confirmed the root cause, nor that it happens on all PRs

I think it might be caused by paging by 100 entries (as this is one of the PRs where we have a lot of actions running).
As 100 is max we either need to verify that there's no issue with SDK or account for possible duplicates.

I really think that we can just do it in a separate issue and not make this blocking, as the issue was present even before (take any PR that updates some library for a lot of plugins).

Namely, the SDK PR #9661 you mentioned actually has this issue:
Take a look at the last reported runs.
There's no entry for plugins/source/aws being successful, but we possibly have some duplicate there because of the paging, that's why we just conclude successfully.

There's no entry for plugins/source/aws being successful, but we possibly have some duplicate there because of the paging, that's why we just conclude successfully.

💯 Agree good catch, seems like there's a duplicate for plugins/source/googleads

I really think that we can just do it in a separate issue and not make this blocking, as the issue was present even before (take any PR that updates some library for a lot of plugins).

Let's start with #9793 and log the IDs.

I don't think we should introduce non urgent (big) CI changes if we have a bug in our CI. The motivation for introducing re-usable workflows was to simplify the caching logic, but since then we discovered that we won't need to refactor it, and only fix 4 workflows.

So re-usable workflows seems to be more about removing code duplication (that we haven't had issues with before).
I don't see the urgency to push them

@erezrokah I've created #9797 to address the duplicates issue (possibly, just do the most obvious filter by ID) and make it working again.

@erezrokah I've created #9797 to address the duplicates issue (possibly, just do the most obvious filter by ID) and make it working again.

Can you spot duplicate IDs in https://github.com/cloudquery/cloudquery/actions/runs/4656875292/jobs/8240920196?pr=9736#step:4:381?

I did not find any. That run is even worse as it doesn't wait for CLI Windows and AWS

I counted 60 validate release, 60 workflows (CLI, Scaffold and plugins), 4 test policies, which seems ok.

Update

Looks like we're seeing 60 validate release workflows, but expecting 58 (see validate release count in Actions: for the expected count), so that explains why the CLI Windows and AWS workflow were skipped.

This seems like a different issue than the duplicate IDs, WDYT?

Drop for now, will take another shot after #9780