Cache User (1 min) and Teams (1 hour) information by artem-panchenko · Pull Request #100 · jenkinsci/github-oauth-plugin

artem-panchenko · 2018-03-26T16:34:13Z

Changes

The goal of this PR is to reduce requests rate to GitHub API by additional caching:

access token -> user info for a minute;
GH user -> teams info for an hour.

Rationale

If you have a service/bot which uses Jenkins and it makes a lot of API calls, fixed GitHub requests rate may quickly become a bottleneck.

Note

It's my first code written in Java, but I was able to compile it and test with Jenkins 2.107.1 :)
It works fine for our organization and helped us to dramatically decrease requests rate (approximately 10x times), that Jenkins makes to GitHub while authenticating API requests from our bot (just in case it uses python-jenkins client).

Call for HELP!

I'd really appreciate if someone helps me to groom the code and add/fix tests!

P.S.: I could miss something fundamental, so please share your thoughts on the issue I described and am trying to fix! Thank you!

samrocketman · 2018-04-10T16:31:00Z

Thanks for your contribution. I’ll look it over and ask for more experienced eyes for help as well.

samrocketman · 2018-04-10T16:34:12Z

Can someone from @jenkinsci/code-reviewers take a look?

Wadeck

Except the small comment on the changelog, the code seems good from my PoV.

Just consider that with this change, modification of GitHub team settings could take up to 1h to be seen inside Jenkins. OTOH it's already the case for org / repositories, so do not seem to bother users.

It's my first code written in Java

Good job :) even with anonymous class, generic handling, etc., you managed it!

Wadeck · 2018-04-11T14:08:11Z

@@ -1,3 +1,7 @@
+# Version 0.30 (Released Apr 1, 2018)


In general, the good practices is to provide the changelog in the PR, the maintainer will gather them to add to the CHANGELOG when the plugin will be released.

(the release process is not automatic after your PR is merged)

Yes, the changlog entry is fine but please remove the date.

samrocketman · 2018-04-11T19:50:01Z

@@ -1,3 +1,7 @@
+# Version 0.30 (Released Apr 1, 2018)


Yes, the changlog entry is fine but please remove the date.

samrocketman · 2018-04-11T19:52:03Z

When you fix the changelog, please also squash your changes into a single commit.

Once that is done, I'll do some manual testing to validate your change does not adversely affect plugin upgrades. If that goes well, then I'll merge your change.

Wadeck · 2018-04-11T19:59:48Z

squash your changes

@samrocketman why do you ask for that ?

for your local testing purpose ?
- you can just use the last commit on its branch
for the future merge ?
- you can merge using squash directly inside GitHub

Perhaps I missed something :)

samrocketman · 2018-04-11T20:14:14Z

squash your changes

There’s no technical requirement. It’s not even a merge requirement. I typically ask but if it’s not done, it does not stop me from merging it. Probably should have been more clear when asking.

artem-panchenko · 2018-04-12T20:19:32Z

@samrocketman @Wadeck thank you for the feedback! I've updated the change log and squashed my commits as you requested.

samrocketman · 2018-04-13T03:58:05Z

Thanks, I'm on a trip at the moment but will definitely look it over for testing. If you're curious I'll be using the jenkins-bootstrap-jervis project under my user to automatically configure the GitHub OAuth plugin (via settings.groovy).

Will review again

samrocketman

Approved

This pull request offers a speedup by a factor of 10 in my own configurations. I imagine, the more thorough the automated configuration, the better the speedup. thanks for this contribution.

Seconds to configure my Jenkins instance.

Before: 24.019s
After: 3.805s

Ref: https://github.com/samrocketman/jenkins-bootstrap-jervis

How did I build?

git fetch origin refs/pull/100/head
git checkout FETCH_HEAD -b PR-100
mvn clean package

Click to expand

Environment:

$ head -n1 /etc/issue
Ubuntu 16.04.4 LTS \n \l

$ uname -rms
Linux 4.13.0-38-generic x86_64

$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: /home/sam/usr/share/apache-maven-3.3.9
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /home/sam/usr/share/java/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.13.0-38-generic", arch: "amd64", family: "unix"

Before

Click to expand

$ time ./jervis_bootstrap.sh 
Waiting for http://localhost:8080/jnlpJars/jenkins-cli.jar to become available.ready.
Nothing changed.  Usage stats are not submitted to the Jenkins project.
Nothing changed.  Job "_jervis_generator" already exists.
Nothing changed.  Job "__clean_abandoned_branches_multibranch" already exists.
Nothing changed.  View "Welcome" already exists.
Nothing changed.  View "GitHub Organizations" already exists.
Nothing changed.  View "Maintenance" already exists.
Nothing changed.  Primary view already set to "Welcome".
Nothing changed.  Markup Formatter already configured.
Nothing changed.  Docker agent credentials already configured.
Nothing changed.  Agent -> master security already disabled.
Nothing changed.  Job DSL script security already disabled.
Nothing changed.  Agent Protocols already configured: [JNLP4-connect, Ping]
Nothing changed.  Global Jenkinsfile already configured.
Nothing changed.  Global Jenkinsfile script already approved.
Nothing changed.  Master already restricts jobs.
Nothing changed.  Admin groovy scripts already approved.
Nothing changed.  Grape repository config used by @Grab already configured.
Nothing changed.  Jenkins settings already configured.
Nothing changed.  github-token credentials already configured.
Nothing changed.  github-user-and-token credentials already configured.
Nothing changed.  webhook-shared-secret credentials already configured.
Nothing changed.  Pipeline Global Shared Libraries already configured.
Nothing changed.  GitHub plugin already configured.
Nothing changed.  Already using random least loaded provisioning strategy for docker.
Configured Yet Another Docker cloud docker-local
Nothing changed.  GitHub security realm already configured.
Jenkins is ready.  Visit http://localhost:8080/
User: samrocketman

real	0m24.019s
user	0m1.324s
sys	0m0.367s

After

Click to expand

$ time ./jervis_bootstrap.sh 
Waiting for http://localhost:8080/jnlpJars/jenkins-cli.jar to become available.ready.
Nothing changed.  Usage stats are not submitted to the Jenkins project.
Nothing changed.  Job "_jervis_generator" already exists.
Nothing changed.  Job "__clean_abandoned_branches_multibranch" already exists.
Nothing changed.  View "Welcome" already exists.
Nothing changed.  View "GitHub Organizations" already exists.
Nothing changed.  View "Maintenance" already exists.
Nothing changed.  Primary view already set to "Welcome".
Nothing changed.  Markup Formatter already configured.
Nothing changed.  Docker agent credentials already configured.
Nothing changed.  Agent -> master security already disabled.
Nothing changed.  Job DSL script security already disabled.
Nothing changed.  Agent Protocols already configured: [JNLP4-connect, Ping]
Nothing changed.  Global Jenkinsfile already configured.
Nothing changed.  Global Jenkinsfile script already approved.
Nothing changed.  Master already restricts jobs.
Nothing changed.  Admin groovy scripts already approved.
Nothing changed.  Grape repository config used by @Grab already configured.
Nothing changed.  Jenkins settings already configured.
Nothing changed.  github-token credentials already configured.
Nothing changed.  github-user-and-token credentials already configured.
Nothing changed.  webhook-shared-secret credentials already configured.
Nothing changed.  Pipeline Global Shared Libraries already configured.
Nothing changed.  GitHub plugin already configured.
Nothing changed.  Already using random least loaded provisioning strategy for docker.
Configured Yet Another Docker cloud docker-local
Nothing changed.  GitHub security realm already configured.
Jenkins is ready.  Visit http://localhost:8080/
User: samrocketman

real	0m3.805s
user	0m1.310s
sys	0m0.393s

artem-panchenko · 2018-04-18T09:30:43Z

Thank you for the support of the PR! I'm very glad that the patch has got into master!

P.S. do you guys know when a new version of the plugin is going to be released? :)

samrocketman · 2018-04-18T14:13:03Z

Looking at merging one more PR before releasing. But if it's not prepared in a week or so I can cut a new release and let it go to the next. I try to make it so that a release has a couple improvements. But sometimes it only has one improvement 😄 .

vrenjith · 2018-11-29T16:38:47Z

Any updates on when we are releasing this?

samrocketman · 2018-12-07T07:29:50Z

@artem-panchenko @vrenjith github-oauth 0.31 was just released which is the first release that includes this patch.

mihir-io · 2019-01-18T14:28:06Z

@artem-panchenko @samrocketman Is there any way to add the option to configure the cache expiration time?

jamesrwhite · 2020-06-26T11:46:47Z

Hi, is there anyway to force this cache to be cleared? I feel this should be mentioned on the wiki as well really as it took me a while to discover this caching even existed.

My use case is I'm using matrix auth to set permissions based on the teams a user is in. If they try and access Jenkins whilst not in the required team, then I add them to that team, they then have to wait an hour before this change takes effect which isn't ideal.

js-timbirkett · 2020-07-08T15:05:31Z

@jamesrwhite - we've hit this too and it's a bit frustrating... Tried:

import org.jenkinsci.plugins.*
GithubAuthenticationToken.clearCaches()

In the script console but it didn't seem to resolve the issue. @mihir-io - cache configuration would be a dream come true.

samrocketman · 2020-07-16T04:31:44Z

@jamesrwhite @js-timbirkett
Here is how I do it in my code base. It works with Job DSL code. It's the first code I run before detecting admin permissions.

/**
  * https://github.com/samrocketman MIT License
  * Clear the authorization cache so that we can immediately discover new
  * permissions for generating a job.  This will wipe and create a fresh cache
  * when a user generates a job.
  *
  * Source code paths
  *     GitHubClientCacheOps https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github/internal/GitHubClientCacheOps.java
  *     GitHubLoginFunction  https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github/internal/GitHubLoginFunction.java
  *     GitHubPluginConfig   https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github/config/GitHubPluginConfig.java
  *     GitHubServerConfig   https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github/config/GitHubServerConfig.java
  *
  * Additional Information:
  *     Usage of uberClassLoader and getExtensionList are because of
  *     limitations in the Job DSL plugin.  These are helper functions provided
  *     by Jenkins core which helps work around classloader limitations (i.e.
  *     classes not available to import)
  */
void wipeAuthorizationCache() {
    Class github_cache_ops = Jenkins.instance.pluginManager.uberClassLoader.loadClass('org.jenkinsci.plugins.github.internal.GitHubClientCacheOps')
    List github_server_configs = Jenkins.instance.getExtensionList('org.jenkinsci.plugins.github.config.GitHubPluginConfig')[0].configs
    github_server_configs.each { github_server_config ->
        github_cache_ops.toCacheDir().apply(github_server_config).delete()
        github_server_config.setCachedClient(null)
        def github_login_function = Jenkins.instance.pluginManager.uberClassLoader.loadClass('org.jenkinsci.plugins.github.internal.GitHubLoginFunction').newInstance()
        github_server_config.setCachedClient(github_login_function.apply(github_server_config))
    }
}
wipeAuthorizationCache()

And here's an example checking for admin access (for example webhook registration).

import com.cloudbees.jenkins.GitHubRepositoryName

// example dummy values for https://github.com/jenkinsci/github-oauth-plugin
String host = 'github.com'
String namespace = 'jenkinsci'
String repo = 'github-oauth-plugin'

if(!(new GitHubRepositoryName(host, namespace, repo)).resolveOne()?.hasAdminAccess()) {
    throw new Exception('''
    |ERROR: please make Jenkins an admin of your repository.  Otherwise, webhooks can't be automatically registered.
    '''.stripMargin().trim())
}

samrocketman · 2020-07-16T04:36:49Z

@artem-panchenko I'm not sure why clearCaches() doesn't work. It didn't for me when I rolled it out so this is my workaround for now. Any idea why it doesn't work to clear the cache?

samrocketman requested a review from Wadeck April 10, 2018 16:31

samrocketman added the code review requested label Apr 10, 2018

Wadeck approved these changes Apr 11, 2018

View reviewed changes

samrocketman added candidate to merge and removed code review requested labels Apr 11, 2018

samrocketman previously requested changes Apr 11, 2018

View reviewed changes

samrocketman added the feedback needed label Apr 11, 2018

Cache User (1 min) and Teams (1 hour) information

76623ea

artem-panchenko force-pushed the apanchenko/add-more-caching branch from cc4ad08 to 76623ea Compare April 12, 2018 20:15

artem-panchenko changed the title ~~[WIP] Cache User (1 min) and Teams (1 hour) information~~ Cache User (1 min) and Teams (1 hour) information Apr 12, 2018

samrocketman removed the feedback needed label Apr 18, 2018

samrocketman approved these changes Apr 18, 2018

View reviewed changes

samrocketman merged commit 715be7a into jenkinsci:master Apr 18, 2018

Uh oh!

Conversation

artem-panchenko commented Mar 26, 2018

Changes

Rationale

Note

Call for HELP!

Uh oh!

samrocketman commented Apr 10, 2018

Uh oh!

samrocketman commented Apr 10, 2018

Uh oh!

Wadeck left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wadeck Apr 11, 2018

Choose a reason for hiding this comment

Uh oh!

samrocketman Apr 11, 2018

Choose a reason for hiding this comment

Uh oh!

samrocketman Apr 11, 2018

Choose a reason for hiding this comment

Uh oh!

samrocketman commented Apr 11, 2018

Uh oh!

Wadeck commented Apr 11, 2018

Uh oh!

samrocketman commented Apr 11, 2018

Uh oh!

artem-panchenko commented Apr 12, 2018

Uh oh!

samrocketman commented Apr 13, 2018

Uh oh!

samrocketman left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Approved

How did I build?

Before

After

Uh oh!

artem-panchenko commented Apr 18, 2018

Uh oh!

samrocketman commented Apr 18, 2018

Uh oh!

vrenjith commented Nov 29, 2018

Uh oh!

samrocketman commented Dec 7, 2018

Uh oh!

mihir-io commented Jan 18, 2019

Uh oh!

jamesrwhite commented Jun 26, 2020

Uh oh!

js-timbirkett commented Jul 8, 2020

Uh oh!

samrocketman commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samrocketman commented Jul 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Wadeck left a comment •

edited

Loading

samrocketman left a comment •

edited

Loading

samrocketman commented Jul 16, 2020 •

edited

Loading