[ENH] replaced hardcoded test server admin key with env variable and secrets (openml#1568)

rohansen856 · web-flow · commit aa04b30cc473 · 2026-02-15T16:44:59.000+01:00
#### Metadata * Reference Issue: openml#1529 * New Tests Added: Yes * Documentation Updated: No * Change Log Entry:  #### Details this PR is made to remove the hardcoded test server admin key from the codebase and replace it with environment variable-based authentication. ## Summary - in `openml/config.py` Added a new environment variable constant for the test server admin key: ```python OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR = "OPENML_TEST_SERVER_ADMIN_KEY" ``` - Testing Base Class Updated in `openml/testing.py`. Modified the `TestBase` class to read the admin key from an environment variable instead of using a hardcoded value: **Before**: ```python admin_key = "abc" ``` **After**: ```python admin_key = os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR) ``` **Note**: - The admin key is now `None` by default when the environment variable is not set - Tests requiring the admin key will fail gracefully if the key is not available Also in the tests, Added `pytest.skipif` decorators to tests that require admin privileges in the following test files: #### `tests/test_openml/test_config.py` **Test**: `test_switch_to_example_configuration` **Added decorator**: ```python @pytest.mark.skipif( not os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR), reason="Test requires admin key. Set OPENML_TEST_SERVER_ADMIN_KEY environment variable.", ) ``` #### `tests/test_datasets/test_dataset_functions.py` **Test**: `test_data_status` **Added decorator**: ```python @pytest.mark.skipif( not os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR), reason="Test requires admin key. Set OPENML_TEST_SERVER_ADMIN_KEY environment variable.", ) ``` **Note**: - These tests will be automatically skipped if the admin key is not provided - Clear skip reason is displayed when tests are skipped - No failures or errors when running tests locally without the admin key ### 4. CI Configuration Update (`.github/workflows/test.yml`) Added the environment variable to all test execution steps in the GitHub Actions workflow: **Steps updated**: - Run tests on Ubuntu Test - Run tests on Ubuntu Production - Run tests on Windows **Added to each step**: ```yaml env: OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }} ``` **Impact**: - CI will use the secret stored in GitHub repository settings - Tests requiring admin privileges will run in CI - The actual key value is never exposed in logs or code @PGijsbers this requires someone to put the admin key in the github secrets which would be a critical step. # Update on reviews the configurations should be done from openml config files in `./.openml/config` for directory level configurations, instead of the added responsibility of a new `.env` file and dependencies. in case of local testing the concerned tests would be skipped in case no key is provided.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -106,6 +106,8 @@ jobs:
 
     - name: Run tests on Ubuntu Test
       if: matrix.os == 'ubuntu-latest'
+      env:
+        OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}
       run: |
         if [ "${{ matrix.code-cov }}" = "true" ]; then
           codecov="--cov=openml --long --cov-report=xml"
@@ -121,6 +123,8 @@ jobs:
 
     - name: Run tests on Ubuntu Production
       if: matrix.os == 'ubuntu-latest'
+      env:
+        OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}
       run: |
         if [ "${{ matrix.code-cov }}" = "true" ]; then
           codecov="--cov=openml --long --cov-report=xml"
@@ -136,6 +140,8 @@ jobs:
 
     - name: Run tests on Windows
       if: matrix.os == 'windows-latest'
+      env:
+        OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}
       run: |  # we need a separate step because of the bash-specific if-statement in the previous one.
         pytest -n 4 --durations=20 --dist load -sv --reruns 5 --reruns-delay 1 -m "not uses_test_server"
 
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -96,6 +96,17 @@ To test your new contribution, add [unit tests](https://github.com/openml/openml
 * Please ensure that the example is run on the test server by beginning with the call to `openml.config.start_using_configuration_for_example()`, which is done by default for tests derived from `TestBase`.
 * Add the `@pytest.mark.sklearn` marker to your unit tests if they have a dependency on scikit-learn.
 
+#### Running Tests That Require Admin Privileges
+
+Some tests require admin privileges on the test server and will be automatically skipped unless you provide an admin API key. For regular contributors, the tests will skip gracefully. For core contributors who need to run these tests locally, you can set up the key by exporting the variable as below before running the tests:
+
+```bash
+# For windows
+$env:OPENML_TEST_SERVER_ADMIN_KEY = "admin-key"
+# For linux/mac
+export OPENML_TEST_SERVER_ADMIN_KEY="admin-key"
+```
+
 ### Pull Request Checklist
 
 You can go to the `openml-python` GitHub repository to create the pull request by [comparing the branch](https://github.com/openml/openml-python/compare) from your fork with the `develop` branch of the `openml-python` repository. When creating a pull request, make sure to follow the comments and structured provided by the template on GitHub.
diff --git a/openml/config.py b/openml/config.py
@@ -25,6 +25,7 @@
 
 OPENML_CACHE_DIR_ENV_VAR = "OPENML_CACHE_DIR"
 OPENML_SKIP_PARQUET_ENV_VAR = "OPENML_SKIP_PARQUET"
+OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR = "OPENML_TEST_SERVER_ADMIN_KEY"
 _TEST_SERVER_NORMAL_USER_KEY = "normaluser"
 
 
diff --git a/openml/testing.py b/openml/testing.py
@@ -48,7 +48,7 @@ class TestBase(unittest.TestCase):
     }
     flow_name_tracker: ClassVar[list[str]] = []
     test_server = "https://test.openml.org/api/v1/xml"
-    admin_key = "abc"
+    admin_key = os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR)
     user_key = openml.config._TEST_SERVER_NORMAL_USER_KEY
 
     # creating logger for tracking files uploaded to test server
diff --git a/tests/test_datasets/test_dataset_functions.py b/tests/test_datasets/test_dataset_functions.py
@@ -599,6 +599,10 @@ def _assert_status_of_dataset(self, *, did: int, status: str):
         assert len(result) == 1
         assert result[did]["status"] == status
 
+    @pytest.mark.skipif(
+        not os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR),
+        reason="Test requires admin key. Set OPENML_TEST_SERVER_ADMIN_KEY environment variable.",
+    )
     @pytest.mark.flaky()
     @pytest.mark.uses_test_server()
     def test_data_status(self):
diff --git a/tests/test_openml/test_config.py b/tests/test_openml/test_config.py
@@ -110,7 +110,7 @@ class TestConfigurationForExamples(openml.testing.TestBase):
     def test_switch_to_example_configuration(self):
         """Verifies the test configuration is loaded properly."""
         # Below is the default test key which would be used anyway, but just for clarity:
-        openml.config.apikey = TestBase.admin_key
+        openml.config.apikey = "any-api-key"
         openml.config.server = self.production_server
 
         openml.config.start_using_configuration_for_example()

Original file line number	Diff line number	Diff line change
`@@ -48,7 +48,7 @@ class TestBase(unittest.TestCase):`
`48`	`48`	`}`
`49`	`49`	`flow_name_tracker: ClassVar[list[str]] = []`
`50`	`50`	`test_server = "https://test.openml.org/api/v1/xml"`
`51`		`- admin_key = "abc"`
	`51`	`+ admin_key = os.environ.get(openml.config.OPENML_TEST_SERVER_ADMIN_KEY_ENV_VAR)`
`52`	`52`	`user_key = openml.config._TEST_SERVER_NORMAL_USER_KEY`
`53`	`53`
`54`	`54`	`# creating logger for tracking files uploaded to test server`