Skip to content

Commit 20fc8fb

Browse files
matthiaskmatthiask
committed
Initial import
0 parents  commit 20fc8fb

File tree

21 files changed

+642
-0
lines changed

21 files changed

+642
-0
lines changed

‎.editorconfig‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# top-most EditorConfig file
2+
root = true
3+
4+
[*]
5+
end_of_line = lf
6+
insert_final_newline = true
7+
charset = utf-8
8+
trim_trailing_whitespace = true
9+
indent_style = space
10+
indent_size = 2
11+
12+
[*.py]
13+
indent_size = 4

‎.github/workflows/tests.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
9+
jobs:
10+
tests:
11+
name: Python ${{ matrix.python-version }}
12+
runs-on: ubuntu-latest
13+
strategy:
14+
fail-fast: false
15+
matrix:
16+
python-version:
17+
- 3.9
18+
19+
steps:
20+
- uses: actions/checkout@v2
21+
- name: Set up Python ${{ matrix.python-version }}
22+
uses: actions/setup-python@v2
23+
with:
24+
python-version: ${{ matrix.python-version }}
25+
- name: Install dependencies
26+
run: |
27+
python -m pip install --upgrade pip wheel setuptools tox
28+
- name: Run tox targets for ${{ matrix.python-version }}
29+
run: |
30+
ENV_PREFIX=$(tr -C -d "0-9" <<< "${{ matrix.python-version }}")
31+
TOXENV=$(tox --listenvs | grep "^py$ENV_PREFIX" | tr '\n' ',') python -m tox
32+
33+
lint:
34+
name: Lint
35+
runs-on: ubuntu-latest
36+
steps:
37+
- uses: actions/checkout@v2
38+
- name: Set up Python
39+
uses: actions/setup-python@v2
40+
with:
41+
python-version: 3.9
42+
- name: Install dependencies
43+
run: |
44+
python -m pip install --upgrade pip tox
45+
- name: Run lint
46+
run: tox -e style

‎.gitignore‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
*.py?
2+
*~
3+
*.sw?
4+
\#*#
5+
/secrets.py
6+
.DS_Store
7+
._*
8+
*.egg-info
9+
/MANIFEST
10+
/_build
11+
/build
12+
dist
13+
tests/test.zip
14+
/docs/_build
15+
/.eggs
16+
.coverage
17+
htmlcov
18+
venv
19+
.tox

‎CHANGELOG.rst‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
==========
2+
Change log
3+
==========
4+
5+
`Next version`_
6+
~~~~~~~~~~~~~~~
7+
8+
`0.1`_ (2021-09-27)
9+
~~~~~~~~~~~~~~~~~~~
10+
11+
- Initial release!
12+
13+
.. _0.1: https://github.com/matthiask/feincms3-data/commit/e50451b5661
14+
.. _1.1: https://github.com/matthiask/feincms3-data/compare/1.0...1.1
15+
.. _Next version: https://github.com/matthiask/feincms3-data/compare/3.0...master

‎LICENSE‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Copyright (c) 2021, Feinheit AG and individual contributors.
2+
All rights reserved.
3+
4+
Redistribution and use in source and binary forms, with or without modification,
5+
are permitted provided that the following conditions are met:
6+
7+
1. Redistributions of source code must retain the above copyright notice,
8+
this list of conditions and the following disclaimer.
9+
10+
2. Redistributions in binary form must reproduce the above copyright
11+
notice, this list of conditions and the following disclaimer in the
12+
documentation and/or other materials provided with the distribution.
13+
14+
3. Neither the name of Feinheit AG nor the names of its contributors
15+
may be used to endorse or promote products derived from this software
16+
without specific prior written permission.
17+
18+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
19+
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
20+
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
22+
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
23+
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
24+
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
25+
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
27+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎MANIFEST.in‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
include LICENSE
2+
include MANIFEST.in
3+
include README.rst
4+
recursive-include feincms3_data/static *
5+
recursive-include feincms3_data/locale *
6+
recursive-include feincms3_data/templates *

‎README.rst‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
=============
2+
feincms3-data
3+
=============
4+
5+
.. image:: https://github.com/matthiask/feincms3-data/actions/workflows/tests.yml/badge.svg
6+
:target: https://github.com/matthiask/feincms3-data/
7+
:alt: CI Status
8+
9+
10+
Why
11+
===
12+
13+
Utilities for loading and dumping database data as JSON.
14+
15+
These utilities (partially) replace Django's built-in ``dumpdata`` and
16+
``loaddata`` management commands.
17+
18+
Suppose you want to move data between systems incrementally. In this case it
19+
isn't sufficient to only know the data which has been created or updated; you
20+
also want to know which data has been deleted in the meantime. Django's
21+
``dumpdata`` and ``loaddata`` management commands only support the former case,
22+
not the latter. They also do not including dependent objects in the dump.
23+
24+
This package offers utilities and management commands to address these
25+
shortcomings.
26+
27+
28+
How
29+
===
30+
31+
``pip install feincms3-data``.
32+
33+
Add ``feincms3-data`` to ``INSTALLED_APPS`` so that the included management
34+
commands are discovered.
35+
36+
Add specs somewhere describing the models and relationships you want to dump,
37+
e.g. in a module named ``app.specs``:
38+
39+
.. code-block:: python
40+
41+
from feincms3_data.data import (
42+
specs_for_app_models,
43+
specs_for_derived_models,
44+
specs_for_models,
45+
)
46+
from app.dashboard import models as dashboard_models
47+
from app.world import models as world_models
48+
49+
50+
def districts(args):
51+
pks = [int(arg) for arg in args.split(",") if arg]
52+
return [
53+
*specs_for_models(
54+
[world_models.District],
55+
{"filter": {"pk__in": pks}},
56+
),
57+
*specs_for_models(
58+
[world_models.Exercise],
59+
{"filter": {"district__in": pks}},
60+
),
61+
*specs_for_derived_models(
62+
world_models.ExercisePlugin,
63+
{"filter": {"parent__district__in": pks}},
64+
),
65+
]
66+
67+
68+
def specs():
69+
return {
70+
"articles": lambda args: specs_for_app_models("articles"),
71+
"pages": lambda args: specs_for_app_models("pages"),
72+
"teachingmaterials": lambda args: specs_for_models(
73+
[
74+
dashboard_models.TeachingMaterialGroup,
75+
dashboard_models.TeachingMaterial,
76+
]
77+
),
78+
"districts": districts,
79+
}
80+
81+
Add a setting with the Python module path to the specs function:
82+
83+
.. code-block:: python
84+
85+
FEINCMS3_DATA_SPECS = "app.specs.specs"
86+
87+
88+
Now, to dump e.g. pages and teachingmaterials you would run::
89+
90+
./manage.py f3dumpdata pages teachingmaterials > tmp/dump.json
91+
92+
To dump the districts with the primary key of 42 and 43 you would run::
93+
94+
./manage.py f3dumpdata districts:42,43 > tmp/districts.json
95+
96+
The resulting JSON file has three top-level keys:
97+
98+
- ``"version": 1``: The version of the dump, because not versioning dumps is a
99+
recipe for pain down the road.
100+
- ``"specs": [...]``: A list of model specs and filters (see the district
101+
filter above).
102+
- ``"objects": [...]``: A list of model instances; uses the same serializer as
103+
Django's ``dumpdata``, everything looks the same.
104+
105+
The dumps can be loaded back into the database by running:
106+
107+
./manage.py f3loaddata -v2 tmp/dump.json tmp/districts.json
108+
109+
Each dump is processed in an individual transaction. The data is first loaded
110+
into the database; at the end, data *matching* the filters but whose primary
111+
key wasn't contained in the dump is deleted from the database.

‎feincms3_data/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
VERSION = (0, 0, 1)
2+
__version__ = ".".join(map(str, VERSION))

‎feincms3_data/data.py‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
import io
2+
import json
3+
from collections import defaultdict
4+
from itertools import chain
5+
6+
from django.apps import apps
7+
from django.core import serializers
8+
from django.db import transaction
9+
10+
11+
def _all_subclasses(cls):
12+
for sc in cls.__subclasses__():
13+
yield sc
14+
yield from _all_subclasses(sc)
15+
16+
17+
def _only_concrete_models(iterable):
18+
for model in iterable:
19+
if not model._meta.abstract and model._meta.managed:
20+
yield model
21+
22+
23+
def specs_for_models(models, spec=None):
24+
spec = {} if spec is None else spec
25+
return (spec | {"model": cls._meta.label_lower} for cls in models)
26+
27+
28+
def specs_for_derived_models(cls, spec=None):
29+
return specs_for_models(_only_concrete_models(_all_subclasses(cls)), spec)
30+
31+
32+
def specs_for_app_models(app, spec=None):
33+
return specs_for_models(
34+
apps.get_app_config(app).get_models(include_auto_created=True), spec
35+
)
36+
37+
38+
def _model_queryset(spec):
39+
queryset = apps.get_model(spec["model"])._default_manager.all()
40+
if filter := spec.get("filter"):
41+
queryset = queryset.filter(**filter)
42+
return queryset
43+
44+
45+
def silence(*a):
46+
pass
47+
48+
49+
def dump_specs(specs):
50+
stream = io.StringIO()
51+
stream.write('{"version": 1, "specs": ')
52+
json.dump(specs, stream)
53+
stream.write(', "objects": ')
54+
serializers.serialize(
55+
"json",
56+
chain.from_iterable(_model_queryset(spec) for spec in specs),
57+
stream=stream,
58+
)
59+
stream.write("}")
60+
return stream.getvalue()
61+
62+
63+
def load_dump(data, *, progress=silence):
64+
assert data["version"] == 1
65+
66+
objects = defaultdict(list)
67+
seen_pks = defaultdict(set)
68+
69+
# Yes, that is a bit stupid
70+
for ds in serializers.deserialize("json", json.dumps(data["objects"])):
71+
objects[ds.object._meta.label_lower].append(ds)
72+
73+
progress(f"Loaded {len(data['objects'])} objects")
74+
75+
with transaction.atomic():
76+
for spec in data["specs"]:
77+
if objs := objects[spec["model"]]:
78+
for ds in objs:
79+
ds.save()
80+
seen_pks[ds.object._meta.label_lower].add(ds.object.pk)
81+
82+
progress(f"Saved {len(objs)} {spec['model']} objects")
83+
84+
for spec in data["specs"]:
85+
queryset = _model_queryset(spec)
86+
if deleted := queryset.exclude(pk__in=seen_pks[spec["model"]]).delete():
87+
progress(f"Deleted {spec['model']} objects: {deleted}")

‎feincms3_data/management/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)