Skip to content

Commit 899c56a

Browse files
jotojoto
committed
Add support for data generalization
This large commit adds a framework to generalize OSM data using different strategies and implements several strategies. The source is all in the new `src/gen` directory. It includes a new command `osm2pgsql-gen` to access this functionality. Future versions might integrate it with the `osm2pgsql` command, but because this is all experimental it is kept separate for now. Call `osm2pgsql-gen` with `-h` to get usage help. See also https://osm2pgsql.org/generalization/ and the chapter on Generalization in the manual: https://osm2pgsql.org/doc/manual.html#generalization For the raster support this adds two new library dependency: CImg and potrace. Both have been around for a long time and are readily available on all systems. The following strategies work on a tile-by-tile basis and operate on polygons: The "vector-union" strategy buffers and unionizes polygons using vector operations. The "raster-union" strategy does a similar thing but does it in raster space which is much faster. First the polygons are rendered into a raster, an open/close operation is called (which basically does the same thing as the buffering in vector space) and finally the resulting raster is vectorized again. The "builtup" strategy is intended to derive a layer of builtup areas from landuse=residential/industrial etc. as well as building cover and dense road networks. The following strategies always work on all data (not tile-based): The "discrete-isolation" strategy rates places based on some importance metric to get a more even distribution of places on the map. The new "rivers" strategy finds important rivers, this is still very much work in progress.
1 parent 344e9d8 commit 899c56a

34 files changed

Lines changed: 3361 additions & 5 deletions

‎.github/actions/ubuntu-prerequisites/action.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,12 @@ runs:
1616
- name: Install software
1717
run: |
1818
sudo apt-get install -yq --no-install-suggests --no-install-recommends \
19+
cimg-dev \
1920
libboost-filesystem-dev \
2021
libboost-system-dev \
2122
libbz2-dev \
2223
libexpat1-dev \
24+
libpotrace-dev \
2325
libpq-dev \
2426
libproj-dev \
2527
pandoc \

‎.github/actions/win-install/action.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,8 @@ runs:
55

66
steps:
77
- name: Install packages
8-
run: vcpkg install bzip2:x64-windows expat:x64-windows zlib:x64-windows proj4:x64-windows boost-geometry:x64-windows boost-system:x64-windows boost-filesystem:x64-windows boost-property-tree:x64-windows lua:x64-windows libpq:x64-windows
8+
run: vcpkg install cimg:x64-windows bzip2:x64-windows expat:x64-windows zlib:x64-windows proj4:x64-windows boost-geometry:x64-windows boost-system:x64-windows boost-filesystem:x64-windows boost-property-tree:x64-windows lua:x64-windows libpq:x64-windows
99
shell: bash
10-
1110
- name: Install psycopg2 and beahve
1211
run: python -m pip install psycopg2 behave
1312
shell: bash

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414

1515
- name: Install prerequisites
1616
run: |
17-
brew install lua boost postgis pandoc
17+
brew install lua boost postgis pandoc cimg potrace
1818
pip3 install psycopg2 behave
1919
pg_ctl -D /usr/local/var/postgres init
2020
pg_ctl -D /usr/local/var/postgres start
@@ -45,6 +45,7 @@ jobs:
4545
env:
4646
CC: gcc-10
4747
CXX: g++-10
48+
EXTRA_FLAGS: -Wno-unused-but-set-parameter # workaround for GCC bug
4849
LUA_VERSION: 5.3
4950
LUAJIT_OPTION: ON
5051
POSTGRESQL_VERSION: 9.6
@@ -79,6 +80,7 @@ jobs:
7980
env:
8081
CC: gcc-10
8182
CXX: g++-10
83+
EXTRA_FLAGS: -Wno-unused-but-set-parameter # workaround for GCC bug
8284
LUA_VERSION: 5.3
8385
LUAJIT_OPTION: OFF
8486
POSTGRESQL_VERSION: 10

‎.github/workflows/test-install.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,14 @@ jobs:
3737
run: |
3838
sudo apt-get purge -yq postgresql*
3939
sudo apt-get install -yq --no-install-suggests --no-install-recommends \
40+
cimg-dev \
4041
libboost-filesystem-dev \
4142
libboost-system-dev \
4243
libbz2-dev \
4344
libexpat1-dev \
4445
liblua${LUA_VERSION}-dev \
4546
libluajit-5.1-dev \
47+
libpotrace-dev \
4648
libpq-dev \
4749
libproj-dev \
4850
lua${LUA_VERSION} \

‎CMakeLists.txt‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,11 @@ include_directories(SYSTEM ${PostgreSQL_INCLUDE_DIRS})
205205
206206
find_package(Threads)
207207
208+
find_path(POTRACE_INCLUDE_DIR potracelib.h)
209+
find_library(POTRACE_LIBRARY NAMES potrace)
210+
211+
find_path(CIMG_INCLUDE_DIR CImg.h)
212+
208213
############### Libraries are found now ########################
209214
210215
set(LIBS ${Boost_LIBRARIES} ${PostgreSQL_LIBRARY} ${OSMIUM_LIBRARIES})
@@ -276,6 +281,33 @@ add_subdirectory(src)
276281
add_executable(osm2pgsql src/osm2pgsql.cpp)
277282
target_link_libraries(osm2pgsql osm2pgsql_lib ${LIBS})
278283
284+
if (${POTRACE_LIBRARY} STREQUAL "POTRACE_LIBRARY-NOTFOUND" OR ${CIMG_INCLUDE_DIR} STREQUAL "CIMG_INCLUDE_DIR-NOTFOUND")
285+
message(STATUS "Did not find cimg and/or potrace library. Not building osm2pgsql-gen.")
286+
else()
287+
if (WITH_LUA)
288+
message(STATUS "Found cimg and potrace library. Building osm2pgsql-gen.")
289+
set(BUILD_GEN 1)
290+
include_directories(SYSTEM ${CIMG_INCLUDE_DIR})
291+
include_directories(SYSTEM ${POTRACE_INCLUDE_DIR})
292+
add_executable(osm2pgsql-gen src/gen/osm2pgsql-gen.cpp
293+
src/gen/canvas.cpp
294+
src/gen/gen-base.cpp
295+
src/gen/gen-create.cpp
296+
src/gen/gen-discrete-isolation.cpp
297+
src/gen/gen-rivers.cpp
298+
src/gen/gen-tile-builtup.cpp
299+
src/gen/gen-tile-raster.cpp
300+
src/gen/gen-tile-vector.cpp
301+
src/gen/gen-tile.cpp
302+
src/gen/params.cpp
303+
src/gen/raster.cpp
304+
src/gen/tracer.cpp)
305+
target_link_libraries(osm2pgsql-gen osm2pgsql_lib ${LIBS} ${POTRACE_LIBRARY})
306+
else()
307+
message(STATUS "No Lua. Not building osm2pgsql-gen.")
308+
endif()
309+
endif()
310+
279311
#############################################################
280312
# Optional "clang-tidy" target
281313
#############################################################
@@ -287,7 +319,7 @@ find_program(CLANG_TIDY
287319
if (CLANG_TIDY)
288320
message(STATUS "Looking for clang-tidy - found ${CLANG_TIDY}")
289321
290-
file(GLOB CT_CHECK_FILES src/*.cpp tests/*cpp)
322+
file(GLOB CT_CHECK_FILES src/*.cpp src/*/*.cpp tests/*cpp)
291323
292324
add_custom_target(clang-tidy
293325
${CLANG_TIDY}

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ Required libraries are
4949
* [zlib](https://www.zlib.net/)
5050
* [Boost libraries](https://www.boost.org/), including geometry, system and
5151
filesystem
52+
* [CImg](https://cimg.eu/) (Optional, see README-gen.md)
53+
* [potrace](https://potrace.sourceforge.net/) (Optional, see README-gen.md)
5254
* [PostgreSQL](https://www.postgresql.org/) client libraries
5355
* [Lua](https://www.lua.org/) (Optional, used for Lua tag transforms
5456
and the flex output)
@@ -80,14 +82,15 @@ On a Debian or Ubuntu system, this can be done with:
8082

8183
```sh
8284
sudo apt-get install make cmake g++ libboost-dev libboost-system-dev \
83-
libboost-filesystem-dev libexpat1-dev zlib1g-dev \
85+
libboost-filesystem-dev libexpat1-dev zlib1g-dev libpotrace-dev cimg-dev \
8486
libbz2-dev libpq-dev libproj-dev lua5.3 liblua5.3-dev pandoc
8587
```
8688

8789
On a Fedora system, use
8890

8991
```sh
9092
sudo dnf install cmake make gcc-c++ boost-devel expat-devel zlib-devel \
93+
potrace-devel cimg-devel \
9194
bzip2-devel postgresql-devel proj-devel proj-epsg lua-devel pandoc
9295
```
9396

‎docs/CMakeLists.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,17 @@ if(PANDOC)
2525
VERBATIM)
2626

2727
list(APPEND MANPAGE_TARGETS osm2pgsql.1)
28+
29+
if(BUILD_GEN)
30+
add_custom_command(OUTPUT osm2pgsql-gen.1
31+
COMMAND ${PANDOC} ${PANDOC_MAN_OPTIONS} -o osm2pgsql-gen.1
32+
${CMAKE_CURRENT_SOURCE_DIR}/osm2pgsql-gen.md
33+
DEPENDS osm2pgsql-gen.md manpage.template
34+
COMMENT "Building manpage osm2pgsql-gen.1"
35+
VERBATIM)
36+
list(APPEND MANPAGE_TARGETS osm2pgsql-gen.1)
37+
endif()
38+
2839
else()
2940
message(STATUS "Looking for pandoc - not found")
3041
message(STATUS " osm2pgsql manual page can not be built")

‎docs/osm2pgsql-gen.1‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
.TH "OSM2PGSQL" "1" "1.8.1" "" ""
2+
.SH NAME
3+
.PP
4+
osm2pgsql-gen - Generalize OpenStreetMap data - EXPERIMENTAL!
5+
.SH SYNOPSIS
6+
.PP
7+
\f[B]osm2pgsql-gen\f[R] [\f[I]OPTIONS\f[R]]\&...
8+
.SH DESCRIPTION
9+
.PP
10+
THIS PROGRAM IS EXPERIMENTAL AND MIGHT CHANGE WITHOUT NOTICE!
11+
.PP
12+
\f[B]osm2pgsql-gen\f[R] reads data imported by \f[B]osm2pgsql\f[R] from
13+
the database, performs various generalization steps specified by a Lua
14+
config file and writes the data back to the database.
15+
It is used in conjunction with and after \f[B]osm2pgsql\f[R] and reads
16+
the same config file.
17+
.PP
18+
This man page can only cover some of the basics and describe the command
19+
line options.
20+
See the Generalization chapter in the osm2pgsql
21+
Manual (https://osm2pgsql.org/doc/manual.html#generalization) for more
22+
information.
23+
.SH OPTIONS
24+
.PP
25+
This program follows the usual GNU command line syntax, with long
26+
options starting with two dashes (\f[C]--\f[R]).
27+
Mandatory arguments to long options are mandatory for short options too.
28+
.SH MAIN OPTIONS
29+
.TP
30+
-a, --append
31+
Run in append mode.
32+
.TP
33+
-c, --create
34+
Run in create mode.
35+
This is the default if \f[B]-a, --append\f[R] is not specified.
36+
.TP
37+
-S, --style=FILE
38+
The Lua config file.
39+
Same as for \f[B]osm2pgsql\f[R].
40+
.TP
41+
-j, -jobs=NUM
42+
Specifies the number of parallel threads used for certain operations.
43+
Setting this to the number of available CPU cores is a reasonable
44+
starting point.
45+
.SH HELP/VERSION OPTIONS
46+
.TP
47+
-h, --help
48+
Print help.
49+
.TP
50+
-V, --version
51+
Print osm2pgsql version.
52+
.SH LOGGING OPTIONS
53+
.TP
54+
--log-level=LEVEL
55+
Set log level (`debug', `info' (default), `warn', or `error').
56+
.TP
57+
--log-sql
58+
Enable logging of SQL commands for debugging.
59+
.SH DATABASE OPTIONS
60+
.TP
61+
-d, --database=NAME
62+
The name of the PostgreSQL database to connect to.
63+
If this parameter contains an \f[C]=\f[R] sign or starts with a valid
64+
URI prefix (\f[C]postgresql://\f[R] or \f[C]postgres://\f[R]), it is
65+
treated as a conninfo string.
66+
See the PostgreSQL manual for details.
67+
.TP
68+
-U, --username=NAME
69+
Postgresql user name.
70+
.TP
71+
-W, --password
72+
Force password prompt.
73+
.TP
74+
-H, --host=HOSTNAME
75+
Database server hostname or unix domain socket location.
76+
.TP
77+
-P, --port=PORT
78+
Database server port.
79+
.SH SEE ALSO
80+
.IP \[bu] 2
81+
osm2pgsql website (https://osm2pgsql.org)
82+
.IP \[bu] 2
83+
osm2pgsql manual (https://osm2pgsql.org/doc/manual.html)
84+
.IP \[bu] 2
85+
\f[B]postgres\f[R](1)
86+
.IP \[bu] 2
87+
\f[B]osm2pgsql\f[R](1)

‎docs/osm2pgsql-gen.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# NAME
2+
3+
osm2pgsql-gen - Generalize OpenStreetMap data - EXPERIMENTAL!
4+
5+
# SYNOPSIS
6+
7+
**osm2pgsql-gen** \[*OPTIONS*\]...
8+
9+
# DESCRIPTION
10+
11+
THIS PROGRAM IS EXPERIMENTAL AND MIGHT CHANGE WITHOUT NOTICE!
12+
13+
**osm2pgsql-gen** reads data imported by **osm2pgsql** from the database,
14+
performs various generalization steps specified by a Lua config file and
15+
writes the data back to the database. It is used in conjunction with and
16+
after **osm2pgsql** and reads the same config file.
17+
18+
This man page can only cover some of the basics and describe the command line
19+
options. See the [Generalization chapter in the osm2pgsql
20+
Manual](https://osm2pgsql.org/doc/manual.html#generalization) for more
21+
information.
22+
23+
# OPTIONS
24+
25+
This program follows the usual GNU command line syntax, with long options
26+
starting with two dashes (`--`). Mandatory arguments to long options are
27+
mandatory for short options too.
28+
29+
# MAIN OPTIONS
30+
31+
-a, \--append
32+
: Run in append mode.
33+
34+
-c, \--create
35+
: Run in create mode. This is the default if **-a, \--append** is not
36+
specified.
37+
38+
-S, \--style=FILE
39+
: The Lua config file. Same as for **osm2pgsql**.
40+
41+
-j, \-jobs=NUM
42+
: Specifies the number of parallel threads used for certain operations.
43+
Setting this to the number of available CPU cores is a reasonable starting
44+
point.
45+
46+
# HELP/VERSION OPTIONS
47+
48+
-h, \--help
49+
: Print help.
50+
51+
-V, \--version
52+
: Print osm2pgsql version.
53+
54+
# LOGGING OPTIONS
55+
56+
\--log-level=LEVEL
57+
: Set log level ('debug', 'info' (default), 'warn', or 'error').
58+
59+
\--log-sql
60+
: Enable logging of SQL commands for debugging.
61+
62+
# DATABASE OPTIONS
63+
64+
-d, \--database=NAME
65+
: The name of the PostgreSQL database to connect to. If this parameter
66+
contains an `=` sign or starts with a valid URI prefix (`postgresql://` or
67+
`postgres://`), it is treated as a conninfo string. See the PostgreSQL
68+
manual for details.
69+
70+
-U, \--username=NAME
71+
: Postgresql user name.
72+
73+
-W, \--password
74+
: Force password prompt.
75+
76+
-H, \--host=HOSTNAME
77+
: Database server hostname or unix domain socket location.
78+
79+
-P, \--port=PORT
80+
: Database server port.
81+
82+
# SEE ALSO
83+
84+
* [osm2pgsql website](https://osm2pgsql.org)
85+
* [osm2pgsql manual](https://osm2pgsql.org/doc/manual.html)
86+
* **postgres**(1)
87+
* **osm2pgsql**(1)
88+

0 commit comments

Comments
 (0)