Introduction

skyway is a command-line OpenStreetMap file converter.

Goals

• Support a wide range of OpenStreetMap file formats
• Provide a user-friendly tag filtering system, capable of scaling to planet-scale data
• Perform conversions quickly, parallelizing computation where possible

Alternatives

Before adopting skyway for your project, please also consider Osmium Tool, a well-trusted and performant application that accomplishes many of the same things.

For a complete list of alternatives, please see the OSM file formats page on the OpenStreetMap wiki.

Installation

Pre-compiled Binaries

Builds of skyway are available on the releases page. Feel free to get in touch if you'd like me to prepare a binary for your platform.

Install Using Cargo

skyway is on crates.io, so you can install it using cargo:

cargo install skyway

Make sure your cargo bin directory is in your $PATH.

Selecting Features

Cargo "features" provide a mechanism to express conditional compilation and optional dependencies. A package defines a set of named features in the [features] table of Cargo.toml, and each feature can either be enabled or disabled. Features for the package being built can be enabled on the command-line with flags such as --features. Features for dependencies can be enabled in the dependency declaration in Cargo.toml.

―The Cargo Book

skyway's Cargo.toml file defines features which allow you to selectively compile certain file or filter formats. By default, all features are enabled. As well as providing a feature name for each individual file and filter format, filtering support can be enabled/disabled using the "filter" feature. Enabling the "cel" feature, for example, will automatically enable "filter", but you can enable "filter" by itself to access generic filtering functionality. Disabling all features leaves a relatively small library of OpenStreetMap conversion tools.

Usage

Here is an example of how to use skyway. For more information, you can run skyway --help.

skyway --from pbf --input input-file.pbf --to json --output output-file.json

If you do not specify an input or output file, skyway will default to standard in and standard out, respectively. This enables you to stream data into and out of skyway, like this:

cat input-file.pbf | skyway --from pbf --to json > output-file.json

Supported Formats

Here is a table showing the formats skyway supports reading and writing. The shortname is used to indicate the format when running skyway, e.g. skyway --from xml .... When paired together, "speedy" readers and writers will generally run faster by passing data between threads.

Key:
❌ – Not Supported
✅ – Supported
⚡ – Speedy!

^†Use the shortname json to read OSM JSON, it is the same parser. Use overpass for writing.

Filtering

A skyway filter decides whether or not to keep an element, or possibly transforms it. If you do not provide a filter to skyway, it will convert the original input as faithfully as it can.

Running skyway with a Filter

To add a filter to skyway, add the --filter [FILTER FILE] option. You may pass multiple filters to evaluate in sequence by passing multiple --filter flags. The file extension does not matter; skyway will detect if the file is in CEL or SkyFilter and parse it accordingly.

How Filters Work

In skyway, the input reader and output writer are run in different threads, with the former passing element objects to the latter as they become available. This multithreaded approach makes it easy to add one or more middlemen, who can receive elements from the reader, transform them, and then pass them along down the chain. skyway provides a "filtering" system to make it easier to add these processing steps in a modular way. Using any number of CEL or SkyFilter expressions, you can create complex processing pipelines.

This system was inspired by Pandoc.

CEL Filters

skyway supports the Common Expression Language (CEL) for filtering elements. Each time the filter is evaluated for a given element, that expression's context (local variables) is updated to match the element's metadata. For now, CEL filters may only return a boolean value, indicating whether or not the element should be kept. Please file an issue if you'd like to see more complex CEL return types supported.

CEL Context

The following table describes each variable available to your expression:

SkyFilter

SkyFilter is a bespoke filtering language designed for skyway, allowing you to transform element data as they pass through skyway.

Specification

An SkyFilter file must start with a header as shown below, with the version matching the version of skyway that you are using, followed by at least two newlines. skyway will warn you if there is a version mismatch. After the header, you can use any combination of selectors and modifiers to manipulate the elements. Every selector must be followed by a tab-indented block of one or more modifiers or nested selectors. Comments start with # and extend through the end of a line.

SkyFilter v0.4.1

TYPE way                                 # selects ways
	HAS "footway"                    # selects elements with a "footway" tag (any value)
		SET "surface" "concrete" # changes the value of the "surface" tag to be "concrete"
		COMMIT                   # immediately commit this element (skip the rest of the filter)
TYPE relation                            # selects relations
	EQUALS "type" "route"            # selects elements with the tag "type" set to "route"
		DROP                     # do not include element in output (skip the rest of the filter)
COMMIT                                   # commit the element

Statements

Below you can find descriptions of each statement supported by SkyFilter.

Selectors

A selector must be followed by one or more tab-indented statements (either a modifier or selection block). You can nest selection blocks.

• TYPE way, node — Selects elements of specified type(s), in a comma-separated list.
• HAS "key" — Selects elements with tag key.
• EQUALS "key" "value" — Selects elements with tag key equalling value.

Modifiers

• COMMIT — Commits element as it currently is to be written to output. Short-circuits the rest of the filter for that element.
• DROP — Drops element, i.e. excluding it entirely from the output. Short-circuits the rest of the filter for that element.
• SET "key" "value" — Sets tag key to value.
• KEEP "key_one", "key_two" — Only keep tags with specified key(s), removing any others from the element.
• RENAME "oldkey" "newkey" — Renames tag oldkey to newkey, keeping the value of the tag the same.
• DELETE "key_one", "key_two" — Removes tag(s) with specified key(s) from the element.