Introduction

skyway is a command-line OpenStreetMap file converter.

Goals

• Support a wide range of OpenStreetMap file formats
• Provide a user-friendly tag filtering system, capable of scaling to planet-scale data
• Perform conversions quickly, parallelizing computation where possible

Alternatives

Before adopting skyway for your project, please also consider Osmium Tool, a well-trusted and performant application that accomplishes many of the same things.

For a complete list of alternatives, please see the OSM file formats page on the OpenStreetMap wiki.

Installation

Pre-compiled Binaries

Builds of skyway are available on the releases page. Feel free to get in touch if you'd like me to prepare a binary for your platform.

Install Using Cargo

skyway is on crates.io, so you can install it using cargo:

cargo install skyway

Make sure your cargo bin directory is in your $PATH.

Selecting Features

Cargo "features" provide a mechanism to express conditional compilation and optional dependencies. A package defines a set of named features in the [features] table of Cargo.toml, and each feature can either be enabled or disabled. Features for the package being built can be enabled on the command-line with flags such as --features. Features for dependencies can be enabled in the dependency declaration in Cargo.toml.

―The Cargo Book

skyway's Cargo.toml file defines features which allow you to selectively compile certain file or filter formats. By default, all features are enabled. As well as providing a feature name for each individual file and filter format, filtering support can be enabled/disabled using the "filter" feature. Enabling the "cel" feature, for example, will automatically enable "filter", but you can enable "filter" by itself to access generic filtering functionality. Disabling all features leaves a relatively small library of OpenStreetMap conversion tools.

Usage

Here is an example of how to use skyway. For more information, you can run skyway --help.

skyway --from pbf --input input-file.pbf --to json --output output-file.json

If you do not specify an input or output file, skyway will default to standard in and standard out, respectively. This enables you to stream data into and out of skyway, like this:

cat input-file.pbf | skyway --from pbf --to json > output-file.json

Supported Formats

Here is a table showing the formats skyway supports reading and writing. The shortname is used to indicate the format when running skyway, e.g. skyway --from xml .... When paired together, "speedy" readers and writers will generally run faster by passing data between threads.

Key:
❌ – Not Supported
✅ – Supported
⚡ – Speedy!

^†Use the shortname json to read Overpass JSON, it uses the same parser as OSM JSON. Use overpass for writing.

Filtering

A skyway filter decides whether or not to keep an element, or possibly transforms it. If you do not provide a filter to skyway, it will convert the original input as faithfully as it can.

Supported Filtering Languages

• SkyFilter
• CEL

Running skyway with a Filter

To add a filter to skyway, add the --filter [FILTER FILE] option. You may pass multiple filters to evaluate in sequence by passing multiple --filter flags.

You should use .skyfilter or .cel file extensions so that skyway can provide helpful parsing error messages. However, this is not necessary and skyway will parse any valid filter file regardless of language.

How Filters Work

In skyway, the input reader and output writer run in parallel, with the former passing element information to the latter as they become available. This approach makes it easy to add one or more middlemen, who receive elements from the reader, transform them, and then pass them along down the chain. skyway provides a "filtering" system to make it easier to add these processing steps in a modular way. Using any number of CEL or SkyFilter expressions, you can create complex processing pipelines.

This system was inspired by Pandoc.

CEL Filters

skyway supports the Common Expression Language (CEL) for filtering elements. Each time the filter is evaluated for a given element, that expression's context (local variables) is updated to match the element's metadata. For now, CEL filters may only return a boolean value, indicating whether or not the element should be kept. Please file an issue if you'd like to see more complex CEL return types supported.

CEL Context

The following table describes each variable available to your expression:

Examples

Elements from a specific changeset:

changeset == 323878

Elements from a specific user id:

uid == 13159395

Only ways:

type == "way"

Elements with the key golf:

has(tags.golf)

SkyFilter

SkyFilter is a bespoke filtering language designed for skyway, allowing you to transform element data as they pass through skyway.

Specification

A SkyFilter file must start with a header as shown below, with the version matching the version of skyway that you are using, followed by a blank line. After the header, you can use any combination of selectors and modifiers to manipulate the elements. Every selector must be followed by a tab-indented block of one or more modifiers or nested selectors. Comments start with # and extend through the end of a line.

It is highly recommended to use the extension .skyfilter when saving SkyFilter files so that skyway knows what kind of filter it's intended to be. When you do this, skyway will provide more detailed parsing errors. Additionally, skyway will warn you if its version does not match that in the header of your SkyFilter file.

SkyFilter v0.6.0

TYPE way                                 # selects ways
	HAS "footway"                    # selects elements with a "footway" tag (any value)
		SET "surface" "concrete" # changes the value of the "surface" tag to be "concrete"
		COMMIT                   # immediately commit this element (skip the rest of the filter)
TYPE relation                            # selects relations
	EQUALS "type" "route"            # selects elements with the tag "type" set to "route"
		DROP                     # do not include element in output (skip the rest of the filter)
COMMIT                                   # commit the element

Statements

Below you can find descriptions of each statement supported by SkyFilter.

Selectors

A selector must be followed by one or more tab-indented statements (either a modifier or selection block). You can nest selection blocks to create more complex filters.

Modifiers

Using Regular Expressions (Regex)

Many SkyFilter statements accept regular expressions (see above). While regular strings are wrapped in double quotes (e.g. "footway") regex strings are prefixed with r before the first quote, like this: r"^tiger:".

SkyFilter uses the regex crate to parse regex expressions. See here for a detailed description of its syntax.