From 9b17bde4fec2dd50716dc670654eca1f1d75fe0a Mon Sep 17 00:00:00 2001 From: Peter Johnson Date: Mon, 11 Jun 2018 12:06:30 +0200 Subject: [PATCH] initial commit --- LICENSE | 21 ++ README.md | 302 ++++++++++++++++++ cmd/docker.sh | 30 ++ cmd/download.sh | 27 ++ cmd/elastic.sh | 31 ++ cmd/import.sh | 27 ++ cmd/prepare.sh | 24 ++ cmd/system.sh | 11 + images/elasticsearch/2.4/Dockerfile | 6 + images/elasticsearch/5.2.1/Dockerfile | 23 ++ images/elasticsearch/5.2.1/elasticsearch.yml | 6 + images/valhalla/Dockerfile | 25 ++ lib/cli.sh | 49 +++ lib/env.sh | 52 +++ pelias | 22 ++ projects/portland-metro/.env | 2 + projects/portland-metro/README.md | 35 ++ projects/portland-metro/docker-compose.yml | 114 +++++++ projects/portland-metro/pelias.json | 162 ++++++++++ .../portland-metro/synonyms/custom_name.txt | 138 ++++++++ .../portland-metro/synonyms/custom_street.txt | 133 ++++++++ 21 files changed, 1240 insertions(+) create mode 100644 LICENSE create mode 100644 README.md create mode 100644 cmd/docker.sh create mode 100644 cmd/download.sh create mode 100644 cmd/elastic.sh create mode 100644 cmd/import.sh create mode 100644 cmd/prepare.sh create mode 100644 cmd/system.sh create mode 100644 images/elasticsearch/2.4/Dockerfile create mode 100644 images/elasticsearch/5.2.1/Dockerfile create mode 100644 images/elasticsearch/5.2.1/elasticsearch.yml create mode 100644 images/valhalla/Dockerfile create mode 100644 lib/cli.sh create mode 100644 lib/env.sh create mode 100755 pelias create mode 100644 projects/portland-metro/.env create mode 100644 projects/portland-metro/README.md create mode 100644 projects/portland-metro/docker-compose.yml create mode 100644 projects/portland-metro/pelias.json create mode 100644 projects/portland-metro/synonyms/custom_name.txt create mode 100644 projects/portland-metro/synonyms/custom_street.txt diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..774263d --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2018 Pelias Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..a54f2a6 --- /dev/null +++ b/README.md @@ -0,0 +1,302 @@ + +# Pelias in Docker + +This repository contains a framework for downloading/preparing and building the the [Pelias Geocoder](https://github.com/pelias/pelias) using Docker and [Docker Compose](https://github.com/docker/compose#docker-compose). + +## Projects + +Example projects are included in the [projects](https://github.com/pelias/docker/tree/master/projects) directory. + +We recommend you start with the `portland-metro` example as a first-time user; once you have successfully completed a build you can use this as a base to create your own projects. + +## Not suitable for large geographies + +We do not recommend running large extracts (anything larger than a US State) inside Docker, the scripts are **not suitable** for full planet builds. If you require global coverage, please see our [install documentation](https://pelias.io/install.html) or consider using the [geocode.earth](https://geocode.earth/) services hosted by members of our core team. + +## Prerequisites + +You will need to have `docker` and `docker-compose` installed before continuing. If you are not using the latest version, please mention that in any bugs reports. + +If you are running OSX, you should also install `brew install coreutils` and max-out your Docker limits in `Docker > Preferences > Advanced`. + +Scripts can easily download tens of GB of geographic data, so ensure you have enough free disk space! + +## Installing the Pelias command + +If you haven't done so already, you will need to ensure the `pelias` command is available on your path. + +You can find the `pelias` file in the root of this repository. + +Advanced users may have a preferance how this is done on their system, but a basic example would be to do something like: + +```bash +git clone https://github.com/pelias/docker.git ~/pelias +ln -s ~/pelias/pelias /usr/local/bin/pelias +``` + +Once the command is correctly installed you should be able to run the following command to confim the pelias command is available on your path: + +```bash +which pelias +``` + +### Resolving PATH issues + +If you are having trouble getting this to work then quickly check that the target of the symlink is listed on your $PATH: + +```bash +tr ':' '\n' <<< "$PATH" +``` + +If you used the `ln -s` command above then the directory `/usr/local/bin` should be listed in the output. + +If the symlink target path is *not* listed, then you will either need to add its location to your $PATH or create a new symlink which points to a location which is already on your $PATH. + +## Configure Environment + +The `pelias` command looks for an `.env` file in your **current working directory**, this file contains information specific to your local environment. + +If this is your first time, you should change directories to an example project before continuing: + +```bash +cd projects/portland-metro +``` + +Ensure that your current working directory contains the files: `.env`, `docker-compose.yml` and `pelias.json` before continuing. + +### Variable: DATA_DIR + +The only mandatory variable in `.env` is `DATA_DIR`. + +This path reflects the directory Pelias will use to store downloaded data and use to build it's other microservices. + +You **must** create a new directory which you will use for this project, for example: + +```bash +mkdir /tmp/pelias +``` + +Then use your text editor to modify the `.env` file to reflect your new path, it should look like this: + +```bash +COMPOSE_PROJECT_NAME=pelias +DATA_DIR=/tmp/pelias +``` + +You can then list the environment variables to ensure they have been correctly set: + +```bash +pelias system env +``` + +### Variables: COMPOSE_* + +The compose variables are optional and are documented here: https://docs.docker.com/compose/env-file/ + +Note: changing the `COMPOSE_PROJECT_NAME` variable is not advisable unless you know what you are doing. If you are migrating from the deprecated `pelias/dockerfiles` repository then you can set `COMPOSE_PROJECT_NAME=dockerfiles` to enable backwards compatibility with containers created using that repository. + +## CLI commands + +```bash +$ pelias + +Usage: pelias [command] [action] [options] + + compose pull update all docker images + compose logs display container logs + compose ps list containers + compose top display the running processes of a container + compose exec execute an arbitrary docker-compose command + compose run execute a docker-compose run command + compose up start one or more docker-compose service(s) + compose kill kill one or more docker-compose service(s) + compose down stop all docker-compose service(s) + download wof (re)download whosonfirst data + download oa (re)download openaddresses data + download osm (re)download openstreetmap data + download tiger (re)download TIGER data + download transit (re)download transit data + download all (re)download all data + elastic drop delete elasticsearch index & all data + elastic create create elasticsearch index with pelias mapping + elastic start start elasticsearch server + elastic stop stop elasticsearch server + elastic status HTTP status code of the elasticsearch service + elastic wait wait for elasticsearch to start up + import wof (re)import whosonfirst data + import oa (re)import openaddresses data + import osm (re)import openstreetmap data + import polylines (re)import polylines data + import transit (re)import transit data + import all (re)import all data + prepare polylines export road network from openstreetmap into polylines format + prepare interpolation build interpolation sqlite databases + prepare placeholder build placeholder sqlite databases + prepare all build all services which have a prepare step + system check ensure the system is correctly configured + system env display environment variables + system update update the pelias command by pulling the latest version + ``` + +### Compose commands + +The compose commands are available as a shortcut to running `docker-compose` directly, they will also ensure that your environment is correctly configured. + +See the docker-compose documentation for more info: https://docs.docker.com/compose/overview/ + +```bash +pelias compose pull update all docker images +pelias compose logs display container logs +pelias compose ps list containers +pelias compose top display the running processes of a container +pelias compose exec execute an arbitrary docker-compose command +pelias compose run execute a docker-compose run command +pelias compose up start one or more docker-compose service(s) +pelias compose kill kill one or more docker-compose service(s) +pelias compose down stop all docker-compose service(s) +``` + +### Download commands + +The download commands will fetch and update geographic data from source. + +For example: `pelias download tiger` will fetch street data from the US Census Bureau and store it in the directory referenced by the `DATA_DIR` environment variable. + +```bash +pelias download wof (re)download whosonfirst data +pelias download oa (re)download openaddresses data +pelias download osm (re)download openstreetmap data +pelias download tiger (re)download TIGER data +pelias download transit (re)download transit data +pelias download all (re)download all data +``` + +### Prepare commands + +The prepare commands are used to run any commands which are required to setup/configure or build microservices. + +For example: `pelias prepare interpolation` will build a street address interpolation index. + +Note: the order of execution is important, the prepare commands require data, so they must be run *after* the download commands have fetched the data. + +```bash +pelias prepare polylines export road network from openstreetmap into polylines format +pelias prepare interpolation build interpolation sqlite databases +pelias prepare placeholder build placeholder sqlite databases +pelias prepare all build all services which have a prepare step +``` + +### Elastic commands + +The elastic commands control starting/stopping/configuring elasticsearch. + +The special `pelias elastic wait` command can be used in scripts to block the script execution until elasticsearch is ready to accept connections. + +```bash +pelias elastic drop delete elasticsearch index & all data +pelias elastic create create elasticsearch index with pelias mapping +pelias elastic start start elasticsearch server +pelias elastic stop stop elasticsearch server +pelias elastic status HTTP status code of the elasticsearch service +pelias elastic wait wait for elasticsearch to start up +``` + +### Import commands + +The import commands import source data in to elasticsearch. + +```bash +pelias import wof (re)import whosonfirst data +pelias import oa (re)import openaddresses data +pelias import osm (re)import openstreetmap data +pelias import polylines (re)import polylines data +pelias import transit (re)import transit data +pelias import all (re)import all data +``` + +### System commands + +The system commands help debug issues with incorrectly set environment variables. + +The `pelias system update` command can be used to ensure that the `pelias` command itself is up-to-date by pulling the latest source code from Github. + +```bash +pelias system check ensure the system is correctly configured +pelias system env display environment variables +pelias system update update the pelias command by pulling the latest version +``` + +## Generic build workflow + +The following shell script can be used to automate a build: + +```bash +#!/bin/bash +set -x + +# create directories +mkdir /code /data + +# clone repo +cd /code +git clone https://github.com/pelias/docker.git +cd docker + +# install pelias script +ln -s "$(pwd)/pelias" /usr/local/bin/pelias + +# cwd +cd projects/portland-metro + +# configure environment +sed -i '/DATA_DIR/d' .env +echo 'DATA_DIR=/data' >> .env + +# run build +pelias compose pull +pelias elastic start +pelias elastic wait +pelias elastic create +pelias download all +pelias prepare all +pelias import all +pelias compose up +``` + +## View status of running containers + +Once the build is complete, you can view the current status and port mappings of the Pelias docker containers: + +```bash +pelias compose ps +``` + +## View logs and debug errors + +You can inspect the container logs for errors by running: + +```bash +pelias compose logs +``` + +## Make an example query + +You can now make queries against your new Pelias build: + +### API + +- http://localhost:4000/v1/search?text=portland +- [http://localhost:4000/v1/search?text=1901 Main St](http://localhost:4000/v1/search?text=1901%20Main%20St) +- http://localhost:4000/v1/reverse?point.lon=-122.650095&point.lat=45.533467 + +### Placeholder + +- http://localhost:4100/demo/#eng + +### PIP (point in polygon) + +- http://localhost:4200/-122.650095/45.533467 + +### Interpolation + +- http://localhost:4300/demo/#13/45.5465/-122.6351 diff --git a/cmd/docker.sh b/cmd/docker.sh new file mode 100644 index 0000000..86a7787 --- /dev/null +++ b/cmd/docker.sh @@ -0,0 +1,30 @@ +#!/bin/bash +set -e; + +function compose_pull(){ compose_exec pull; } +register 'compose' 'pull' 'update all docker images' compose_pull + +function compose_logs(){ compose_exec logs $@; } +register 'compose' 'logs' 'display container logs' compose_logs + +function compose_ps(){ compose_exec ps $@; } +register 'compose' 'ps' 'list containers' compose_ps + +function compose_top(){ compose_exec top $@; } +register 'compose' 'top' 'display the running processes of a container' compose_top + +function compose_exec(){ docker-compose $@; } +register 'compose' 'exec' 'execute an arbitrary docker-compose command' compose_exec + +function compose_run(){ docker-compose run -T --rm $@; } +register 'compose' 'run' 'execute a docker-compose run command' compose_run + +function compose_up(){ docker-compose up -d $@; } +register 'compose' 'up' 'start one or more docker-compose service(s)' compose_up + +function compose_kill(){ docker-compose kill $@; } +register 'compose' 'kill' 'kill one or more docker-compose service(s)' compose_kill + +function compose_down(){ docker-compose down; } +register 'compose' 'down' 'stop all docker-compose service(s)' compose_down + diff --git a/cmd/download.sh b/cmd/download.sh new file mode 100644 index 0000000..fac4a1f --- /dev/null +++ b/cmd/download.sh @@ -0,0 +1,27 @@ +#!/bin/bash +set -e; + +# per-source downloads +function download_wof(){ compose_run 'whosonfirst' npm run download; } +function download_oa(){ compose_run 'openaddresses' npm run download; } +function download_osm(){ compose_run 'openstreetmap' npm run download; } +function download_tiger(){ compose_run 'interpolation' npm run download-tiger; } +function download_transit(){ compose_run 'transit' npm run download; } + +register 'download' 'wof' '(re)download whosonfirst data' download_wof +register 'download' 'oa' '(re)download openaddresses data' download_oa +register 'download' 'osm' '(re)download openstreetmap data' download_osm +register 'download' 'tiger' '(re)download TIGER data' download_tiger +register 'download' 'transit' '(re)download transit data' download_transit + +# download all the data to be used by imports +function download_all(){ + download_wof & + download_oa & + download_osm & + download_tiger & + download_transit & + wait +} + +register 'download' 'all' '(re)download all data' download_all diff --git a/cmd/elastic.sh b/cmd/elastic.sh new file mode 100644 index 0000000..eb22d47 --- /dev/null +++ b/cmd/elastic.sh @@ -0,0 +1,31 @@ +#!/bin/bash +set -e; + +function elastic_schema_drop(){ compose_run 'schema' node scripts/drop_index "$@" || true; } +function elastic_schema_create(){ compose_run 'schema' npm run create_index; } +function elastic_start(){ compose_exec up -d elasticsearch; } +function elastic_stop(){ compose_exec kill elasticsearch; } + +register 'elastic' 'drop' 'delete elasticsearch index & all data' elastic_schema_drop +register 'elastic' 'create' 'create elasticsearch index with pelias mapping' elastic_schema_create +register 'elastic' 'start' 'start elasticsearch server' elastic_start +register 'elastic' 'stop' 'stop elasticsearch server' elastic_stop + +# to use this function: +# if test $(elastic_status) -ne 200; then +function elastic_status(){ curl --output /dev/null --silent --write-out "%{http_code}" "http://${ELASTIC_HOST:-localhost:9200}" || true; } + +# the same function but with a trailing newline +function elastic_status_newline(){ echo $(elastic_status); } +register 'elastic' 'status' 'HTTP status code of the elasticsearch service' elastic_status_newline + +function elastic_wait(){ + echo 'waiting for elasticsearch service to come up'; + until test $(elastic_status) -eq 200; do + printf '.' + sleep 2 + done + echo +} + +register 'elastic' 'wait' 'wait for elasticsearch to start up' elastic_wait diff --git a/cmd/import.sh b/cmd/import.sh new file mode 100644 index 0000000..32de4b1 --- /dev/null +++ b/cmd/import.sh @@ -0,0 +1,27 @@ +#!/bin/bash +set -e; + +# per-source imports +function import_wof(){ compose_run 'whosonfirst' npm start; } +function import_oa(){ compose_run 'openaddresses' npm start; } +function import_osm(){ compose_run 'openstreetmap' npm start; } +function import_polylines(){ compose_run 'polylines' npm start; } +function import_transit(){ compose_run 'transit' npm start; } + +register 'import' 'wof' '(re)import whosonfirst data' import_wof +register 'import' 'oa' '(re)import openaddresses data' import_oa +register 'import' 'osm' '(re)import openstreetmap data' import_osm +register 'import' 'polylines' '(re)import polylines data' import_polylines +register 'import' 'transit' '(re)import transit data' import_transit + +# import all the data to be used by imports +# note: running importers in parallel can cause issues due to high CPU & RAM requirements. +function import_all(){ + import_wof + import_oa + import_osm + import_polylines + import_transit +} + +register 'import' 'all' '(re)import all data' import_all diff --git a/cmd/prepare.sh b/cmd/prepare.sh new file mode 100644 index 0000000..768bbc3 --- /dev/null +++ b/cmd/prepare.sh @@ -0,0 +1,24 @@ +#!/bin/bash +set -e; + +# per-source prepares +function prepare_polylines(){ compose_run 'polylines' bash ./docker_extract.sh; } +function prepare_interpolation(){ compose_run 'interpolation' bash ./docker_build.sh; } +function prepare_placeholder(){ + compose_run 'placeholder' npm run extract + compose_run 'placeholder' npm run build +} + +register 'prepare' 'polylines' 'export road network from openstreetmap into polylines format' prepare_polylines +register 'prepare' 'interpolation' 'build interpolation sqlite databases' prepare_interpolation +register 'prepare' 'placeholder' 'build placeholder sqlite databases' prepare_placeholder + +# prepare all the data to be used by imports +function prepare_all(){ + prepare_polylines & + prepare_placeholder & + wait + prepare_interpolation +} + +register 'prepare' 'all' 'build all services which have a prepare step' prepare_all diff --git a/cmd/system.sh b/cmd/system.sh new file mode 100644 index 0000000..225e272 --- /dev/null +++ b/cmd/system.sh @@ -0,0 +1,11 @@ +#!/bin/bash +set -e; + +function system_check(){ env_check; } +register 'system' 'check' 'ensure the system is correctly configured' system_check + +function system_env(){ env; } +register 'system' 'env' 'display environment variables' system_env + +function system_update(){ git -C $(dirname "${BASH_SOURCE[0]}") pull; } +register 'system' 'update' 'update the pelias command by pulling the latest version' system_update \ No newline at end of file diff --git a/images/elasticsearch/2.4/Dockerfile b/images/elasticsearch/2.4/Dockerfile new file mode 100644 index 0000000..d785704 --- /dev/null +++ b/images/elasticsearch/2.4/Dockerfile @@ -0,0 +1,6 @@ +# base image +FROM elasticsearch:2.4 + +# configure plugins +RUN /usr/share/elasticsearch/bin/plugin install analysis-icu +RUN /usr/share/elasticsearch/bin/plugin install cloud-aws diff --git a/images/elasticsearch/5.2.1/Dockerfile b/images/elasticsearch/5.2.1/Dockerfile new file mode 100644 index 0000000..be21352 --- /dev/null +++ b/images/elasticsearch/5.2.1/Dockerfile @@ -0,0 +1,23 @@ +# see: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html + +# base image +FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.1 +USER root + +# environmental settings +ENV ES_JAVA_OPTS '-Xms512m -Xmx512m' +ENV xpack.security.enabled 'false' +ENV xpack.monitoring.enabled 'false' +ENV cluster.name 'pelias-dev' +ENV bootstrap.memory_lock 'true' +RUN echo 'vm.max_map_count=262144' >> /etc/sysctl.conf + +# configure plugins +RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu + +# elasticsearch config +ADD elasticsearch.yml /usr/share/elasticsearch/config/ +RUN chown elasticsearch:elasticsearch config/elasticsearch.yml + +# run as elasticsearch user +USER elasticsearch diff --git a/images/elasticsearch/5.2.1/elasticsearch.yml b/images/elasticsearch/5.2.1/elasticsearch.yml new file mode 100644 index 0000000..5bbca33 --- /dev/null +++ b/images/elasticsearch/5.2.1/elasticsearch.yml @@ -0,0 +1,6 @@ +xpack.security.enabled: false +bootstrap.memory_lock: true +network.host: 0.0.0.0 +http.port: 9200 +node.master: true +node.data: true diff --git a/images/valhalla/Dockerfile b/images/valhalla/Dockerfile new file mode 100644 index 0000000..f7b0a0c --- /dev/null +++ b/images/valhalla/Dockerfile @@ -0,0 +1,25 @@ +# base image +FROM pelias/baseimage + +# grab all of the valhalla software from ppa +RUN apt-get update && \ + apt-get install -y software-properties-common python-software-properties && \ + add-apt-repository -y ppa:kevinkreiser/prime-server && \ + add-apt-repository -y ppa:valhalla-routing/valhalla && \ + apt-get update && \ + apt-get install -y valhalla-bin && \ + rm -rf /var/lib/apt/lists/*; + +# change working dir +RUN mkdir -p /code/valhalla +WORKDIR /code/valhalla + +# generate config +RUN valhalla_build_config \ + --mjolnir-tile-dir '/data/valhalla' \ + --mjolnir-tile-extract '/data/valhalla.tar' \ + --mjolnir-timezone '/data/valhalla/timezones.sqlite' \ + --mjolnir-admin '/data/valhalla/admins.sqlite' > valhalla.json + +# build script +RUN echo 'valhalla_build_tiles -c valhalla.json /data/openstreetmap/*.osm.pbf; valhalla_export_edges --config valhalla.json > /data/polylines/pbf_extract.polyline;' > ./docker_build.sh diff --git a/lib/cli.sh b/lib/cli.sh new file mode 100644 index 0000000..a47fb0a --- /dev/null +++ b/lib/cli.sh @@ -0,0 +1,49 @@ +#!/bin/bash +set -e; + +declare -a commands +declare -a actions +declare -a hints +declare -a functions + +function register(){ + commands+=("$1") + actions+=("$2") + hints+=("$3") + functions+=("$4") +} + +function help(){ + printf 'Usage: %s [command] [action] [options]\n\n' ${0} + + for (( i = 0; i < ${#commands[@]}; ++i )); do + echo -e " ${commands[$i]}\t${actions[$i]}\t ${hints[$i]}" + done | column -ts $'\t' + + echo +} + +function cli(){ + cmd="${1}"; shift || true + action="${1}"; shift || true + valid_command=false + valid_action=false + + for (( i = 0; i < ${#commands[@]}; ++i )); do + if [ "${cmd}" = "${commands[$i]}" ]; then + valid_command=true + if [ "${action}" = "${actions[$i]}" ]; then + valid_action=true + "${functions[$i]}" "$@" + exit $? + fi + fi + done + echo + + [ -z "${cmd}" ] || [ "$valid_command" = true ] || printf 'invalid command "%s"\n\n' "${cmd}" + [ -z "${action}" ] || [ "$valid_action" = true ] || printf 'invalid action "%s"\n\n' "${action}" + help + + exit 1 +} diff --git a/lib/env.sh b/lib/env.sh new file mode 100644 index 0000000..7c5142b --- /dev/null +++ b/lib/env.sh @@ -0,0 +1,52 @@ +#!/bin/bash +set -e; + +# disable verbose logging +ENV_DISPLAY_WARNINGS=false + +# ensure the user environment is correctly set up +function env_check(){ + if [ -z "${DATA_DIR}" ]; then + echo "You must set the DATA_DIR env var to a valid directory on your local machine." + echo + echo "Edit the '.env' file in this repository, update the DATA_DIR to a valid path and try again." + echo "Alternatively, you can set the variable in your environment using a command such as 'export DATA_DIR=/tmp'." + exit 1 + elif [ ! -d "${DATA_DIR}" ]; then + printf "The directory specified by DATA_DIR does not exist: %s\n" ${DATA_DIR} + echo + echo "Edit the '.env' file in this repository, update the DATA_DIR to a valid path and try again." + echo "Alternatively, you can set the variable in your environment using a command such as 'export DATA_DIR=/tmp'." + exit 1 + fi +} + +# loads environment vars from a stream (such as a file) +# example: env_load_stream < .env +function env_load_stream(){ + while IFS='=' read -r key value; do + ([ -z $key ] || [ -z $value ]) && printf 'Invalid environment var "%s=%s"\n' $key $value && exit 1 + if [ -z ${!key} ]; then + export "${key}=${value}" + elif $ENV_DISPLAY_WARNINGS; then + printf '[warn] skip setting environment var "%s=%s", already set "%s=%s"\n' $key $value $key ${!key} + fi + done +} + +# ensure locale is correctly set? +# export LC_ALL=en_US.UTF-8 + +# load DATA_DIR and other vars from docker-compose .env file +# note: strips comments and empty lines +[ -f .env ] && env_load_stream < <(grep -v '^$\|^\s*$\#' .env) + +# use the default compose file unless one was specified +# if [ -z "${COMPOSE_FILE}" ]; then +# if [ ! -f "docker-compose.yml" ]; then +# export COMPOSE_FILE="${BASEDIR}/docker-compose.yml" +# fi +# fi + +# ensure the user env is correctly set up +env_check diff --git a/pelias b/pelias new file mode 100755 index 0000000..0f383f5 --- /dev/null +++ b/pelias @@ -0,0 +1,22 @@ +#!/bin/bash +set -e + +# OSX comes bundled with versions of readlink, sed, parallel etc which are not +# compatible with the linux tools. Force OSX users to install the GNU +# compatible versions (prefixed with 'g', such as 'greadlink', 'gsed' etc.). +export CMD_READLINK='readlink' +if [[ "$OSTYPE" == "darwin"* ]]; then + if [ -x "$(command -v greadlink)" ]; then + CMD_READLINK='greadlink'; + else + 2>&1 echo 'OSX: you must install the gnu standard tooling using:' + 2>&1 echo 'brew install coreutils' + fi +fi + +# resolve path to this file (following symlinks) and load libs +BASEDIR=$( dirname $( ${CMD_READLINK} -f "${BASH_SOURCE[0]}" ) ) +for f in ${BASEDIR}/lib/* ${BASEDIR}/cmd/*; do source $f; done + +# cli runner +cli "$@" diff --git a/projects/portland-metro/.env b/projects/portland-metro/.env new file mode 100644 index 0000000..136b49a --- /dev/null +++ b/projects/portland-metro/.env @@ -0,0 +1,2 @@ +COMPOSE_PROJECT_NAME=pelias +DATA_DIR=/tmp/pelias/portland-metro diff --git a/projects/portland-metro/README.md b/projects/portland-metro/README.md new file mode 100644 index 0000000..0a08517 --- /dev/null +++ b/projects/portland-metro/README.md @@ -0,0 +1,35 @@ + +# Portland Metro Area + +This project is configured to download/prepare/build a complete Pelias installation for Portland, Oregon. + +It is intended as an example for other projects, feel free to copy->paste these files to a new project directory to kick-start your own project. + +# Setup + +Please refer to the instructions at https://github.com/pelias/docker in order to install and configure your docker environment. + +The minimum configuration required in order to run this project are [installing prerequisites](https://github.com/pelias/docker#prerequisites), [install the pelias command](https://github.com/pelias/docker#installing-the-pelias-command) and [configure the environment](https://github.com/pelias/docker#configure-environment). + +Please ensure that's all working fine before continuing. + +# Run a Build + +To run a complete build, execute the following commands: + +```bash +pelias compose pull +pelias elastic start +pelias elastic wait +pelias elastic create +pelias download all +pelias prepare all +pelias import all +pelias compose up +``` + +# Make an Example Query + +You can now make queries against your new Pelias build: + +http://localhost:4000/v1/search?text=pdx diff --git a/projects/portland-metro/docker-compose.yml b/projects/portland-metro/docker-compose.yml new file mode 100644 index 0000000..2f21124 --- /dev/null +++ b/projects/portland-metro/docker-compose.yml @@ -0,0 +1,114 @@ +version: '3' +networks: + default: + driver: bridge +volumes: + libpostaldata: + driver: local +services: + libpostal_baseimage: + image: pelias/libpostal_baseimage + container_name: pelias_libpostal_baseimage + libpostal: + image: pelias/go-whosonfirst-libpostal + container_name: pelias_libpostal + restart: always + ports: [ "8080:8080" ] + schema: + image: pelias/schema:portland-synonyms + container_name: pelias_schema + volumes: + - "./pelias.json:/code/pelias.json" + - "./synonyms/custom_name.txt:/code/pelias/schema/synonyms/custom_name.txt" + - "./synonyms/custom_street.txt:/code/pelias/schema/synonyms/custom_street.txt" + api: + image: pelias/api:master + container_name: pelias_api + restart: always + environment: [ "PORT=4000" ] + ports: [ "4000:4000" ] + volumes: + - "./pelias.json:/code/pelias.json" + - "libpostaldata:/usr/share/libpostal" + placeholder: + image: pelias/placeholder:master + container_name: pelias_placeholder + restart: always + environment: [ "PORT=4100" ] + ports: [ "4100:4100" ] + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + whosonfirst: + image: pelias/whosonfirst:master + container_name: pelias_whosonfirst + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + openstreetmap: + image: pelias/openstreetmap:master + container_name: pelias_openstreetmap + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + openaddresses: + image: pelias/openaddresses:master + container_name: pelias_openaddresses + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + transit: + image: pelias/transit:master + container_name: pelias_transit + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + polylines: + image: pelias/polylines:master + container_name: pelias_polylines + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + interpolation: + depends_on: [ "libpostal_baseimage" ] + image: pelias/interpolation:master + container_name: pelias_interpolation + restart: always + environment: [ "PORT=4300" ] + ports: [ "4300:4300" ] + volumes: + - "./pelias.json:/code/pelias.json" + - "libpostaldata:/usr/share/libpostal" + - "${DATA_DIR}:/data" + pip: + image: pelias/pip-service:master + container_name: pelias_pip-service + restart: always + environment: [ "PORT=4200" ] + ports: [ "4200:4200" ] + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + document-service: + image: pelias/document-service:master + container_name: pelias_document-service + restart: always + ports: [ "5000:5000" ] + volumes: + - "./pelias.json:/code/pelias.json" + - "${DATA_DIR}:/data" + elasticsearch: + image: pelias/elasticsearch + container_name: pelias_elasticsearch + restart: always + ports: [ "9200:9200", "9300:9300" ] + volumes: + - "${DATA_DIR}/elasticsearch:/usr/share/elasticsearch/data" + ulimits: + memlock: + soft: -1 + hard: -1 + nofile: + soft: 65536 + hard: 65536 + cap_add: [ "IPC_LOCK" ] diff --git a/projects/portland-metro/pelias.json b/projects/portland-metro/pelias.json new file mode 100644 index 0000000..35bf4bd --- /dev/null +++ b/projects/portland-metro/pelias.json @@ -0,0 +1,162 @@ +{ + "logger": { + "level": "info", + "timestamp": false + }, + "esclient": { + "hosts": [ + { "host": "elasticsearch" } + ] + }, + "elasticsearch": { + "settings": { + "index": { + "refresh_interval": "10s", + "number_of_replicas": "0", + "number_of_shards": "1" + } + } + }, + "api": { + "textAnalyzer": "libpostal", + "services": { + "pip": { "url": "http://pip-service:4200" }, + "libpostal": { "url": "http://libpostal:8080" }, + "placeholder": { "url": "http://placeholder:4100" }, + "interpolation": { "url": "http://interpolation:4300" } + }, + "defaultParameters": { + "focus.point.lat": 45.52, + "focus.point.lon": -122.67 + } + }, + "imports": { + "adminLookup": { + "enabled": true + }, + "geonames": { + "datapath": "/data/geonames", + "countryCode": "ALL" + }, + "openstreetmap": { + "download": [ + { "sourceURL": "https://s3.amazonaws.com/metro-extracts.nextzen.org/portland_oregon.osm.pbf" } + ], + "leveldbpath": "/tmp", + "datapath": "/data/openstreetmap", + "import": [{ + "filename": "portland_oregon.osm.pbf" + }] + }, + "openaddresses": { + "datapath": "/data/openaddresses", + "files": [ + "us/or/portland_metro.csv", + "us/or/city_of_salem.csv", + "us/or/marion_and_polk.csv", + "us/or/marion.csv", + "us/or/hood_river.csv", + "us/wa/city_of_richland.csv", + "us/wa/clark.csv" + ] + }, + "polyline": { + "datapath": "/data/polylines", + "files": [ "extract.0sv" ] + }, + "whosonfirst": { + "datapath": "/data/whosonfirst", + "importVenues": false, + "importPostalcodes": true, + "importPlace": [ + "85688513", + "85688623" + ] + }, + "transit": { + "datapath": "/data/transit", + "feeds": [ + { + "layerId": "bikeshare", + "url": "http://biketownpdx.socialbicycles.com/opendata/station_information.json", + "filename": "BIKETOWN-hubs.json", + "agencyId": "BIKETOWN", + "agencyName": "BIKETOWN", + "layerName": "bikeshare" + }, + { + "layerId": "stops", + "url": "http://developer.trimet.org/schedule/gtfs.zip", + "filename": "TRIMET-stops.txt", + "agencyId": "TRIMET", + "agencyName": "TriMet", + "layerName": "Stop" + }, + { + "layerId": "stops", + "url": "http://www.c-tran.com/images/Google/GoogleTransitUpload.zip", + "filename": "C-TRAN-stops.txt", + "agencyId": "C-TRAN", + "agencyName": "C-TRAN", + "layerName": "Stop" + }, + { + "layerId": "stops", + "url": "http://oregon-gtfs.com/gtfs_data/rideconnection-or-us/rideconnection-or-us.zip", + "filename": "RIDECONNECTION-stops.txt", + "agencyId": "RIDECONNECTION", + "agencyName": "Ride Connection", + "layerName": "Stop" + }, + { + "layerId": "stops", + "url": "http://cherriots.org/developer/gtfs.zip", + "filename": "CHERRIOTS-stops.txt", + "agencyId": "CHERRIOTS", + "agencyName": "Cherriots", + "layerName": "Stop" + }, + { + "layerId": "stops", + "url": "http://oregon-gtfs.com/gtfs_data/sandy-or-us/sandy-or-us.zip", + "filename": "SAM-stops.txt", + "agencyId": "SAM", + "agencyName": "SAM", + "layerName": "Stop" + }, + { + "layerId": "stops", + "url": "http://data.trilliumtransit.com/gtfs/wilsonville-or-us/wilsonville-or-us.zip", + "filename": "SMART-stops.txt", + "agencyId": "SMART", + "agencyName": "SMART", + "layerName": "Stop" + }, + { + "layerId": "stops", + "url": "http://oregon-gtfs.com/gtfs_data/swanisland-or-us/swanisland-or-us.zip", + "filename": "SWAN-stops.txt", + "agencyId": "SWAN", + "agencyName": "Swan Island", + "layerName": "Stop" + } + ] + }, + "interpolation": { + "download": { + "tiger": { + "datapath": "/data/tiger", + "states": [ + { "state_code": 41, "county_code": 67, "name": "Washington County, OR" }, + { "state_code": 41, "county_code": 51, "name": "Multnomah County, OR" }, + { "state_code": 41, "county_code": 5, "name": "Clackamas County, OR" }, + { "state_code": 41, "county_code": 71, "name": "Yamhill County, OR" }, + { "state_code": 41, "county_code": 47, "name": "Marion County, OR" }, + { "state_code": 41, "county_code": 53, "name": "Polk County, OR" }, + { "state_code": 53, "county_code": 11, "name": "Clark County, WA" } + ] + } + } + } + } +} diff --git a/projects/portland-metro/synonyms/custom_name.txt b/projects/portland-metro/synonyms/custom_name.txt new file mode 100644 index 0000000..38e0d35 --- /dev/null +++ b/projects/portland-metro/synonyms/custom_name.txt @@ -0,0 +1,138 @@ + +shopping,store,shop,retail,grocery,mall + +# Transit Centers +Oregon City Transit Center,Oregon City TC,OCTC +Barbur Blvd Transit Center,BBTC,Barbur Blvd TC,Barbur TC +Beaverton Transit Center,BTC,Beaverton TC +Clackamas Town Center Transit Center,CTC,Clackamas TC,Clackamas Town Center TC,Clackamas Transit Center +Gateway/NE 99th Ave Transit Center,GTC,Gateway TC,Gateway Transit Center +Gresham Central Transit Center,GCTC,Gresham Central TC,Gresham TC +Hillsboro Center/SE 3rd Ave Transit Center, HCTC, Hillsboro TC, Hillsboro Transit Center +Hollywood/NE 42nd Ave Transit Center,HTC, HOllywood TC,HOllywood Transit Center +Lake Oswego Transit Center, LOTC,Lake Oswego TC +N Lombard Transit Center, Lombard Transit Center, Lombard TC,NLTC +Parkrose/Sumner Transit Center, Parkrose Transit Center,Parkrose TC,PRSTC,PTC +Rose Quarter Transit Center,RQTC, Rose Quarter TC +Sunset Transit Center,Sunset TC, STC +Tigard Transit Center,Tigard TC,TTC +Washington Square Transit Center,WSTC,Washington Square TC +Willow Creek/SW 185th Ave Transit Center, Willow Creek TC, Willow Creek Transit Center, WCTC + +# Rail Stations +Yamhill District MAX Station,YAMX +Washington/SE 12th Ave MAX Station, WASX +Tuality Hospital/SE 8th Ave MAX Station, TUAX +Skidmore Fountain MAX Station, SKIX +SE Powell Blvd MAX Station,POWX +SE Holgate BLvd MAX Station,HOLX +SE Fuller Rd MAX Station,FULX +SE Flavel St MAX Station,FLAX +SE Division St MAX Station,DIVX +Ruby Junction/E 197th Ave MAX Station,RUBX +Rosa Parks MAX Station,N Portland Blvd MAX Station,ROSX,RPX +Rockwood/E 188th Ave MAX Station,ROCX +Quatama/NW 205th Ave MAX Station,QUAX +Providence Park MAX Station,Civic Stadium MAX Station,JELD-WEN Field MAX Station,PGE Park MAX Station,PPX,PROX +Portland Int'l Airport MAX Station; PDXX +Pioneer Square South MAX Station,PCSS MAX +Pioneer Square North MAX Station,PCSN MAX +Parkrose/Sumner TC MAX Station,Parkrose MAX Station,PTC +Overlook Park MAX Station,OVEX +Orenco/NW 231st Ave MAX Station,OREX +Old Town/Chinatown MAX Station,OLDX +Oak/SW 1st Ave MAX Station,OAKX +N Prescott St MAX Station,PREX +N Killingsworth St MAX Station,KILX +NE 60th Ave MAX Station,60X +NE 82nd Ave MAX Station,82X +Mt Hood Ave MAX Station,MTHX +Millikan Way MAX Station,MILX +Merlo Rd/SW 158th Ave MAX Station,MERX +Lloyd Center/NE 11th Ave MAX Station,LLOX +Library/SW 9th Ave MAX Station,LIBX +Lents Town Center/Foster Rd MAX Station,LENX +Kings Hill/SW Salmon St MAX Station,KINX +Kenton/N Denver Ave MAX Station,KENX +Interstate/Rose Quarter MAX Station,IRQ MAX,IRQ +Hawthorn Farm MAX Station,HAWX +Goose Hollow/SW Jefferson St MAX Station,GHX,GOOX +Galleria/SW 10th Ave MAX Station,GALX +Fair Complex/Hillsboro Airport MAX Station,FAIX +Expo Center MAX Station,EXPX +E 162nd Ave MAX Station,162X +E 172nd Ave MAX Station,172X +E 181st Ave MAX Station,181X +E 102nd Ave MAX Station,102X +E 122nd Ave MAX Station,122X +E 148th Ave MAX Station,148X +Elmonica/SW 170th Ave MAX Station,ELMX,Elmonica +Delta Park/Vanport MAX Station,DELX,DPV MAX,DPX +Convention Center MAX Station,CCX,CONX +Cleveland Ave MAX Station,CLEX +Cascades MAX Station,CASX +Beaverton Creek MAX Station,BCRX +Beaverton Central MAX Station,BEAX +Albina/Mississippi MAX Station,ALBX + +# TriMet Facilities +ATP/Nela,Nela +Center Street Bus Maintenance, Center Garage +Elmonica Operations Facility,Elmo,Elmonica +Merlo Operations Facility,Merlo Garage, Merlo +Powell Operations Facility,Powell Garage +Transit Mobility Center,ATP Mobility Center,Mobility Center +TriMet Administration Headquarters,Admin Building,Administration Building,Harrison Square,TriMet Administrative Offices +TriMet Operations Headquarters,Center St, Center Street,TriMet Lost & Found +#Trimet Ticket Office,TTO +Vintage Trolley Barn,Trolley Barn + +# Misc Landmarks +# FXP: removed psu and pdx, since they led to ambig hits +Oregon Zoo,Zoo,Washington Park Zoo +Portland State University,Portland State +Portland International Raceway,PIR +Portland International Airport,Portland Airport +Portland Community College - Cascade,PCC Cascade +Portland Community College - Rock Creek,PCC Rock Creek +Portland Community College - Southeast,PCC Southeast, PCC Southeast Center +Portland Community College - Sylvania,PCC Sylvania +Portland Community College - Willow Creek Center,PCC Willow Creek Center +Pioneer Courthouse Square,Downtown Portland,PCS +Oregon Rail Heritage Center,ORHF +Oregon Museum of Science and Industry,OMSI +Oregon Health & Science University,OHSU,Oregon Health Sciences University +Oregon College of Oriental Medicine,OCOM +Moda Center at the Rose Quarter,Moda Center,Rose Garden Arena +Clackamas Community College,CCC +Clackamas Community College Harmony Campus, CCCH +Gordon Faber Recreation Complex,Hillsboro Stadium,Ron Tonkin Field +Providence Park,Civic Stadium,JELD-WEN Field,PGE Park +OSHU Collaborative Life Sciences Building, Skourtes Tower, CLSB, School of Dentistry +Aging & Disability, DHS +African Youth and Community Organization, AYCO + +north, N +south, S +east, E +west, W +north west => northwest +north east => northeast +south west => southwest +south east => southeast +northwest, NW +northeast, NE +southwest, SW +southeast, SE + +Portland, Portland City Center +Int'l, international +clackamus, clackamas +clack, clackamas +p-town, portland +casscade, cascade + +# street names +Tualatin Valley Hwy,TV Hwy,TVH,TVHWY +Beaverton-Hillsdale Hwy,Beaverton- Hillsdale Hwy,BH Hwy,BHH +MLK,M L King,Martin Luther King diff --git a/projects/portland-metro/synonyms/custom_street.txt b/projects/portland-metro/synonyms/custom_street.txt new file mode 100644 index 0000000..32a4be7 --- /dev/null +++ b/projects/portland-metro/synonyms/custom_street.txt @@ -0,0 +1,133 @@ + +Tualatin Valley Hwy,TV Hwy,TVH,TVHWY +Beaverton Hillsdale HWY,Beaverton-Hillsdale HWY,BH HYW,BHH +Beaverton Hillsdale,Beaverton-Hillsdale,BH +Martin Luther King,ML King,M L King,MLK +Doernbecher,Dornbecher,Doernbecker,Dornbecker,Dornbecka +Native American Youth => NAYA +Transit Center => TC +Park Ride => PR +Park & Ride => PR +Park and Ride => PR +caeser,cesar,chavez,39th +metro,portland +oregon,or +carshare,car2go,Zipcar,Flexcar,GetAround,ucarshare,uhaul,hertz,wecar,enterprise +ticket,fare,outlet,store,retail +dmv,motor,vehicles +court house => courthouse +court-house => courthouse +cc,court,courts,courth,courthouse +law,jail,prison,jail,courthouse,enforcement,juvenile,justice,juvenile,correctional +jewvi,juvi,juvenile + +first,1st +second,2nd +third,3rd +forth,4th +fifth,5th +sixth,6th +seventh,7th +eighth,8th +nineth,9th +tenth,10th +eleventh,11th +twelfth,12th +thirteenth,13th +fourteenth,14th +fifteenth,15th +sixteenth,16th +seventeenth,17th +eighteenth,18th +nineteenth,19th +twentyth,20th +twentyfirst,21st +twentysecond,22nd +twentythrid,23rd + +# USPS Suffix Abbreviations +# http://www.usps.com/ncsc/lookups/abbreviations.html#suffix +st,st.,street,str +ave,ave.,av,av.,avenue,AVNUE,AVENUE,AVENU +dr,drive,dr.,drv,driv,drives +BOULEVARD,BLVD,BOULEVARD,BOUL,BOULEVARD,BOULEVARD,BOULV +BRIDGE,BRDGE,BRG +BYPASS,BYP,BYPA,BYPAS,BYPS +CLIFF,CLF,CLIFFS,CLFS +CLUB,CLB +COMMON,CMN +CORNER,COR,CORNERS,CORS +CENTER,CEN,CENT,CENTER,CENTR,CENTRE,CNTER,CNTR,CTR,CENTERS,CTRS +CIR,CIRC,CIRCL,CIRCLE,CRCL,CIRCLE,CRCLE +COURT,CRT,CT,COURTS +CTSCK,CR,CREEK,CRK +CROSSING,CRSSING,CRSSNG,XING +CRECENT,CRES,CRESCENT,CRESENT,CRSCNT,CRSENT,CRSNT +EXP,EXPR,EXPRESS,EXPRESSWAY,EXPW,EXPY +FALL,FALLS,FLS +FERRY,FRRY,FRY +FIELD,FLD,FIELDS,FLDS +FREEWAY,FREEWY,FRWAY,FRWY,FWY +GARDEN,GARDN,GDN,GRDEN,GRDN,GARDENS,GDNS,GRDNS +GATEWAY,GATEWY,GATWAY,GTWAY,GTWY +GROV,GROVE,GRV,GROVES +HARB,HARBOR,HARBR,HBR,HRBOR,HARBORS +HEIGHT,HEIGHTS,HGTS,HT,HTS +HIGHWAY,HIGHWY,HIWAY,HIWY,HWAY,HWY +HILL,HL,HILLS,HLS +IS,ISLAND,ISLND,ISLANDS,ISLNDS,ISS,ISLE,ISLES +JCT,JCTION,JCTN,JUNCTION,JUNCTN,JUNCTON,JCTNS,JCTS,JUNCTIONS +LAKE,LK,LAKES,LKS +LANDING,LNDG,LNDNG +LA,LANE,LANES,LN +LOOP,LOOPS,LP +MANOR,MNR,MANORS,MNRS +MDW,MEADOW,MDWS,MEADOWS,MEDOWS +MILL,ML,MILLS,MLS +MNT,MOUNT,MT +MNTAIN,MNTN,MOUNTAIN,MOUNTIN,MTIN,MTN,MNTNS,MOUNTAINS +ORCH,ORCHARD,ORCHRD +PARK,PK,PRK,PARKS +PARKWAY,PARKWY,PKWAY,PKWY,PKY,PARKWAYS,PKWYS +PLAZA,PLZ,PLZA +PLACE,PL +PR,PRAIRIE,PRARIE,PRR +RANCH,RANCHES,RNCH,RNCHS +RDG,RDGE,RIDGE,RDGS,RIDGES +RIV,RIVER,RIVR,RVR +RD,ROAD,RDS,ROADS +RTESHLS,SHOALS +SHOAR,SHORE,SHR,SHOARS,SHORES,SHRS +SPG,SPNG,SPRING,SPRNG,SPGS,SPNGS,SPRINGS,SPRNGS +SQ,SQR,SQRE,SQU,SQUARE,SQRS,SQUARES +STA,STATION,STATN,STN +STREAM,STREME,STRM +ST,STR,STREET,STRT,STREETS +SMT,SUMIT,SUMITT,SUMMIT +TER,TERR,TERRACE +TR,TRAIL,TRAILS,TRL,TRLS +TUNEL,TUNL,TUNLS,TUNNEL,TUNNELS,TUNNL +UN,UNION,UNIONS +VALLEY,VALLY,VLLY,VLY,VALLEYS,VLYS +VDCT,VIA,VIADCT,VIADUCT +VIEW,VW,VIEWS,VWS +VILL,VILLAG,VILLAGE,VILLG,VILLIAGE,VLG,VILLAGES,VLGS +VILLE,VL +VIS,VIST,VISTA,VST,VSTA +WAY,WY,WAYS + +north, N +south, S +east, E +west, W +north west => northwest +north east => northeast +south west => southwest +south east => southeast +northwest, NW +northeast, NE +southwest, SW +southeast, SE + +broadway st => broadway +broadway street => broadway