Browse Source

initial commit

pull/1/head
Peter Johnson 7 years ago
commit
9b17bde4fe
  1. 21
      LICENSE
  2. 302
      README.md
  3. 30
      cmd/docker.sh
  4. 27
      cmd/download.sh
  5. 31
      cmd/elastic.sh
  6. 27
      cmd/import.sh
  7. 24
      cmd/prepare.sh
  8. 11
      cmd/system.sh
  9. 6
      images/elasticsearch/2.4/Dockerfile
  10. 23
      images/elasticsearch/5.2.1/Dockerfile
  11. 6
      images/elasticsearch/5.2.1/elasticsearch.yml
  12. 25
      images/valhalla/Dockerfile
  13. 49
      lib/cli.sh
  14. 52
      lib/env.sh
  15. 22
      pelias
  16. 2
      projects/portland-metro/.env
  17. 35
      projects/portland-metro/README.md
  18. 114
      projects/portland-metro/docker-compose.yml
  19. 162
      projects/portland-metro/pelias.json
  20. 138
      projects/portland-metro/synonyms/custom_name.txt
  21. 133
      projects/portland-metro/synonyms/custom_street.txt

21
LICENSE

@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2018 Pelias Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

302
README.md

@ -0,0 +1,302 @@
# Pelias in Docker
This repository contains a framework for downloading/preparing and building the the [Pelias Geocoder](https://github.com/pelias/pelias) using Docker and [Docker Compose](https://github.com/docker/compose#docker-compose).
## Projects
Example projects are included in the [projects](https://github.com/pelias/docker/tree/master/projects) directory.
We recommend you start with the `portland-metro` example as a first-time user; once you have successfully completed a build you can use this as a base to create your own projects.
## Not suitable for large geographies
We do not recommend running large extracts (anything larger than a US State) inside Docker, the scripts are **not suitable** for full planet builds. If you require global coverage, please see our [install documentation](https://pelias.io/install.html) or consider using the [geocode.earth](https://geocode.earth/) services hosted by members of our core team.
## Prerequisites
You will need to have `docker` and `docker-compose` installed before continuing. If you are not using the latest version, please mention that in any bugs reports.
If you are running OSX, you should also install `brew install coreutils` and max-out your Docker limits in `Docker > Preferences > Advanced`.
Scripts can easily download tens of GB of geographic data, so ensure you have enough free disk space!
## Installing the Pelias command
If you haven't done so already, you will need to ensure the `pelias` command is available on your path.
You can find the `pelias` file in the root of this repository.
Advanced users may have a preferance how this is done on their system, but a basic example would be to do something like:
```bash
git clone https://github.com/pelias/docker.git ~/pelias
ln -s ~/pelias/pelias /usr/local/bin/pelias
```
Once the command is correctly installed you should be able to run the following command to confim the pelias command is available on your path:
```bash
which pelias
```
### Resolving PATH issues
If you are having trouble getting this to work then quickly check that the target of the symlink is listed on your $PATH:
```bash
tr ':' '\n' <<< "$PATH"
```
If you used the `ln -s` command above then the directory `/usr/local/bin` should be listed in the output.
If the symlink target path is *not* listed, then you will either need to add its location to your $PATH or create a new symlink which points to a location which is already on your $PATH.
## Configure Environment
The `pelias` command looks for an `.env` file in your **current working directory**, this file contains information specific to your local environment.
If this is your first time, you should change directories to an example project before continuing:
```bash
cd projects/portland-metro
```
Ensure that your current working directory contains the files: `.env`, `docker-compose.yml` and `pelias.json` before continuing.
### Variable: DATA_DIR
The only mandatory variable in `.env` is `DATA_DIR`.
This path reflects the directory Pelias will use to store downloaded data and use to build it's other microservices.
You **must** create a new directory which you will use for this project, for example:
```bash
mkdir /tmp/pelias
```
Then use your text editor to modify the `.env` file to reflect your new path, it should look like this:
```bash
COMPOSE_PROJECT_NAME=pelias
DATA_DIR=/tmp/pelias
```
You can then list the environment variables to ensure they have been correctly set:
```bash
pelias system env
```
### Variables: COMPOSE_*
The compose variables are optional and are documented here: https://docs.docker.com/compose/env-file/
Note: changing the `COMPOSE_PROJECT_NAME` variable is not advisable unless you know what you are doing. If you are migrating from the deprecated `pelias/dockerfiles` repository then you can set `COMPOSE_PROJECT_NAME=dockerfiles` to enable backwards compatibility with containers created using that repository.
## CLI commands
```bash
$ pelias
Usage: pelias [command] [action] [options]
compose pull update all docker images
compose logs display container logs
compose ps list containers
compose top display the running processes of a container
compose exec execute an arbitrary docker-compose command
compose run execute a docker-compose run command
compose up start one or more docker-compose service(s)
compose kill kill one or more docker-compose service(s)
compose down stop all docker-compose service(s)
download wof (re)download whosonfirst data
download oa (re)download openaddresses data
download osm (re)download openstreetmap data
download tiger (re)download TIGER data
download transit (re)download transit data
download all (re)download all data
elastic drop delete elasticsearch index & all data
elastic create create elasticsearch index with pelias mapping
elastic start start elasticsearch server
elastic stop stop elasticsearch server
elastic status HTTP status code of the elasticsearch service
elastic wait wait for elasticsearch to start up
import wof (re)import whosonfirst data
import oa (re)import openaddresses data
import osm (re)import openstreetmap data
import polylines (re)import polylines data
import transit (re)import transit data
import all (re)import all data
prepare polylines export road network from openstreetmap into polylines format
prepare interpolation build interpolation sqlite databases
prepare placeholder build placeholder sqlite databases
prepare all build all services which have a prepare step
system check ensure the system is correctly configured
system env display environment variables
system update update the pelias command by pulling the latest version
```
### Compose commands
The compose commands are available as a shortcut to running `docker-compose` directly, they will also ensure that your environment is correctly configured.
See the docker-compose documentation for more info: https://docs.docker.com/compose/overview/
```bash
pelias compose pull update all docker images
pelias compose logs display container logs
pelias compose ps list containers
pelias compose top display the running processes of a container
pelias compose exec execute an arbitrary docker-compose command
pelias compose run execute a docker-compose run command
pelias compose up start one or more docker-compose service(s)
pelias compose kill kill one or more docker-compose service(s)
pelias compose down stop all docker-compose service(s)
```
### Download commands
The download commands will fetch and update geographic data from source.
For example: `pelias download tiger` will fetch street data from the US Census Bureau and store it in the directory referenced by the `DATA_DIR` environment variable.
```bash
pelias download wof (re)download whosonfirst data
pelias download oa (re)download openaddresses data
pelias download osm (re)download openstreetmap data
pelias download tiger (re)download TIGER data
pelias download transit (re)download transit data
pelias download all (re)download all data
```
### Prepare commands
The prepare commands are used to run any commands which are required to setup/configure or build microservices.
For example: `pelias prepare interpolation` will build a street address interpolation index.
Note: the order of execution is important, the prepare commands require data, so they must be run *after* the download commands have fetched the data.
```bash
pelias prepare polylines export road network from openstreetmap into polylines format
pelias prepare interpolation build interpolation sqlite databases
pelias prepare placeholder build placeholder sqlite databases
pelias prepare all build all services which have a prepare step
```
### Elastic commands
The elastic commands control starting/stopping/configuring elasticsearch.
The special `pelias elastic wait` command can be used in scripts to block the script execution until elasticsearch is ready to accept connections.
```bash
pelias elastic drop delete elasticsearch index & all data
pelias elastic create create elasticsearch index with pelias mapping
pelias elastic start start elasticsearch server
pelias elastic stop stop elasticsearch server
pelias elastic status HTTP status code of the elasticsearch service
pelias elastic wait wait for elasticsearch to start up
```
### Import commands
The import commands import source data in to elasticsearch.
```bash
pelias import wof (re)import whosonfirst data
pelias import oa (re)import openaddresses data
pelias import osm (re)import openstreetmap data
pelias import polylines (re)import polylines data
pelias import transit (re)import transit data
pelias import all (re)import all data
```
### System commands
The system commands help debug issues with incorrectly set environment variables.
The `pelias system update` command can be used to ensure that the `pelias` command itself is up-to-date by pulling the latest source code from Github.
```bash
pelias system check ensure the system is correctly configured
pelias system env display environment variables
pelias system update update the pelias command by pulling the latest version
```
## Generic build workflow
The following shell script can be used to automate a build:
```bash
#!/bin/bash
set -x
# create directories
mkdir /code /data
# clone repo
cd /code
git clone https://github.com/pelias/docker.git
cd docker
# install pelias script
ln -s "$(pwd)/pelias" /usr/local/bin/pelias
# cwd
cd projects/portland-metro
# configure environment
sed -i '/DATA_DIR/d' .env
echo 'DATA_DIR=/data' >> .env
# run build
pelias compose pull
pelias elastic start
pelias elastic wait
pelias elastic create
pelias download all
pelias prepare all
pelias import all
pelias compose up
```
## View status of running containers
Once the build is complete, you can view the current status and port mappings of the Pelias docker containers:
```bash
pelias compose ps
```
## View logs and debug errors
You can inspect the container logs for errors by running:
```bash
pelias compose logs
```
## Make an example query
You can now make queries against your new Pelias build:
### API
- http://localhost:4000/v1/search?text=portland
- [http://localhost:4000/v1/search?text=1901 Main St](http://localhost:4000/v1/search?text=1901%20Main%20St)
- http://localhost:4000/v1/reverse?point.lon=-122.650095&point.lat=45.533467
### Placeholder
- http://localhost:4100/demo/#eng
### PIP (point in polygon)
- http://localhost:4200/-122.650095/45.533467
### Interpolation
- http://localhost:4300/demo/#13/45.5465/-122.6351

30
cmd/docker.sh

@ -0,0 +1,30 @@
#!/bin/bash
set -e;
function compose_pull(){ compose_exec pull; }
register 'compose' 'pull' 'update all docker images' compose_pull
function compose_logs(){ compose_exec logs $@; }
register 'compose' 'logs' 'display container logs' compose_logs
function compose_ps(){ compose_exec ps $@; }
register 'compose' 'ps' 'list containers' compose_ps
function compose_top(){ compose_exec top $@; }
register 'compose' 'top' 'display the running processes of a container' compose_top
function compose_exec(){ docker-compose $@; }
register 'compose' 'exec' 'execute an arbitrary docker-compose command' compose_exec
function compose_run(){ docker-compose run -T --rm $@; }
register 'compose' 'run' 'execute a docker-compose run command' compose_run
function compose_up(){ docker-compose up -d $@; }
register 'compose' 'up' 'start one or more docker-compose service(s)' compose_up
function compose_kill(){ docker-compose kill $@; }
register 'compose' 'kill' 'kill one or more docker-compose service(s)' compose_kill
function compose_down(){ docker-compose down; }
register 'compose' 'down' 'stop all docker-compose service(s)' compose_down

27
cmd/download.sh

@ -0,0 +1,27 @@
#!/bin/bash
set -e;
# per-source downloads
function download_wof(){ compose_run 'whosonfirst' npm run download; }
function download_oa(){ compose_run 'openaddresses' npm run download; }
function download_osm(){ compose_run 'openstreetmap' npm run download; }
function download_tiger(){ compose_run 'interpolation' npm run download-tiger; }
function download_transit(){ compose_run 'transit' npm run download; }
register 'download' 'wof' '(re)download whosonfirst data' download_wof
register 'download' 'oa' '(re)download openaddresses data' download_oa
register 'download' 'osm' '(re)download openstreetmap data' download_osm
register 'download' 'tiger' '(re)download TIGER data' download_tiger
register 'download' 'transit' '(re)download transit data' download_transit
# download all the data to be used by imports
function download_all(){
download_wof &
download_oa &
download_osm &
download_tiger &
download_transit &
wait
}
register 'download' 'all' '(re)download all data' download_all

31
cmd/elastic.sh

@ -0,0 +1,31 @@
#!/bin/bash
set -e;
function elastic_schema_drop(){ compose_run 'schema' node scripts/drop_index "$@" || true; }
function elastic_schema_create(){ compose_run 'schema' npm run create_index; }
function elastic_start(){ compose_exec up -d elasticsearch; }
function elastic_stop(){ compose_exec kill elasticsearch; }
register 'elastic' 'drop' 'delete elasticsearch index & all data' elastic_schema_drop
register 'elastic' 'create' 'create elasticsearch index with pelias mapping' elastic_schema_create
register 'elastic' 'start' 'start elasticsearch server' elastic_start
register 'elastic' 'stop' 'stop elasticsearch server' elastic_stop
# to use this function:
# if test $(elastic_status) -ne 200; then
function elastic_status(){ curl --output /dev/null --silent --write-out "%{http_code}" "http://${ELASTIC_HOST:-localhost:9200}" || true; }
# the same function but with a trailing newline
function elastic_status_newline(){ echo $(elastic_status); }
register 'elastic' 'status' 'HTTP status code of the elasticsearch service' elastic_status_newline
function elastic_wait(){
echo 'waiting for elasticsearch service to come up';
until test $(elastic_status) -eq 200; do
printf '.'
sleep 2
done
echo
}
register 'elastic' 'wait' 'wait for elasticsearch to start up' elastic_wait

27
cmd/import.sh

@ -0,0 +1,27 @@
#!/bin/bash
set -e;
# per-source imports
function import_wof(){ compose_run 'whosonfirst' npm start; }
function import_oa(){ compose_run 'openaddresses' npm start; }
function import_osm(){ compose_run 'openstreetmap' npm start; }
function import_polylines(){ compose_run 'polylines' npm start; }
function import_transit(){ compose_run 'transit' npm start; }
register 'import' 'wof' '(re)import whosonfirst data' import_wof
register 'import' 'oa' '(re)import openaddresses data' import_oa
register 'import' 'osm' '(re)import openstreetmap data' import_osm
register 'import' 'polylines' '(re)import polylines data' import_polylines
register 'import' 'transit' '(re)import transit data' import_transit
# import all the data to be used by imports
# note: running importers in parallel can cause issues due to high CPU & RAM requirements.
function import_all(){
import_wof
import_oa
import_osm
import_polylines
import_transit
}
register 'import' 'all' '(re)import all data' import_all

24
cmd/prepare.sh

@ -0,0 +1,24 @@
#!/bin/bash
set -e;
# per-source prepares
function prepare_polylines(){ compose_run 'polylines' bash ./docker_extract.sh; }
function prepare_interpolation(){ compose_run 'interpolation' bash ./docker_build.sh; }
function prepare_placeholder(){
compose_run 'placeholder' npm run extract
compose_run 'placeholder' npm run build
}
register 'prepare' 'polylines' 'export road network from openstreetmap into polylines format' prepare_polylines
register 'prepare' 'interpolation' 'build interpolation sqlite databases' prepare_interpolation
register 'prepare' 'placeholder' 'build placeholder sqlite databases' prepare_placeholder
# prepare all the data to be used by imports
function prepare_all(){
prepare_polylines &
prepare_placeholder &
wait
prepare_interpolation
}
register 'prepare' 'all' 'build all services which have a prepare step' prepare_all

11
cmd/system.sh

@ -0,0 +1,11 @@
#!/bin/bash
set -e;
function system_check(){ env_check; }
register 'system' 'check' 'ensure the system is correctly configured' system_check
function system_env(){ env; }
register 'system' 'env' 'display environment variables' system_env
function system_update(){ git -C $(dirname "${BASH_SOURCE[0]}") pull; }
register 'system' 'update' 'update the pelias command by pulling the latest version' system_update

6
images/elasticsearch/2.4/Dockerfile

@ -0,0 +1,6 @@
# base image
FROM elasticsearch:2.4
# configure plugins
RUN /usr/share/elasticsearch/bin/plugin install analysis-icu
RUN /usr/share/elasticsearch/bin/plugin install cloud-aws

23
images/elasticsearch/5.2.1/Dockerfile

@ -0,0 +1,23 @@
# see: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
# base image
FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.1
USER root
# environmental settings
ENV ES_JAVA_OPTS '-Xms512m -Xmx512m'
ENV xpack.security.enabled 'false'
ENV xpack.monitoring.enabled 'false'
ENV cluster.name 'pelias-dev'
ENV bootstrap.memory_lock 'true'
RUN echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
# configure plugins
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu
# elasticsearch config
ADD elasticsearch.yml /usr/share/elasticsearch/config/
RUN chown elasticsearch:elasticsearch config/elasticsearch.yml
# run as elasticsearch user
USER elasticsearch

6
images/elasticsearch/5.2.1/elasticsearch.yml

@ -0,0 +1,6 @@
xpack.security.enabled: false
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
node.master: true
node.data: true

25
images/valhalla/Dockerfile

@ -0,0 +1,25 @@
# base image
FROM pelias/baseimage
# grab all of the valhalla software from ppa
RUN apt-get update && \
apt-get install -y software-properties-common python-software-properties && \
add-apt-repository -y ppa:kevinkreiser/prime-server && \
add-apt-repository -y ppa:valhalla-routing/valhalla && \
apt-get update && \
apt-get install -y valhalla-bin && \
rm -rf /var/lib/apt/lists/*;
# change working dir
RUN mkdir -p /code/valhalla
WORKDIR /code/valhalla
# generate config
RUN valhalla_build_config \
--mjolnir-tile-dir '/data/valhalla' \
--mjolnir-tile-extract '/data/valhalla.tar' \
--mjolnir-timezone '/data/valhalla/timezones.sqlite' \
--mjolnir-admin '/data/valhalla/admins.sqlite' > valhalla.json
# build script
RUN echo 'valhalla_build_tiles -c valhalla.json /data/openstreetmap/*.osm.pbf; valhalla_export_edges --config valhalla.json > /data/polylines/pbf_extract.polyline;' > ./docker_build.sh

49
lib/cli.sh

@ -0,0 +1,49 @@
#!/bin/bash
set -e;
declare -a commands
declare -a actions
declare -a hints
declare -a functions
function register(){
commands+=("$1")
actions+=("$2")
hints+=("$3")
functions+=("$4")
}
function help(){
printf 'Usage: %s [command] [action] [options]\n\n' ${0}
for (( i = 0; i < ${#commands[@]}; ++i )); do
echo -e " ${commands[$i]}\t${actions[$i]}\t ${hints[$i]}"
done | column -ts $'\t'
echo
}
function cli(){
cmd="${1}"; shift || true
action="${1}"; shift || true
valid_command=false
valid_action=false
for (( i = 0; i < ${#commands[@]}; ++i )); do
if [ "${cmd}" = "${commands[$i]}" ]; then
valid_command=true
if [ "${action}" = "${actions[$i]}" ]; then
valid_action=true
"${functions[$i]}" "$@"
exit $?
fi
fi
done
echo
[ -z "${cmd}" ] || [ "$valid_command" = true ] || printf 'invalid command "%s"\n\n' "${cmd}"
[ -z "${action}" ] || [ "$valid_action" = true ] || printf 'invalid action "%s"\n\n' "${action}"
help
exit 1
}

52
lib/env.sh

@ -0,0 +1,52 @@
#!/bin/bash
set -e;
# disable verbose logging
ENV_DISPLAY_WARNINGS=false
# ensure the user environment is correctly set up
function env_check(){
if [ -z "${DATA_DIR}" ]; then
echo "You must set the DATA_DIR env var to a valid directory on your local machine."
echo
echo "Edit the '.env' file in this repository, update the DATA_DIR to a valid path and try again."
echo "Alternatively, you can set the variable in your environment using a command such as 'export DATA_DIR=/tmp'."
exit 1
elif [ ! -d "${DATA_DIR}" ]; then
printf "The directory specified by DATA_DIR does not exist: %s\n" ${DATA_DIR}
echo
echo "Edit the '.env' file in this repository, update the DATA_DIR to a valid path and try again."
echo "Alternatively, you can set the variable in your environment using a command such as 'export DATA_DIR=/tmp'."
exit 1
fi
}
# loads environment vars from a stream (such as a file)
# example: env_load_stream < .env
function env_load_stream(){
while IFS='=' read -r key value; do
([ -z $key ] || [ -z $value ]) && printf 'Invalid environment var "%s=%s"\n' $key $value && exit 1
if [ -z ${!key} ]; then
export "${key}=${value}"
elif $ENV_DISPLAY_WARNINGS; then
printf '[warn] skip setting environment var "%s=%s", already set "%s=%s"\n' $key $value $key ${!key}
fi
done
}
# ensure locale is correctly set?
# export LC_ALL=en_US.UTF-8
# load DATA_DIR and other vars from docker-compose .env file
# note: strips comments and empty lines
[ -f .env ] && env_load_stream < <(grep -v '^$\|^\s*$\#' .env)
# use the default compose file unless one was specified
# if [ -z "${COMPOSE_FILE}" ]; then
# if [ ! -f "docker-compose.yml" ]; then
# export COMPOSE_FILE="${BASEDIR}/docker-compose.yml"
# fi
# fi
# ensure the user env is correctly set up
env_check

22
pelias

@ -0,0 +1,22 @@
#!/bin/bash
set -e
# OSX comes bundled with versions of readlink, sed, parallel etc which are not
# compatible with the linux tools. Force OSX users to install the GNU
# compatible versions (prefixed with 'g', such as 'greadlink', 'gsed' etc.).
export CMD_READLINK='readlink'
if [[ "$OSTYPE" == "darwin"* ]]; then
if [ -x "$(command -v greadlink)" ]; then
CMD_READLINK='greadlink';
else
2>&1 echo 'OSX: you must install the gnu standard tooling using:'
2>&1 echo 'brew install coreutils'
fi
fi
# resolve path to this file (following symlinks) and load libs
BASEDIR=$( dirname $( ${CMD_READLINK} -f "${BASH_SOURCE[0]}" ) )
for f in ${BASEDIR}/lib/* ${BASEDIR}/cmd/*; do source $f; done
# cli runner
cli "$@"

2
projects/portland-metro/.env

@ -0,0 +1,2 @@
COMPOSE_PROJECT_NAME=pelias
DATA_DIR=/tmp/pelias/portland-metro

35
projects/portland-metro/README.md

@ -0,0 +1,35 @@
# Portland Metro Area
This project is configured to download/prepare/build a complete Pelias installation for Portland, Oregon.
It is intended as an example for other projects, feel free to copy->paste these files to a new project directory to kick-start your own project.
# Setup
Please refer to the instructions at https://github.com/pelias/docker in order to install and configure your docker environment.
The minimum configuration required in order to run this project are [installing prerequisites](https://github.com/pelias/docker#prerequisites), [install the pelias command](https://github.com/pelias/docker#installing-the-pelias-command) and [configure the environment](https://github.com/pelias/docker#configure-environment).
Please ensure that's all working fine before continuing.
# Run a Build
To run a complete build, execute the following commands:
```bash
pelias compose pull
pelias elastic start
pelias elastic wait
pelias elastic create
pelias download all
pelias prepare all
pelias import all
pelias compose up
```
# Make an Example Query
You can now make queries against your new Pelias build:
http://localhost:4000/v1/search?text=pdx

114
projects/portland-metro/docker-compose.yml

@ -0,0 +1,114 @@
version: '3'
networks:
default:
driver: bridge
volumes:
libpostaldata:
driver: local
services:
libpostal_baseimage:
image: pelias/libpostal_baseimage
container_name: pelias_libpostal_baseimage
libpostal:
image: pelias/go-whosonfirst-libpostal
container_name: pelias_libpostal
restart: always
ports: [ "8080:8080" ]
schema:
image: pelias/schema:portland-synonyms
container_name: pelias_schema
volumes:
- "./pelias.json:/code/pelias.json"
- "./synonyms/custom_name.txt:/code/pelias/schema/synonyms/custom_name.txt"
- "./synonyms/custom_street.txt:/code/pelias/schema/synonyms/custom_street.txt"
api:
image: pelias/api:master
container_name: pelias_api
restart: always
environment: [ "PORT=4000" ]
ports: [ "4000:4000" ]
volumes:
- "./pelias.json:/code/pelias.json"
- "libpostaldata:/usr/share/libpostal"
placeholder:
image: pelias/placeholder:master
container_name: pelias_placeholder
restart: always
environment: [ "PORT=4100" ]
ports: [ "4100:4100" ]
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
whosonfirst:
image: pelias/whosonfirst:master
container_name: pelias_whosonfirst
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
openstreetmap:
image: pelias/openstreetmap:master
container_name: pelias_openstreetmap
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
openaddresses:
image: pelias/openaddresses:master
container_name: pelias_openaddresses
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
transit:
image: pelias/transit:master
container_name: pelias_transit
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
polylines:
image: pelias/polylines:master
container_name: pelias_polylines
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
interpolation:
depends_on: [ "libpostal_baseimage" ]
image: pelias/interpolation:master
container_name: pelias_interpolation
restart: always
environment: [ "PORT=4300" ]
ports: [ "4300:4300" ]
volumes:
- "./pelias.json:/code/pelias.json"
- "libpostaldata:/usr/share/libpostal"
- "${DATA_DIR}:/data"
pip:
image: pelias/pip-service:master
container_name: pelias_pip-service
restart: always
environment: [ "PORT=4200" ]
ports: [ "4200:4200" ]
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
document-service:
image: pelias/document-service:master
container_name: pelias_document-service
restart: always
ports: [ "5000:5000" ]
volumes:
- "./pelias.json:/code/pelias.json"
- "${DATA_DIR}:/data"
elasticsearch:
image: pelias/elasticsearch
container_name: pelias_elasticsearch
restart: always
ports: [ "9200:9200", "9300:9300" ]
volumes:
- "${DATA_DIR}/elasticsearch:/usr/share/elasticsearch/data"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add: [ "IPC_LOCK" ]

162
projects/portland-metro/pelias.json

@ -0,0 +1,162 @@
{
"logger": {
"level": "info",
"timestamp": false
},
"esclient": {
"hosts": [
{ "host": "elasticsearch" }
]
},
"elasticsearch": {
"settings": {
"index": {
"refresh_interval": "10s",
"number_of_replicas": "0",
"number_of_shards": "1"
}
}
},
"api": {
"textAnalyzer": "libpostal",
"services": {
"pip": { "url": "http://pip-service:4200" },
"libpostal": { "url": "http://libpostal:8080" },
"placeholder": { "url": "http://placeholder:4100" },
"interpolation": { "url": "http://interpolation:4300" }
},
"defaultParameters": {
"focus.point.lat": 45.52,
"focus.point.lon": -122.67
}
},
"imports": {
"adminLookup": {
"enabled": true
},
"geonames": {
"datapath": "/data/geonames",
"countryCode": "ALL"
},
"openstreetmap": {
"download": [
{ "sourceURL": "https://s3.amazonaws.com/metro-extracts.nextzen.org/portland_oregon.osm.pbf" }
],
"leveldbpath": "/tmp",
"datapath": "/data/openstreetmap",
"import": [{
"filename": "portland_oregon.osm.pbf"
}]
},
"openaddresses": {
"datapath": "/data/openaddresses",
"files": [
"us/or/portland_metro.csv",
"us/or/city_of_salem.csv",
"us/or/marion_and_polk.csv",
"us/or/marion.csv",
"us/or/hood_river.csv",
"us/wa/city_of_richland.csv",
"us/wa/clark.csv"
]
},
"polyline": {
"datapath": "/data/polylines",
"files": [ "extract.0sv" ]
},
"whosonfirst": {
"datapath": "/data/whosonfirst",
"importVenues": false,
"importPostalcodes": true,
"importPlace": [
"85688513",
"85688623"
]
},
"transit": {
"datapath": "/data/transit",
"feeds": [
{
"layerId": "bikeshare",
"url": "http://biketownpdx.socialbicycles.com/opendata/station_information.json",
"filename": "BIKETOWN-hubs.json",
"agencyId": "BIKETOWN",
"agencyName": "BIKETOWN",
"layerName": "bikeshare"
},
{
"layerId": "stops",
"url": "http://developer.trimet.org/schedule/gtfs.zip",
"filename": "TRIMET-stops.txt",
"agencyId": "TRIMET",
"agencyName": "TriMet",
"layerName": "Stop"
},
{
"layerId": "stops",
"url": "http://www.c-tran.com/images/Google/GoogleTransitUpload.zip",
"filename": "C-TRAN-stops.txt",
"agencyId": "C-TRAN",
"agencyName": "C-TRAN",
"layerName": "Stop"
},
{
"layerId": "stops",
"url": "http://oregon-gtfs.com/gtfs_data/rideconnection-or-us/rideconnection-or-us.zip",
"filename": "RIDECONNECTION-stops.txt",
"agencyId": "RIDECONNECTION",
"agencyName": "Ride Connection",
"layerName": "Stop"
},
{
"layerId": "stops",
"url": "http://cherriots.org/developer/gtfs.zip",
"filename": "CHERRIOTS-stops.txt",
"agencyId": "CHERRIOTS",
"agencyName": "Cherriots",
"layerName": "Stop"
},
{
"layerId": "stops",
"url": "http://oregon-gtfs.com/gtfs_data/sandy-or-us/sandy-or-us.zip",
"filename": "SAM-stops.txt",
"agencyId": "SAM",
"agencyName": "SAM",
"layerName": "Stop"
},
{
"layerId": "stops",
"url": "http://data.trilliumtransit.com/gtfs/wilsonville-or-us/wilsonville-or-us.zip",
"filename": "SMART-stops.txt",
"agencyId": "SMART",
"agencyName": "SMART",
"layerName": "Stop"
},
{
"layerId": "stops",
"url": "http://oregon-gtfs.com/gtfs_data/swanisland-or-us/swanisland-or-us.zip",
"filename": "SWAN-stops.txt",
"agencyId": "SWAN",
"agencyName": "Swan Island",
"layerName": "Stop"
}
]
},
"interpolation": {
"download": {
"tiger": {
"datapath": "/data/tiger",
"states": [
{ "state_code": 41, "county_code": 67, "name": "Washington County, OR" },
{ "state_code": 41, "county_code": 51, "name": "Multnomah County, OR" },
{ "state_code": 41, "county_code": 5, "name": "Clackamas County, OR" },
{ "state_code": 41, "county_code": 71, "name": "Yamhill County, OR" },
{ "state_code": 41, "county_code": 47, "name": "Marion County, OR" },
{ "state_code": 41, "county_code": 53, "name": "Polk County, OR" },
{ "state_code": 53, "county_code": 11, "name": "Clark County, WA" }
]
}
}
}
}
}

138
projects/portland-metro/synonyms/custom_name.txt

@ -0,0 +1,138 @@
shopping,store,shop,retail,grocery,mall
# Transit Centers
Oregon City Transit Center,Oregon City TC,OCTC
Barbur Blvd Transit Center,BBTC,Barbur Blvd TC,Barbur TC
Beaverton Transit Center,BTC,Beaverton TC
Clackamas Town Center Transit Center,CTC,Clackamas TC,Clackamas Town Center TC,Clackamas Transit Center
Gateway/NE 99th Ave Transit Center,GTC,Gateway TC,Gateway Transit Center
Gresham Central Transit Center,GCTC,Gresham Central TC,Gresham TC
Hillsboro Center/SE 3rd Ave Transit Center, HCTC, Hillsboro TC, Hillsboro Transit Center
Hollywood/NE 42nd Ave Transit Center,HTC, HOllywood TC,HOllywood Transit Center
Lake Oswego Transit Center, LOTC,Lake Oswego TC
N Lombard Transit Center, Lombard Transit Center, Lombard TC,NLTC
Parkrose/Sumner Transit Center, Parkrose Transit Center,Parkrose TC,PRSTC,PTC
Rose Quarter Transit Center,RQTC, Rose Quarter TC
Sunset Transit Center,Sunset TC, STC
Tigard Transit Center,Tigard TC,TTC
Washington Square Transit Center,WSTC,Washington Square TC
Willow Creek/SW 185th Ave Transit Center, Willow Creek TC, Willow Creek Transit Center, WCTC
# Rail Stations
Yamhill District MAX Station,YAMX
Washington/SE 12th Ave MAX Station, WASX
Tuality Hospital/SE 8th Ave MAX Station, TUAX
Skidmore Fountain MAX Station, SKIX
SE Powell Blvd MAX Station,POWX
SE Holgate BLvd MAX Station,HOLX
SE Fuller Rd MAX Station,FULX
SE Flavel St MAX Station,FLAX
SE Division St MAX Station,DIVX
Ruby Junction/E 197th Ave MAX Station,RUBX
Rosa Parks MAX Station,N Portland Blvd MAX Station,ROSX,RPX
Rockwood/E 188th Ave MAX Station,ROCX
Quatama/NW 205th Ave MAX Station,QUAX
Providence Park MAX Station,Civic Stadium MAX Station,JELD-WEN Field MAX Station,PGE Park MAX Station,PPX,PROX
Portland Int'l Airport MAX Station; PDXX
Pioneer Square South MAX Station,PCSS MAX
Pioneer Square North MAX Station,PCSN MAX
Parkrose/Sumner TC MAX Station,Parkrose MAX Station,PTC
Overlook Park MAX Station,OVEX
Orenco/NW 231st Ave MAX Station,OREX
Old Town/Chinatown MAX Station,OLDX
Oak/SW 1st Ave MAX Station,OAKX
N Prescott St MAX Station,PREX
N Killingsworth St MAX Station,KILX
NE 60th Ave MAX Station,60X
NE 82nd Ave MAX Station,82X
Mt Hood Ave MAX Station,MTHX
Millikan Way MAX Station,MILX
Merlo Rd/SW 158th Ave MAX Station,MERX
Lloyd Center/NE 11th Ave MAX Station,LLOX
Library/SW 9th Ave MAX Station,LIBX
Lents Town Center/Foster Rd MAX Station,LENX
Kings Hill/SW Salmon St MAX Station,KINX
Kenton/N Denver Ave MAX Station,KENX
Interstate/Rose Quarter MAX Station,IRQ MAX,IRQ
Hawthorn Farm MAX Station,HAWX
Goose Hollow/SW Jefferson St MAX Station,GHX,GOOX
Galleria/SW 10th Ave MAX Station,GALX
Fair Complex/Hillsboro Airport MAX Station,FAIX
Expo Center MAX Station,EXPX
E 162nd Ave MAX Station,162X
E 172nd Ave MAX Station,172X
E 181st Ave MAX Station,181X
E 102nd Ave MAX Station,102X
E 122nd Ave MAX Station,122X
E 148th Ave MAX Station,148X
Elmonica/SW 170th Ave MAX Station,ELMX,Elmonica
Delta Park/Vanport MAX Station,DELX,DPV MAX,DPX
Convention Center MAX Station,CCX,CONX
Cleveland Ave MAX Station,CLEX
Cascades MAX Station,CASX
Beaverton Creek MAX Station,BCRX
Beaverton Central MAX Station,BEAX
Albina/Mississippi MAX Station,ALBX
# TriMet Facilities
ATP/Nela,Nela
Center Street Bus Maintenance, Center Garage
Elmonica Operations Facility,Elmo,Elmonica
Merlo Operations Facility,Merlo Garage, Merlo
Powell Operations Facility,Powell Garage
Transit Mobility Center,ATP Mobility Center,Mobility Center
TriMet Administration Headquarters,Admin Building,Administration Building,Harrison Square,TriMet Administrative Offices
TriMet Operations Headquarters,Center St, Center Street,TriMet Lost & Found
#Trimet Ticket Office,TTO
Vintage Trolley Barn,Trolley Barn
# Misc Landmarks
# FXP: removed psu and pdx, since they led to ambig hits
Oregon Zoo,Zoo,Washington Park Zoo
Portland State University,Portland State
Portland International Raceway,PIR
Portland International Airport,Portland Airport
Portland Community College - Cascade,PCC Cascade
Portland Community College - Rock Creek,PCC Rock Creek
Portland Community College - Southeast,PCC Southeast, PCC Southeast Center
Portland Community College - Sylvania,PCC Sylvania
Portland Community College - Willow Creek Center,PCC Willow Creek Center
Pioneer Courthouse Square,Downtown Portland,PCS
Oregon Rail Heritage Center,ORHF
Oregon Museum of Science and Industry,OMSI
Oregon Health & Science University,OHSU,Oregon Health Sciences University
Oregon College of Oriental Medicine,OCOM
Moda Center at the Rose Quarter,Moda Center,Rose Garden Arena
Clackamas Community College,CCC
Clackamas Community College Harmony Campus, CCCH
Gordon Faber Recreation Complex,Hillsboro Stadium,Ron Tonkin Field
Providence Park,Civic Stadium,JELD-WEN Field,PGE Park
OSHU Collaborative Life Sciences Building, Skourtes Tower, CLSB, School of Dentistry
Aging & Disability, DHS
African Youth and Community Organization, AYCO
north, N
south, S
east, E
west, W
north west => northwest
north east => northeast
south west => southwest
south east => southeast
northwest, NW
northeast, NE
southwest, SW
southeast, SE
Portland, Portland City Center
Int'l, international
clackamus, clackamas
clack, clackamas
p-town, portland
casscade, cascade
# street names
Tualatin Valley Hwy,TV Hwy,TVH,TVHWY
Beaverton-Hillsdale Hwy,Beaverton- Hillsdale Hwy,BH Hwy,BHH
MLK,M L King,Martin Luther King

133
projects/portland-metro/synonyms/custom_street.txt

@ -0,0 +1,133 @@
Tualatin Valley Hwy,TV Hwy,TVH,TVHWY
Beaverton Hillsdale HWY,Beaverton-Hillsdale HWY,BH HYW,BHH
Beaverton Hillsdale,Beaverton-Hillsdale,BH
Martin Luther King,ML King,M L King,MLK
Doernbecher,Dornbecher,Doernbecker,Dornbecker,Dornbecka
Native American Youth => NAYA
Transit Center => TC
Park Ride => PR
Park & Ride => PR
Park and Ride => PR
caeser,cesar,chavez,39th
metro,portland
oregon,or
carshare,car2go,Zipcar,Flexcar,GetAround,ucarshare,uhaul,hertz,wecar,enterprise
ticket,fare,outlet,store,retail
dmv,motor,vehicles
court house => courthouse
court-house => courthouse
cc,court,courts,courth,courthouse
law,jail,prison,jail,courthouse,enforcement,juvenile,justice,juvenile,correctional
jewvi,juvi,juvenile
first,1st
second,2nd
third,3rd
forth,4th
fifth,5th
sixth,6th
seventh,7th
eighth,8th
nineth,9th
tenth,10th
eleventh,11th
twelfth,12th
thirteenth,13th
fourteenth,14th
fifteenth,15th
sixteenth,16th
seventeenth,17th
eighteenth,18th
nineteenth,19th
twentyth,20th
twentyfirst,21st
twentysecond,22nd
twentythrid,23rd
# USPS Suffix Abbreviations
# http://www.usps.com/ncsc/lookups/abbreviations.html#suffix
st,st.,street,str
ave,ave.,av,av.,avenue,AVNUE,AVENUE,AVENU
dr,drive,dr.,drv,driv,drives
BOULEVARD,BLVD,BOULEVARD,BOUL,BOULEVARD,BOULEVARD,BOULV
BRIDGE,BRDGE,BRG
BYPASS,BYP,BYPA,BYPAS,BYPS
CLIFF,CLF,CLIFFS,CLFS
CLUB,CLB
COMMON,CMN
CORNER,COR,CORNERS,CORS
CENTER,CEN,CENT,CENTER,CENTR,CENTRE,CNTER,CNTR,CTR,CENTERS,CTRS
CIR,CIRC,CIRCL,CIRCLE,CRCL,CIRCLE,CRCLE
COURT,CRT,CT,COURTS
CTSCK,CR,CREEK,CRK
CROSSING,CRSSING,CRSSNG,XING
CRECENT,CRES,CRESCENT,CRESENT,CRSCNT,CRSENT,CRSNT
EXP,EXPR,EXPRESS,EXPRESSWAY,EXPW,EXPY
FALL,FALLS,FLS
FERRY,FRRY,FRY
FIELD,FLD,FIELDS,FLDS
FREEWAY,FREEWY,FRWAY,FRWY,FWY
GARDEN,GARDN,GDN,GRDEN,GRDN,GARDENS,GDNS,GRDNS
GATEWAY,GATEWY,GATWAY,GTWAY,GTWY
GROV,GROVE,GRV,GROVES
HARB,HARBOR,HARBR,HBR,HRBOR,HARBORS
HEIGHT,HEIGHTS,HGTS,HT,HTS
HIGHWAY,HIGHWY,HIWAY,HIWY,HWAY,HWY
HILL,HL,HILLS,HLS
IS,ISLAND,ISLND,ISLANDS,ISLNDS,ISS,ISLE,ISLES
JCT,JCTION,JCTN,JUNCTION,JUNCTN,JUNCTON,JCTNS,JCTS,JUNCTIONS
LAKE,LK,LAKES,LKS
LANDING,LNDG,LNDNG
LA,LANE,LANES,LN
LOOP,LOOPS,LP
MANOR,MNR,MANORS,MNRS
MDW,MEADOW,MDWS,MEADOWS,MEDOWS
MILL,ML,MILLS,MLS
MNT,MOUNT,MT
MNTAIN,MNTN,MOUNTAIN,MOUNTIN,MTIN,MTN,MNTNS,MOUNTAINS
ORCH,ORCHARD,ORCHRD
PARK,PK,PRK,PARKS
PARKWAY,PARKWY,PKWAY,PKWY,PKY,PARKWAYS,PKWYS
PLAZA,PLZ,PLZA
PLACE,PL
PR,PRAIRIE,PRARIE,PRR
RANCH,RANCHES,RNCH,RNCHS
RDG,RDGE,RIDGE,RDGS,RIDGES
RIV,RIVER,RIVR,RVR
RD,ROAD,RDS,ROADS
RTESHLS,SHOALS
SHOAR,SHORE,SHR,SHOARS,SHORES,SHRS
SPG,SPNG,SPRING,SPRNG,SPGS,SPNGS,SPRINGS,SPRNGS
SQ,SQR,SQRE,SQU,SQUARE,SQRS,SQUARES
STA,STATION,STATN,STN
STREAM,STREME,STRM
ST,STR,STREET,STRT,STREETS
SMT,SUMIT,SUMITT,SUMMIT
TER,TERR,TERRACE
TR,TRAIL,TRAILS,TRL,TRLS
TUNEL,TUNL,TUNLS,TUNNEL,TUNNELS,TUNNL
UN,UNION,UNIONS
VALLEY,VALLY,VLLY,VLY,VALLEYS,VLYS
VDCT,VIA,VIADCT,VIADUCT
VIEW,VW,VIEWS,VWS
VILL,VILLAG,VILLAGE,VILLG,VILLIAGE,VLG,VILLAGES,VLGS
VILLE,VL
VIS,VIST,VISTA,VST,VSTA
WAY,WY,WAYS
north, N
south, S
east, E
west, W
north west => northwest
north east => northeast
south west => southwest
south east => southeast
northwest, NW
northeast, NE
southwest, SW
southeast, SE
broadway st => broadway
broadway street => broadway
Loading…
Cancel
Save