pelias/api - api - CogtoSrc

Commit Graph

Author	SHA1	Message	Date
Julian Simioni	df8f1bb332	feat(autocomplete) add hard distance filter to short focus.point queries Short autocomplete inputs are very difficult to serve in a performant and low-latency way. With shorter inputs, many more documents match for just about any input string. In our testing, one to three character input texts generally match up to 100 million documents out of a 560 million document full planet build. There's really no way to make scoring 100 million documents fast, so in order to achieve acceptable performance (ideally, <100ms P99 latency), it's worth looking at ways to either avoid querying Elasticsearch or reducing the scope of autocomplete queries. Short autocomplete queries without a focus.point parameter can be cached. There are only 47,000 possible 1-3 character alphanumerical inputs. At this time, caching is outside the scope of Pelias itself but can easily be implemented with Varnish, Nginx, Fastly, Cloudfront, and lots of other tools and services. Queries with a `focus.point` are effectively uncachable however, since the coordinate chosen will often be unique. This PR uses the `focus.point` coordinate to build a hard filter limiting the search to documents only within a certain radius of the coordinate. This can reduce the number of documents searched and improve performance, while still returning results that are useful. It takes two parameters, driven by `pelias-config`: - `api.autocomplete.focusHardLimitTextLength': the maximum length of text for which a hard distance filter will be constructed - `api.autocomplete.focusHardLimitMultiplier`: the length of the input text will be multiplied by this number to get the total hard filter radius in kilometers. For example, with `focusHardLimitTextLength` 4, and `focusHardLimitMultiplier` 50, the following hard filters would be constructed: \| text length \| max distance \| \| ---- \| ----\| \| 1 \| 50 \| \| 2 \| 100 \| \| 3 \| 150 \| \| 4+ \| unlimited \|	6 years ago
Julian Simioni	003c182df5	feat(log): add structured data to address parser logs These logs have been very confusing to us for a long time, and we have never bothered to collect enought data to understand them better.	6 years ago
Julian Simioni	4adf4b3dd7	feat(queries): Normalize all query names They should start with the endpoint (ideally), and address_search_using_ids should not have the same query name as 'search_fallback'.	6 years ago
Julian Simioni	d681a114d6	feat(log): Remove most unstructured controller logs These are nose now, the structured logs have much better info	6 years ago
Julian Simioni	49a988ce37	Remove non-JSON log lines We can have logs with a lot more fidelity by logging structured JSON directly.	6 years ago
Julian Simioni	c20737f458	fix(boundary.country): use boundary.country as filter By definition, all boundary.country query matches will either be identical, or not a match. Thus, it does not make sense to put the query clause for boundary.country in the `must` section of the query. In theory, because our queries would generally combine this `must` clause with others, there shouldn't be any performance improvement (or regression) from this change. However, semantically, this clause fits better as a `filter`, and in the case of a bug causing a degenerate query with the `boundary.country` query clause as the only one under the `must` section, this would have a big impact.	6 years ago
Julian Simioni	800eb8ca03	Move lots of logging from info to debug These are log lines that are not really useful in a production context, and just create a lot of noise.	7 years ago
Julian Simioni	561f07950b	Remove focus_selected_layers query view This query extends the standard focus query view with hardcoded layers for which the query applies. The intent was to apply the focus scoring only to non-admin areas, but the list of layers was already out of date, as it was missing streets. The query is fundamentally problematic with custom layers as well.	7 years ago
Julian Simioni	204f5297de	Remove all leading newlines in files	7 years ago
Julian Simioni	02aac0c13e	Remove use stricts	7 years ago
Stephen Hess	02cfa36d24	switched structured query tests away from fixtures	7 years ago
Adam Rousell	f86ee4a40f	Added postalcode rules for node-postal call and fallbacks	8 years ago
Stephen Hess	447d0f74b9	don't serialize input:layers when passing to query	8 years ago
Stephen Hess	f176b86657	drastically revised comments	8 years ago
Stephen Hess	7037573d8b	removed task comment	8 years ago
Stephen Hess	1fef7ac41c	removed unreachable paths in query/search	8 years ago
Stephen Hess	1f4a29bb0d	removed unused variable	8 years ago
Stephen Hess	99a1868c47	removed population/popularity functions from search using ids	8 years ago
Stephen Hess	1d5151dd49	updated meta.type to 'fallback'	8 years ago
Stephen Hess	5bcfbe76bc	set search_using_ids query_type to 'original'	8 years ago
Stephen Hess	ea3572539f	combined non-county layers into one	8 years ago
Stephen Hess	74419bddc1	added support for focus-only function	8 years ago
Stephen Hess	568a258406	code cleanup, shorter function syntax	8 years ago
Stephen Hess	386897c2c4	added query for address search with ids	8 years ago
Lily He	2a73bb34ac	combined checks for boundary circle radius into one block	8 years ago
Lily He	57b133ba42	check and set default for undefined boundary circle radius in reverse	8 years ago
Stephen Hess	67360a4d0b	added geonames_deprecation to reverse sanitizing	8 years ago
Stephen Hess	10b1d28201	only query ES for non-coarse layers on non-coarse reverse requests	8 years ago
Diana Shkolnikov	242d40760d	add logging of query input types	8 years ago
Diana Shkolnikov	a52691c61e	feat: search by postalcode	8 years ago
Julian Simioni	abb4fa6113	Adjust autocomplete focus boost This lowers the autocomplete focus boost from 40 to 15. The idea is that, because the boost for population is 20, the focus can't possibly override a popular city, but it can come close. On our default dev build, no acceptance tests fail but the san francisco autocomplete test now passes! It's not clear if this fixes WOF venue issues yet, since the latest build there failed :(	8 years ago
Diana Shkolnikov	4f999adce1	fix: empty values in parsed text caused exceptions	8 years ago
Diana Shkolnikov	4fa7fae546	fix: add parser and query param logging	8 years ago
Diana Shkolnikov	bccae79e70	get rid of parsed_text logging because privacy	8 years ago
Diana Shkolnikov	f7287c448e	fix: minimize debug logging	8 years ago
Stephen Hess	31f5cc2537	bumped query version, updated query name changes + boosts	8 years ago
Stephen Hess	be2a336038	updated naming from component to structured	8 years ago
Stephen Hess	7670eef3fb	added support for city+country inputs	8 years ago
Stephen Hess	d2c32b5afd	added unit tests for component_geocoding query started folder structure for fixture as it's getting unwieldy	8 years ago
Stephen Hess	08f4aa4b95	renamed files to be more indicative of what the function is	8 years ago
Stephen Hess	d03a8c4586	added support for component geocoding - created `/component` route - broke out trimByGranularityComponent but could conceivably be combined with existing - added `address` support to text_parser - added `component` sanitizer wrapper	8 years ago
Stephen Hess	5dedcf5032	changed method name	8 years ago
Stephen Hess	acb669b7b3	added support for optional country	8 years ago
Stephen Hess	54b70c7d3c	removed lodash requirement	8 years ago
Stephen Hess	b8bd6c1252	enabled city/state only inputs for fallback queries	8 years ago
Julian Simioni	78cc4a5de3	Add boundary.rect handling to query/autocomplete	8 years ago
Stephen Hess	4dbaff6589	removed hard-coded `size` effectively reverts https://github.com/pelias/api/pull/684	8 years ago
Stephen Hess	7555095399	added `boost:address` and `boost:street` to search defaults bumped query version to 8.8.0 defaults are `10` and `5`, respectively	8 years ago
Stephen Hess	c7b07c271e	updated hardcoded ES query size to 50	8 years ago
Stephen Hess	b408b96948	hardcode size to 100 for FallbackQuery	8 years ago

1 2 3 4 5 ...

291 Commits (focus-point-hard-distance-filter)