pelias/api - api - CogtoSrc

Author	SHA1	Message	Date
Julian Simioni	df8f1bb332	feat(autocomplete) add hard distance filter to short focus.point queries Short autocomplete inputs are very difficult to serve in a performant and low-latency way. With shorter inputs, many more documents match for just about any input string. In our testing, one to three character input texts generally match up to 100 million documents out of a 560 million document full planet build. There's really no way to make scoring 100 million documents fast, so in order to achieve acceptable performance (ideally, <100ms P99 latency), it's worth looking at ways to either avoid querying Elasticsearch or reducing the scope of autocomplete queries. Short autocomplete queries without a focus.point parameter can be cached. There are only 47,000 possible 1-3 character alphanumerical inputs. At this time, caching is outside the scope of Pelias itself but can easily be implemented with Varnish, Nginx, Fastly, Cloudfront, and lots of other tools and services. Queries with a `focus.point` are effectively uncachable however, since the coordinate chosen will often be unique. This PR uses the `focus.point` coordinate to build a hard filter limiting the search to documents only within a certain radius of the coordinate. This can reduce the number of documents searched and improve performance, while still returning results that are useful. It takes two parameters, driven by `pelias-config`: - `api.autocomplete.focusHardLimitTextLength': the maximum length of text for which a hard distance filter will be constructed - `api.autocomplete.focusHardLimitMultiplier`: the length of the input text will be multiplied by this number to get the total hard filter radius in kilometers. For example, with `focusHardLimitTextLength` 4, and `focusHardLimitMultiplier` 50, the following hard filters would be constructed: \| text length \| max distance \| \| ---- \| ----\| \| 1 \| 50 \| \| 2 \| 100 \| \| 3 \| 150 \| \| 4+ \| unlimited \|	6 years ago
Julian Simioni	49a988ce37	Remove non-JSON log lines We can have logs with a lot more fidelity by logging structured JSON directly.	6 years ago
Julian Simioni	c20737f458	fix(boundary.country): use boundary.country as filter By definition, all boundary.country query matches will either be identical, or not a match. Thus, it does not make sense to put the query clause for boundary.country in the `must` section of the query. In theory, because our queries would generally combine this `must` clause with others, there shouldn't be any performance improvement (or regression) from this change. However, semantically, this clause fits better as a `filter`, and in the case of a bug causing a degenerate query with the `boundary.country` query clause as the only one under the `must` section, this would have a big impact.	6 years ago
Julian Simioni	561f07950b	Remove focus_selected_layers query view This query extends the standard focus query view with hardcoded layers for which the query applies. The intent was to apply the focus scoring only to non-admin areas, but the list of layers was already out of date, as it was missing streets. The query is fundamentally problematic with custom layers as well.	7 years ago
Julian Simioni	02aac0c13e	Remove use stricts	7 years ago
Diana Shkolnikov	4fa7fae546	fix: add parser and query param logging	8 years ago
Julian Simioni	78cc4a5de3	Add boundary.rect handling to query/autocomplete	8 years ago
Diana Shkolnikov	9ff383cc2b	change the query module interfaces back to simple functions	8 years ago
Diana Shkolnikov	3888ae0385	change query module interfaces to allow for additional meta data	8 years ago
Stephen Hess	484af8c044	changed from `text_parser` to `text_parser_addressit` this ensures that `clean.parsed_text` is correctly converted to query as expected	8 years ago
Stephen Hess	01d0338486	added text_parser that uses addressit format for autocomplete rather than wholesale converting to libpostal in one release, the decision was made to only use libpostal for /search and not /autocomplete. Until such time that libpostal can be used for parsing autocomplete queries, text_parser_autocomplete.js will contain the converter between addressit and internal parsing format.	8 years ago
Julian Simioni	c5e5bbcf56	Add boundary.country filter to /v1/autocomplete	9 years ago
Julian Simioni	aa7942cbb9	Set up layer filtering for autocomplete and reverse This was missed by me when working on https://github.com/pelias/api/pull/580, but caught by the acceptance tests! Unfortunately it was caught after going to production.	9 years ago
missinglink	979aab1ac3	ensure that problematic single grams are removed from the query	9 years ago
missinglink	ee73774c89	add tokenizer, refactor how we determine if a token is 'complete' or 'incomplete'	9 years ago
missinglink	2398f05f8d	fix borough matching for both autocomplete and search endpoints	9 years ago
missinglink	ca0c51b0fd	don't strip single digits from query	9 years ago
missinglink	01a3233a7b	add a view to boost exact matches	9 years ago
missinglink	e40c9ef326	increase focus weight from 10->40 and simplify population/popularity subview	9 years ago
Peter Johnson	0abda099a6	bugfixes and more tests	9 years ago
Diana Shkolnikov	f3ba492e05	Remove all usage of alpha3/admin0/admin1, update query building to reflect new names in pelias-query	9 years ago
missinglink	8a213b70bb	simplify pass 3	9 years ago
missinglink	8474dede20	simplify pass 2	9 years ago
missinglink	6711aa45e0	simplify pass 1	9 years ago
missinglink	9ceb8f37d2	more tweaks	9 years ago
Peter Johnson	3236d97d4a	switch back to using standard analyzer for ngrams must condition	9 years ago
Peter Johnson	22fa0d0458	hack hack hack	9 years ago
Peter Johnson	3d29b09805	more tweaks	9 years ago
Peter Johnson	b974f304e7	trim	9 years ago
Peter Johnson	661d19ccf2	more tweaks	9 years ago
Peter Johnson	a022e62acb	more tweaks	9 years ago
Peter Johnson	609fc4e699	enable text parsing	9 years ago
Peter Johnson	95f2d367c2	enable text parsing	9 years ago
Peter Johnson	cf9c56928c	fix	9 years ago
Peter Johnson	35d48dc154	autocomplete tweaks	9 years ago
Diana Shkolnikov	9fa5fc5a77	calcSize became middleware (exposed and fixed bug in query defaults)	9 years ago
Diana Shkolnikov	54187dde67	Add dedupe middleware Dedupe middleware removes __exact__ dupes and truncates the results to the specified size.	9 years ago
Peter Johnson	49d37eafed	one config per query	9 years ago
Peter Johnson	cb679888ff	better population and popularity boosting	9 years ago
Peter Johnson	80f419846d	admin boost for sqrt(popularity), autocomplete focus not restricted to phrase matches	9 years ago
Peter Johnson	348d2e70e2	more conformance	9 years ago
Julian Simioni	f3acf3b308	Check for numeric value of lat/lon to avoid null island bug Using the check-types module, check that lat/lon values are numbers, instead of checking their truthyness, to ensure that queries for null island work correctly.	9 years ago
Julian Simioni	dd9654657e	Use flat clean structure in query/autocomplete.js	9 years ago
Peter Johnson	36185197aa	move query defaults to API repo	9 years ago
Peter Johnson	d568c031a3	move stringify logic to tests, code clean up	9 years ago
Peter Johnson	0d00e19825	remove debugging	9 years ago
Peter Johnson	fa209c9b50	add autocomplete route, further query clean up	9 years ago