diff --git a/README.md b/README.md index 6a84396..56a7e88 100644 --- a/README.md +++ b/README.md @@ -3,23 +3,21 @@ Search Library

- -

Web's fastest and most memory-flexible full-text search library with zero dependencies.

-When it comes to raw search speed FlexSearch outperforms every single searching library out there and also provides flexible search capabilities like multi-word matching, phonetic transformations or partial matching. -Depending on the used options it also provides the most memory-efficient index. Keep in mind that updating and/or removing existing items from the index has a significant cost. When your index needs to be updated very often then BulkSearch may be a better choice. +When it comes to raw search speed FlexSearch outperforms every single searching library out there and also provides flexible search capabilities like multi-field search, phonetic transformations or partial matching. +Depending on the used options it also provides the most memory-efficient index. Keep in mind that updating __existing__ items or removing items from the index has a significant cost. When existing items of your index needs to be updated/removed continuously then BulkSearch may be a better choice. FlexSearch also provides you a non-blocking asynchronous processing model as well as web workers to perform any updates or queries on the index in parallel through dedicated balanced threads. -FlexSearch Server is also available here: https://github.com/nextapps-de/flexsearch-server. +FlexSearch Server is available here: https://github.com/nextapps-de/flexsearch-server. Installation Guide  •  API Reference  •  Custom Builds  •  Flexsearch Server  •  Changelog @@ -390,7 +388,7 @@ Comparison: _* Index Size: The size of memory the index requires_
_** Memory Allocation: The amount of memory which was additionally allocated during a row of 10 queries_
-_*** The preset "fastest" was used for this test_
+_*** The preset "fast" was used for this test_
_**** The preset "memory" was used for this test_ Library Comparison: Benchmark "Gulliver's Travels" @@ -398,27 +396,56 @@ Library Comparison: ## Contextual Search +> __Note:__ This feature is actually not enabled by default. Read here how to enable. + +FlexSearch introduce a new scoring mechanism called __Contextual Search__ which was invented by Thomas Wilkerling, the author of this library. A Contextual Search incredibly boost up queries to a complete new level but also requires some additionally memory (depending on ___depth___). +The basic idea of this concept is to limit relevance by its context instead of calculating relevance through the whole (unlimited) distance. +In this way contextual search also improves the results of relevance-based queries on a large amount of text data. + > "TF-IDF and all kinds of variations (like BM25) is a big mistake in searching algorithms today. They don't provide neither: a meaningful relevance of a term nor the importance of it! Like many pseudo-intelligent algorithms this is also just an example of mathematical stupidity." — Thomas Wilkerling, _Contextual-based Scoring_, 2018 -FlexSearch introduce a new scoring mechanism called __Contextual Search__ which was invented by Thomas Wilkerling, the author of this library. A Contextual Search incredibly boost up queries to a complete new level but also requires some additionally memory. -The basic idea of this concept is to limit relevance by its context instead of calculating relevance through the whole (unlimited) distance. -Imagine you add a text block of some sentences to an index ID. Assuming the query includes a combination of first and last word from this text block, are they really relevant to each other? -In this way contextual search also improves the results of relevance-based queries on large amount of text data. +#### Model of context-based scoring

-__Note:__ This feature is actually not enabled by default. Read here how to enable. + +#### Lexical Pre-Scored Dictionary / Context-based Map +The index consists of an in-memory pre-scored dictionary as its base. The biggest complexity of these algorithm occurs during the calculation of intersections. As a consequence each additional term in the query has a significant potential to increase complexity. A contextual map comes into play __when the query contains more than 1 term__ and increase effect for each additional term by cutting down the complexity for the intersection calculations. Instead of an increase, the complexity is lowered for each additional term. The contextual index itself is also based on a pre-scored dictionary and follows a memory-friendly strategy. -#### Compare BulkSearch vs. FlexSearch + + + + + + + + + + + + + + + + + + + + +
TypeComplexity
Each single term query:1
Lexical Pre-Scored Dictionary (Solo):TERM_COUNT * TERM_MATCHES
Lexical Pre-Scored Dictionary + Context-based Map:TERM_MATCHES / TERM_COUNT
+ +The complexity for one single term is always 1. + +## Compare BulkSearch vs. FlexSearch - + + @@ -435,7 +462,7 @@ __Note:__ This feature is actually not enabled by default. Read - + @@ -455,14 +482,14 @@ __Note:__ This feature is actually not enabled by default. Read Index.__destroy__() - Index.__init__(\) - Index.__info__() +- Index.__find__() +- Index.__where__() - Index.__addMatcher__({_KEY: VALUE_}) - Index.__encode__(string) - Index.__export__() @@ -1297,7 +1326,7 @@ index.where(function(item){ }); ``` -#### Combine fuzzy search with a where-clause: +#### Combine fuzzy search with a where-clause Add some content, e.g.: ```js @@ -1331,7 +1360,9 @@ index.search("foo", { ## Tags -Tagging is pretty much the same like adding an index to a database column. Whenever you use ___where___ on an indexed/tagged attribute will improve performance drastically but also at a cost of additional memory. +> __IMPORTANT NOTICE:__ This feature will be removed due to the lack of scaling and redundancy. + +Tagging is pretty much the same like adding an additional index to a database column. Whenever you use ___where___ on an indexed/tagged attribute will improve performance drastically but also at a cost of additional memory. > The colon notation also has to be applied for tags respectively. @@ -1541,7 +1572,7 @@ FlexSearch ist highly customizable. Make use of the the right "match"
"score"
"balance"
- "fastest" + "fast" @@ -1638,11 +1669,19 @@ FlexSearch ist highly customizable. Make use of the the right + + + + + + - +
- BulkSearch - FlexSearch + BulkSearchFlexSearch
AccessWeaks
  • less powerful contextual search
  • less memory efficient (has to be defragmented from time to time)
  • updating / deleting existing items from index is slow
  • adding items to the index optimized for super partial matching (tokenize: "full") is slow
  • updating existing / deleting items from index is slow
  • adding items to the index optimized for partial matching (tokenize: "forward" / "reverse" / "full") is slow
The configuration profile. Choose your preferation.
@@ -1629,7 +1660,7 @@ FlexSearch ist highly customizable. Make use of the the right 		depth

 false
- {number:0-9} + {number} 		Enable/Disable contextual indexing and also sets contextual distance of relevance.
threshold

 false
- {number:0-9} + {number} 		Enable/Disable the threshold of minimum relevance all results should have.

Note: It is also possible to set a lower threshold for indexing and pass a higher value when calling index.search(options).
resolution + {number} + Sets the scoring resolution (default: 9).
stemmer


 @@ -1664,7 +1703,7 @@ FlexSearch ist highly customizable. Make use of the the right
rtl


rtl

 true
false @@ -1966,6 +2005,8 @@ The required memory for the index depends on several options:
+Adding, removing or updating existing items has a similar complexity. + #### Compare Memory Consumption @@ -1983,7 +2024,8 @@ __"default"__: Standard profile ```js { encode: "icase", - tokenize: "forward" + tokenize: "forward", + resolution: 9 } ``` @@ -1992,7 +2034,8 @@ __"memory"__: Memory-optimized profile { encode: "extra", tokenize: "strict", - threshold: 7 + threshold: 0, + resolution: 1 } ``` @@ -2002,7 +2045,8 @@ __"speed"__: Speed-optimized profile { encode: "icase", tokenize: "strict", - threshold: 7, + threshold: 1, + resolution: 3, depth: 2 } ``` @@ -2012,7 +2056,9 @@ __"match"__: Matching-tolerant profile ```js { encode: "extra", - tokenize: "full" + tokenize: "full", + threshold: 1, + resolution: 3 } ``` @@ -2022,8 +2068,9 @@ __"score"__: Relevance-optimized profile { encode: "extra", tokenize: "strict", - threshold: 5, - depth: 5 + threshold: 1, + resolution: 9, + depth: 4 } ``` @@ -2033,17 +2080,19 @@ __"balance"__: Most-balanced profile { encode: "balance", tokenize: "strict", - threshold: 6, + threshold: 0, + resolution: 3, depth: 3 } ``` -__"fastest"__: Absolute fastest profile +__"fast"__: Absolute fastest profile ```js { encode: "icase", - threshold: 9, + threshold: 8, + resolution: 9, depth: 1 } ``` @@ -2052,6 +2101,41 @@ Compare these presets: - Relevance Scoring
- Benchmarks +## Performance Guide + +Methods to retrieve results sorted from fastest to slowest: + +1. `index.find(id) -> doc` +2. `index.where({field: string}) -> Arrary` with a tag on the same field +3. `index.search(query) -> Arrary` when just adding _id_ and _content_ to the index (no documents) +4. `index.search(query) -> Arrary` when using documents +5. `index.search(query, { where }) -> Arrary` when using documents and a where clause +6. `index.where({field: [string, string]}) -> Arrary` when a tag was set to one of two fields +7. `index.where({field: string}) -> Arrary` when no tag was set to this field + +Methods to change index from fastest to slowest: + +1. `index.add(id, string)` +2. `index.add(docs)` +3. `index.delete(id, string)` +4. `index.delete(docs)` +5. `index.update(id, string)` +6. `index.update(docs)` + +Performance Checklist: + +- Using just id-content-pairs for the index performs almost faster than using docs +- An additional where-clause in `index.search()` has a significant cost +- When adding multiple fields of documents to the index try to set the lowest possible preset for each field +- Make sure the auto-balanced ___cache___ is enabled and has a meaningful value +- Using `index.where()` to find documents is very slow when using a tagged field +- Getting a document by ID via `index.find(id)` is extremely fast +- Do not enable ___async___ as well as ___worker___ when the index does not claim it +- Use numeric IDs (the datatype length of IDs influences the memory consumption significantly) +- Verify if you can activate _contextual index_ by setting the ___depth___ to a minimum meaningful value +- Pass a ___limit___ when searching (lower is better) +- Pass a minimum ___threshold___ when searching (higher is better) + ## Best Practices ##### Split Complexity @@ -2064,7 +2148,39 @@ var adventure = new FlexSearch(); var comedy = new FlexSearch(); ``` -This way you can also provide different settings for each category. +This way you can also provide different settings for each category. This is actually the fastest way to perform a fuzzy search. + +To make this workaround more extendable you can use a short helper: + +```js +var index = {}; + +function add(id, cat, content){ + (index[cat] || ( + index[cat] = new FlexSearch + )).add(id, content); +} + +function search(cat, query){ + return index[cat] ? + index[cat].search(query) : []; +} +``` + +Add content to the index: +```js +add(1, "action", "Movie Title"); +add(2, "adventure", "Movie Title"); +add(3, "comedy", "Movie Title"); +``` + +Perform queries: +```js +var results = search("action", "movie title"); // --> [1] +``` + + + +Split indexes by categories improves performance significantly. -Filter queries by categories will hugely improve performance. ##### Use numeric IDs It is recommended to use numeric id values as reference when adding content to the index. The byte length of passed ids influences the memory consumption significantly. If this is not possible you should consider to use a index table and map the ids with indexes, this becomes important especially when using contextual indexes on a large amount of content. + ## Export/Import Index @@ -2286,6 +2406,11 @@ node compile SUPPORT_WORKER=true true, false + + SUPPORT_WHERE + true, false + + SUPPORT_WORKER true, false diff --git a/dist/flexsearch.compact.js b/dist/flexsearch.compact.js index ed74873..e490606 100644 --- a/dist/flexsearch.compact.js +++ b/dist/flexsearch.compact.js @@ -1,27 +1,28 @@ /* - FlexSearch v0.5.0 + FlexSearch v0.5.1 Copyright 2019 Nextapps GmbH Author: Thomas Wilkerling Released under the Apache 2.0 Licence https://github.com/nextapps-de/flexsearch */ -'use strict';(function(g,t,A){let d;(d=A.define)&&d.amd?d([],function(){return t}):(d=A.modules)?d[g.toLowerCase()]=t:"object"===typeof exports?module.exports=t:A[g]=t})("FlexSearch",function(){function g(a,b){const c=b?b.id:a&&a.id;this.id=c||0===c?c:P++;this.init(a,b);A(this,"index",function(){return this.c});A(this,"length",function(){return Object.keys(this.c).length})}function t(a,b){a=a.concat.apply([],a);b&&(D(b)||(w=b.split(":"),1=f&&(a=a[9-(h+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]=e);return h}function H(a,b){if(a){const c=Object.keys(a);for(let e=0,h=c.length;ea?1:a?-1:0}function T(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function R(a,b){a=a[w];b=b[w];return ab?1:0}function Q(a,b){const c=w.length;for(let e= -0;eb?1:0}function U(a,b,c,e){let h=[],k;const f=a.length;if(1b&&(h=h.slice(0,b));return h}function F(a){return"string"===typeof a}function D(a){return"function"===typeof a}function z(a){return"object"===typeof a}function G(a){return"undefined"===typeof a}function K(a){const b=Array(a);for(let c=0;cg;c--)l=f.substring(g, -c),E(u,k,l,a,b,m,r)}break;default:if(n=E(u,k,f,a,1,m,r),q&&1=r)for(n=k._ctx[f]||(k._ctx[f]=v()),f=this.g[f]||(this.g[f]=K(10-(r||0))),m=b-q,l=b+q+1,0>m&&(m=0),l>B&&(l=B);mm&&(b=l));for(m=u?1:0;m=e&&(a=a[k-(g+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]=d);return g}function I(a,b){if(a){const c=Object.keys(a);for(let d=0,g=c.length;da?1:a?-1:0}function U(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function S(a,b){a=a[x];b=b[x];return ab?1:0}function R(a,b){const c=x.length;for(let d= +0;db?1:0}function V(a,b,c,d,g){c=[];let h;const e=a.length;if(1b&&(c=c.slice(0,b));return c}function H(a){return"string"===typeof a}function F(a){return"function"===typeof a}function y(a){return"object"===typeof a}function D(a){return"undefined"===typeof a}function L(a){const b=Array(a);for(let c=0;cl;c--)m=e.substring(l,c),G(t,h,m,a,b,q,p,r-1)}break;default:if(k=G(t,h,e,a,1,q,p,r-1),n&&1=p)for(k=h._ctx[e]||(h._ctx[e]=v()),e=this.h[e]||(this.h[e]=L(r-(p||0))),q=b-n,m=b+n+1,0>q&&(q=0),m>C&&(m=C);q=m&&(this.B=this.h),this.G&&this.B===this.h&&(this.cache&& -this.w.set(b,this.u),this.G(this.u),this.u=[]));return this}function l(a,b){a=a.concat.apply([],a);b&&(K(b)||(E=b.split(":"),1=e&&(a=a[9-(c+.5>>0)],a=a[d]||(a[d]=[]),a[a.length]=m);return c}function G(a, -b){if(a)for(var d=Object.keys(a),c=0,e=d.length;c -a?1:a?-1:0}function aa(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function ba(a,b){a=a[E];b=b[E];return ab?1:0}function ca(a,b){for(var d=E.length,c=0;cb?1:0}function P(a,b,d,c){var e=[],h=a.length;if(1b&&(e=e.slice(0,b));return e}function M(a){return"string"===typeof a}function K(a){return"function"===typeof a}function J(a){return"object"===typeof a}function H(a){return"undefined"=== -typeof a}function S(a){for(var b=Array(a),d=0;d=this.a.length&&(this.v=0),this.a[this.v].postMessage({add:!0,id:a,content:b}),this.c[h]=""+this.v,d&&d(),this;if(!e){if(this.async&&"function"!==typeof importScripts){var m=this;h=new Promise(function(d){setTimeout(function(){m.add(a, -b,null,c,!0);m=null;d()})});if(d)h.then(d);else return h;return this}if(d)return this.add(a,b,null,c,!0),d(),this}b=this.encode(b);if(!b.length)return this;d=this.g;e=K(d)?d(b):b.split(U);var q=C();q._ctx=C();for(var f=this.threshold,w=this.depth,n=this.m,g=e.length,u=this.F,t=0;tx;z--)y=l.substring(x,z),A(n,q,y,a,r,p,f);break;default:if(v=A(n,q,l,a,1,p,f),w&&1=f)for(v=q._ctx[l]||(q._ctx[l]=C()),l=this.s[l]||(this.s[l]=S(10-(f||0))),p=t-w,y=t+w+1,0>p&&(p=0),y>g&&(y=g);pm&&(y=n));for(m=u?1:0;mf;c--)l=g[c-1],g[c]=l,e[l]=c;g[f]=a;e[a]=f}}}return b};return a}();return e}(function(){var f={},k="undefined"!==typeof Blob&&"undefined"!==typeof URL&&URL.createObjectURL;return function(e,g,l,I,c){l=k?URL.createObjectURL(new Blob(["("+l.toString()+")()"],{type:"text/javascript"})):e+".es5.js";e+="-"+g;f[e]|| -(f[e]=[]);f[e][c]=new Worker(l);f[e][c].onmessage=I;console.log("Register Worker: "+e+"@"+c);return f[e][c]}}()),this); +'use strict';function S(f){var k=0;return function(){return k=g&&(this.D=this.i),this.I&&this.D===this.i&&(this.cache&& +this.A.set(b,this.v),this.I(this.v),this.v=[]));return this}function l(a,b){a=a.concat.apply([],a);b&&(L(b)||(F=b.split(":"),1=e&&(a=a[h-(m+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]=g);return m} +function H(a,b){if(a)for(var c=Object.keys(a),g=0,m=c.length;ga?1:a?-1:0}function ca(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function da(a,b){a=a[F];b=b[F];return ab?1:0}function ea(a,b){for(var c=F.length,g=0;gb?1:0}function Q(a,b,c,g,d){c=[];var e=a.length;if(1b&&(c=c.slice(0,b));return c}function M(a){return"string"===typeof a}function I(a){return a.constructor===Array}function L(a){return"function"===typeof a}function J(a){return"object"=== +typeof a}function E(a){return"undefined"===typeof a}function U(a){for(var b=Array(a),c=0;c=this.a.length&&(this.w=0),this.a[this.w].postMessage({add:!0,id:a,content:b}),this.c[g]=""+this.w,c&&c(),this;if(!e){if(this.async&&"function"!==typeof importScripts){var m=this;g=new Promise(function(c){setTimeout(function(){m.add(a,b,null,d,!0);m=null;c()})});if(c)g.then(c);else return g;return this}if(c)return this.add(a,b,null,d,!0),c(),this}b=this.encode(b);if(!b.length)return this;c=this.h; +e=L(c)?c(b):b.split(W);var h=C();h._ctx=C();for(var f=this.threshold,p=this.depth,q=this.f,z=this.o,r=e.length,n=this.H,l=0;lx;y--)A=u.substring(x,y),B(z,h,A,a,t,w,f,q-1);break;default:if(v=B(z,h,u,a,1,w,f,q- +1),p&&1=f)for(v=h._ctx[u]||(h._ctx[u]=C()),u=this.u[u]||(this.u[u]=U(q-(f||0))),w=l-p,A=l+p+1,0>w&&(w=0),A>r&&(A=r);wf;d--)h=l[d-1],l[d]=h,e[h]=d;l[f]=a;e[a]=f}}}return b};return a}();return e}(function(){var f={},k="undefined"!==typeof Blob&&"undefined"!==typeof URL&&URL.createObjectURL;return function(e,n,l,K,d){l=k?URL.createObjectURL(new Blob(["("+l.toString()+")()"],{type:"text/javascript"})): +e+".es5.js";e+="-"+n;f[e]||(f[e]=[]);f[e][d]=new Worker(l);f[e][d].onmessage=K;console.log("Register Worker: "+e+"@"+d);return f[e][d]}}()),this); diff --git a/dist/flexsearch.light.js b/dist/flexsearch.light.js index 0f9ce9f..c2ff7af 100644 --- a/dist/flexsearch.light.js +++ b/dist/flexsearch.light.js @@ -1,18 +1,18 @@ /* - FlexSearch v0.5.0 + FlexSearch v0.5.1 Copyright 2019 Nextapps GmbH Author: Thomas Wilkerling Released under the Apache 2.0 Licence https://github.com/nextapps-de/flexsearch */ -'use strict';(function(e,l,u){let n;(n=u.define)&&n.amd?n([],function(){return l}):(n=u.modules)?n[e.toLowerCase()]=l:"object"===typeof exports?module.exports=l:u[e]=l})("FlexSearch",function(){function e(a,c){const b=c?c.id:a&&a.id;this.id=b||0===b?b:G++;this.init(a,c);l(this,"index",function(){return this.a});l(this,"length",function(){return Object.keys(this.a).length})}function l(a,c,b){Object.defineProperty(a,c,{get:b})}function u(a,c){for(let b=0;b=g&&(a=a[9-(d+.5>>0)],a=a[b]||(a[b]=[]),a[a.length]=f);return d}function y(a,c){if(a){const b=Object.keys(a);for(let f=0,d=b.length;fa?1:a?-1:0}function I(a,c){a=a.length-c.length;return 0>a?-1:a?1:0}function x(a){return"function"=== -typeof a}function z(a){return"undefined"===typeof a}function C(a){const c=Array(a);for(let b=0;bh;b--)m=p.substring(h,b),n(r,f,m,a,c,k,d)}break;default:if(g=n(r,f,p,a,1,k,d),B&&1=d)for(g=f._ctx[p]||(f._ctx[p]=t()),p=this.c[p]||(this.c[p]= -C(10-(d||0))),k=e-B,m=e+B+1,0>k&&(k=0),m>q&&(m=q);kc&&(g=g.slice(0,c)));e=g}return e};e.prototype.clear=function(){return this.destroy().init()}; -e.prototype.destroy=function(){this.g=this.c=this.a=null;return this};const w={icase:function(a){return a.toLowerCase()}};return e}(!1),this); +'use strict';(function(d,l,v){let t;(t=v.define)&&t.amd?t([],function(){return l}):(t=v.modules)?t[d.toLowerCase()]=l:"object"===typeof exports?module.exports=l:v[d]=l})("FlexSearch",function(){function d(a,c){const b=c?c.id:a&&a.id;this.id=b||0===b?b:G++;this.init(a,c);l(this,"index",function(){return this.b});l(this,"length",function(){return Object.keys(this.b).length})}function l(a,c,b){Object.defineProperty(a,c,{get:b})}function v(a,c){for(let b=0;b=g&&(a=a[h-(e+.5>>0)],a=a[b]||(a[b]=[]),a[a.length]=f);return e}function A(a,c){if(a){const b=Object.keys(a);for(let f=0,e=b.length;fa?1:a?-1:0}function I(a,c){a=a.length-c.length;return 0>a?-1:a?1:0}function z(a){return"function"=== +typeof a}function y(a){return"undefined"===typeof a}function C(a){const c=Array(a);for(let b=0;b +k;c--)m=p.substring(k,c),t(q,f,m,a,b,h,e,r-1)}break;default:if(g=t(q,f,p,a,1,h,e,r-1),n&&1=e)for(g=f._ctx[p]||(f._ctx[p]=u()),p=this.f[p]||(this.f[p]=C(r-(e||0))),h=d-n,m=d+n+1,0>h&&(h=0),m>x&&(m=x);hc&&(g=g.slice(0,c)));d=g}return d};d.prototype.clear=function(){return this.destroy().init()};d.prototype.destroy=function(){this.h=this.f=this.b=null;return this};const w={icase:function(a){return a.toLowerCase()}};return d}(!1),this); diff --git a/dist/flexsearch.min.js b/dist/flexsearch.min.js index 5facd5b..df8a329 100644 --- a/dist/flexsearch.min.js +++ b/dist/flexsearch.min.js @@ -1,37 +1,38 @@ /* - FlexSearch v0.5.0 + FlexSearch v0.5.1 Copyright 2019 Nextapps GmbH Author: Thomas Wilkerling Released under the Apache 2.0 Licence https://github.com/nextapps-de/flexsearch */ -'use strict';(function(x,F,f){let y;(y=f.define)&&y.amd?y([],function(){return F}):(y=f.modules)?y[x.toLowerCase()]=F:"object"===typeof exports?module.exports=F:f[x]=F})("FlexSearch",function U(x){function f(a,b){const c=b?b.id:a&&a.id;this.id=c||0===c?c:V++;this.init(a,b);K(this,"index",function(){return this.b});K(this,"length",function(){return Object.keys(this.b).length})}function y(a,b,c,d){this.s!==this.f&&(this.j=this.j.concat(c),this.s++,d&&this.j.length>=d&&(this.s=this.f),this.F&&this.s=== -this.f&&(this.cache&&this.m.set(b,this.j),this.F(this.j),this.j=[]));return this}function A(a,b){a=a.concat.apply([],a);b&&(G(b)||(z=b.split(":"),1=k&&(a=a[9-(e+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]=d); -return e}function N(a,b){if(a){const c=Object.keys(a);for(let d=0,e=c.length;da?1:a?-1:0}function Z(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function X(a,b){a=a[z];b=b[z];return ab?1:0}function W(a,b){const c=z.length;for(let d=0;db?1:0}function aa(a,b,c,d){let e=[],g;const k=a.length;if(1b&&(e=e.slice(0,b));return e}function J(a){return"string"===typeof a}function G(a){return"function"===typeof a}function D(a){return"object"=== -typeof a}function B(a){return"undefined"===typeof a}function P(a){const b=Array(a);for(let c=0;c=this.l.length&&(this.C=0),this.l[this.C].postMessage({add:!0,id:a,content:b}),this.b[g]=""+this.C,c&&c(),this;if(!e){if(this.async&&"function"!==typeof importScripts){let e=this;g=new Promise(function(c){setTimeout(function(){e.add(a, -b,null,d,!0);e=null;c()})});if(c)g.then(c);else return g;return this}if(c)return this.add(a,b,null,d,!0),c(),this}b=this.encode(b);if(!b.length)return this;c=this.c;e=G(c)?c(b):b.split(R);const m=q();m._ctx=q();const n=this.threshold,r=this.depth,t=this.h,v=e.length,L=this.D;for(let b=0;bf;c--)l=k.substring(f,c),I(t,m,l,a,b,h,n)}break;default:if(p=I(t,m,k,a,1,h,n),r&&1=n)for(p=m._ctx[k]||(m._ctx[k]=q()),k=this.i[k]||(this.i[k]=P(10-(n||0))),h=b-r,l=b+r+1,0>h&&(h=0),l>v&&(l=v);hr&&(a=m));for(r=n?1:0;rc;d--)e=f[d-1],f[d]=e,b[e]=d;f[c]=a;b[a]=c}}}return b};return a}();return f}(function(){const x={},F="undefined"!==typeof Blob&&"undefined"!==typeof URL&&URL.createObjectURL; -return function(f,y,A,K,h){A=F?URL.createObjectURL(new Blob(["("+A.toString()+")()"],{type:"text/javascript"})):f+".min.js";f+="-"+y;x[f]||(x[f]=[]);x[f][h]=new Worker(A);x[f][h].onmessage=K;return x[f][h]}}()),this); +'use strict';(function(w,F,g){let y;(y=g.define)&&y.amd?y([],function(){return F}):(y=g.modules)?y[w.toLowerCase()]=F:"object"===typeof exports?module.exports=F:g[w]=F})("FlexSearch",function U(w){function g(a,b){const c=b?b.id:a&&a.id;this.id=c||0===c?c:V++;this.init(a,b);K(this,"index",function(){return this.b});K(this,"length",function(){return Object.keys(this.b).length})}function y(a,b,c,d){this.u!==this.g&&(this.l=this.l.concat(c),this.u++,d&&this.l.length>=d&&(this.u=this.g),this.G&&this.u=== +this.g&&(this.cache&&this.o.set(b,this.l),this.G(this.l),this.l=[]));return this}function B(a,b){a=a.concat.apply([],a);b&&(G(b)||(z=b.split(":"),1=k&&(a=a[h-(e+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]= +d);return e}function N(a,b){if(a){const c=Object.keys(a);for(let d=0,e=c.length;da?1:a?-1:0}function Z(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function X(a,b){a=a[z];b=b[z];return ab?1:0}function W(a,b){const c=z.length;for(let d=0;db?1:0}function aa(a,b,c,d,e){c=[];let l;const k=a.length;if(1b&&(c=c.slice(0,b));return c}function H(a){return"string"===typeof a}function x(a){return a.constructor=== +Array}function G(a){return"function"===typeof a}function A(a){return"object"===typeof a}function v(a){return"undefined"===typeof a}function P(a){const b=Array(a);for(let c=0;c=this.m.length&&(this.D=0),this.m[this.D].postMessage({add:!0,id:a,content:b}),this.b[l]=""+this.D,c&&c(),this;if(!e){if(this.async&&"function"!==typeof importScripts){let e=this;l=new Promise(function(c){setTimeout(function(){e.add(a,b,null,d,!0);e=null;c()})});if(c)l.then(c);else return l;return this}if(c)return this.add(a,b,null,d,!0),c(),this}b=this.encode(b);if(!b.length)return this;c=this.f; +e=G(c)?c(b):b.split(R);const m=r();m._ctx=r();const p=this.threshold,C=this.depth,t=this.c,q=this.i,D=e.length,L=this.F;for(let b=0;bg;c--)n=k.substring(g,c),J(q,m,n,a,b,f,p,t-1)}break;default:if(h=J(q, +m,k,a,1,f,p,t-1),C&&1=p)for(h=m._ctx[k]||(m._ctx[k]=r()),k=this.j[k]||(this.j[k]=P(t-(p||0))),f=b-C,n=b+C+1,0>f&&(f=0),n>D&&(n=D);fc;d--)e=f[d-1],f[d]=e,b[e]=d;f[c]=a;b[a]=c}}}return b};return a}();return g}(function(){const w= +{},F="undefined"!==typeof Blob&&"undefined"!==typeof URL&&URL.createObjectURL;return function(g,y,B,K,f){B=F?URL.createObjectURL(new Blob(["("+B.toString()+")()"],{type:"text/javascript"})):g+".min.js";g+="-"+y;w[g]||(w[g]=[]);w[g][f]=new Worker(B);w[g][f].onmessage=K;return w[g][f]}}()),this); diff --git a/dist/flexsearch.node.js b/dist/flexsearch.node.js index 29ef3cd..9ea08db 100644 --- a/dist/flexsearch.node.js +++ b/dist/flexsearch.node.js @@ -1,33 +1,34 @@ /* - FlexSearch v0.5.0 + FlexSearch v0.5.1 Copyright 2019 Nextapps GmbH Author: Thomas Wilkerling Released under the Apache 2.0 Licence https://github.com/nextapps-de/flexsearch */ -'use strict';(function(h,w,C){let f;(f=C.define)&&f.amd?f([],function(){return w}):(f=C.modules)?f[h.toLowerCase()]=w:"object"===typeof exports?module.exports=w:C[h]=w})("FlexSearch",function(){function h(a,b){const c=b?b.id:a&&a.id;this.id=c||0===c?c:R++;this.init(a,b);C(this,"index",function(){return this.b});C(this,"length",function(){return Object.keys(this.b).length})}function w(a,b){a=a.concat.apply([],a);b&&(D(b)||(x=b.split(":"),1=l&&(a=a[9-(e+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]=d);return e}function K(a,b){if(a){const c=Object.keys(a);for(let d=0,e=c.length;da?1:a?-1:0}function V(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function T(a,b){a=a[x];b=b[x];return ab?1:0}function S(a,b){const c=x.length;for(let d= -0;db?1:0}function W(a,b,c,d){let e=[],g;const l=a.length;if(1b&&(e=e.slice(0,b));return e}function I(a){return"string"===typeof a}function D(a){return"function"===typeof a}function z(a){return"object"===typeof a}function B(a){return"undefined"===typeof a}function M(a){const b=Array(a);for(let c=0;cm;c--)n=l.substring(m,c),H(r,h,n,a,b,f,p)}break;default:if(k=H(r,h,l,a,1,f,p),E&&1=p)for(k=h._ctx[l]||(h._ctx[l]=u()),l=this.h[l]||(this.h[l]=M(10-(p||0))),f=b-E,n=b+E+1,0>f&&(f=0),n>t&&(n=t);fh&&(a=m));for(h=p?1:0;hc;d--)e=f[d-1],f[d]=e,b[e]=d;f[c]=a;b[a]=c}}}return b};return a}();return h}(!1),this); +'use strict';(function(h,y,D){let e;(e=D.define)&&e.amd?e([],function(){return y}):(e=D.modules)?e[h.toLowerCase()]=y:"object"===typeof exports?module.exports=y:D[h]=y})("FlexSearch",function(){function h(a,b){const c=b?b.id:a&&a.id;this.id=c||0===c?c:S++;this.init(a,b);D(this,"index",function(){return this.c});D(this,"length",function(){return Object.keys(this.c).length})}function y(a,b){a=a.concat.apply([],a);b&&(E(b)||(z=b.split(":"),1=g&&(a=a[k-(f+.5>>0)],a=a[c]||(a[c]=[]),a[a.length]=d);return f}function L(a,b){if(a){const c=Object.keys(a);for(let d=0,f=c.length;da?1:a?-1:0}function W(a,b){a=a.length-b.length;return 0>a?-1:a?1:0}function U(a,b){a=a[z];b=b[z];return ab?1:0}function T(a,b){const c=z.length;for(let d= +0;db?1:0}function X(a,b,c,d,f){c=[];let l;const g=a.length;if(1b&&(c=c.slice(0,b));return c}function F(a){return"string"===typeof a}function x(a){return a.constructor===Array}function E(a){return"function"===typeof a}function A(a){return"object"===typeof a}function v(a){return"undefined"===typeof a}function N(a){const b=Array(a);for(let c=0;cm;c--)n=g.substring(m,c),H(I,p,n,a,b,e,q,u-1)}break;default:if(k=H(I,p,g,a,1,e,q,u-1),h&&1=q)for(k=p._ctx[g]||(p._ctx[g]=t()),g=this.i[g]||(this.i[g]=N(u-(q||0))),e=b-h,n=b+h+1,0>e&&(e=0),n>B&&(n=B);ec;d--)f=e[d-1],e[d]=f,b[f]=d;e[c]=a;b[a]=c}}}return b};return a}();return h}(!1),this); diff --git a/flexsearch.js b/flexsearch.js index 12f13e0..b9b9d4a 100644 --- a/flexsearch.js +++ b/flexsearch.js @@ -1,5 +1,5 @@ /**! - * @preserve FlexSearch v0.5.0 + * @preserve FlexSearch v0.5.1 * Copyright 2019 Nextapps GmbH * Author: Thomas Wilkerling * Released under the Apache 2.0 Licence @@ -41,15 +41,17 @@ async: false, worker: false, rtl: false, + doc: false, + paging: false, - // minimum scoring (0 - 9) + // maximum scoring + resolution: 9, + + // minimum scoring threshold: 0, // contextual depth - depth: 0, - - // multi-field documents - doc: false + depth: 0 }; /** @@ -62,39 +64,46 @@ "memory": { encode: SUPPORT_ENCODER ? "extra" : "icase", tokenize: "strict", - threshold: 7 + threshold: 0, + resolution: 1 }, "speed": { encode: "icase", tokenize: "strict", - threshold: 7, + threshold: 1, + resolution: 3, depth: 2 }, "match": { encode: SUPPORT_ENCODER ? "extra" : "icase", - tokenize: "full" + tokenize: "full", + threshold: 1, + resolution: 3 }, "score": { encode: SUPPORT_ENCODER ? "extra" : "icase", tokenize: "strict", - threshold: 5, + threshold: 1, + resolution: 9, depth: 4 }, "balance": { encode: SUPPORT_ENCODER ? "balance" : "icase", tokenize: "strict", - threshold: 6, + threshold: 0, + resolution: 3, depth: 3 }, - "fastest": { + "fast": { encode: "icase", tokenize: "strict", - threshold: 9, + threshold: 8, + resolution: 9, depth: 1 } }; @@ -114,19 +123,6 @@ let id_counter = 0; - /** - * @enum {number} - */ - - /* - const enum_task = { - - add: 0, - update: 1, - remove: 2 - }; - */ - /** * NOTE: This could make issues on some languages (chinese, korean, thai) * TODO: extract all language-specific stuff from the core @@ -295,6 +291,14 @@ return this; } + const tag_options = { + + "encode": false, + "tokenize": function(doc){ + return [doc]; + } + }; + /** * @param {Object|string=} options * @param {Object=} settings @@ -432,6 +436,23 @@ custom ); + /** @private */ + this.resolution = ( + + is_undefined(custom = options["resolution"]) ? + + custom = preset.resolution || + this.resolution || + defaults.resolution + : + custom + ); + + if(custom <= this.threshold){ + + this.resolution = this.threshold + 1; + } + /** @private */ this.depth = ( @@ -444,6 +465,27 @@ custom ); + /** @private */ + this.paging = ( + + is_undefined(custom = options["paging"]) ? + + this.paging || + defaults.paging + : + custom + ); + + // TODO: provide boost + + /** @private */ + /* + this.boost = ( + + (custom = options["boost"]) ? custom : 0 + ); + */ + if(SUPPORT_SUGGESTIONS){ /** @private */ @@ -511,7 +553,7 @@ // initialize primary index /** @private */ - this._map = create_object_array(10 - (this.threshold || 0)); + this._map = create_object_array(this.resolution - (this.threshold || 0)); /** @private */ this._ctx = create_object(); /** @private */ @@ -529,12 +571,36 @@ let field = doc["field"]; let tag = doc["tag"]; - doc["id"] = doc["id"].split(":"); + if(!is_array(doc["id"])){ + + doc["id"] = doc["id"].split(":"); + } if(SUPPORT_WHERE && tag){ this._tag = create_object(); + let field_custom = create_object(); + + if(field){ + + if(is_string(field)){ + + field_custom[field] = options; + } + else if(is_array(field)){ + + for(let i = 0; i < field.length; i++){ + + field_custom[field[i]] = options; + } + } + else if(is_object(field)){ + + field_custom = field; + } + } + if(!is_array(tag)){ doc["tag"] = tag = [tag]; @@ -542,30 +608,57 @@ for(let i = 0; i < tag.length; i++){ - this._tag[tag[i]] = create_object(); + //TODO: delegate tag indexes to intersection + //field_custom[tag[i]] = tag_options; - tag[i] = tag[i].split(":"); + this._tag[tag[i]] = create_object(); } + + this._tags = tag; + + field = field_custom; } if(field){ + let has_custom; + if(!is_array(field)){ - doc["field"] = field = [field]; + if(is_object(field)){ + + has_custom = field; + doc["field"] = field = get_keys(field); + } + else{ + + doc["field"] = field = [field]; + } } for(let i = 0; i < field.length; i++){ - ref[field[i]] = i; - field[i] = field[i].split(":"); - index[i] = new FlexSearch(options); - index[i]._doc = this._doc; + if(!is_array(field[i])){ - if(SUPPORT_WHERE && tag){ + if(has_custom){ - index[i]._tag = this._tag; + options = has_custom[field[i]]; + } + + ref[field[i]] = i; + field[i] = field[i].split(":"); } + + // TODO: move fields to main index to provide pagination + + index[i] = new FlexSearch(options); + index[i]._doc = this._doc; + + // if(SUPPORT_WHERE && tag){ + // + // index[i]._tag = this._tag; + // index[i]._tags = tag; + // } } } } @@ -826,6 +919,7 @@ const threshold = this.threshold; const depth = this.depth; + const resolution = this.resolution; const map = this._map; const word_length = words.length; const rtl = this.rtl; @@ -863,7 +957,8 @@ id, rtl ? 1 : (length - a) / length, context_score, - threshold + threshold, + resolution - 1 ); } @@ -885,7 +980,8 @@ id, rtl ? (a + 1) / length : 1, context_score, - threshold + threshold, + resolution - 1 ); } @@ -909,7 +1005,8 @@ id, partial_score, context_score, - threshold + threshold, + resolution - 1 ); } } @@ -930,13 +1027,14 @@ // TODO compute and pass distance of ngram sequences as the initial score for each word 1, context_score, - threshold + threshold, + resolution - 1 ); if(depth && (word_length > 1) && (score >= threshold)){ const ctxDupes = dupes["_ctx"][value] || (dupes["_ctx"][value] = create_object()); - const ctxTmp = this._ctx[value] || (this._ctx[value] = create_object_array(10 - (threshold || 0))); + const ctxTmp = this._ctx[value] || (this._ctx[value] = create_object_array(resolution - (threshold || 0))); let x = i - depth; let y = i + depth + 1; @@ -953,8 +1051,9 @@ words[x], id, 0, - 10 - (x < i ? i - x : x - i), - threshold + resolution - (x < i ? i - x : x - i), + threshold, + resolution - 1 ); } } @@ -986,24 +1085,24 @@ /** * @param {!string} job - * @param docs + * @param doc * @param {Function=} callback * @returns {*} */ - FlexSearch.prototype.handle_docs = function(job, docs, callback){ + FlexSearch.prototype.handle_docs = function(job, doc, callback){ - if(is_array(docs)){ + if(is_array(doc)){ - for(let i = 0, len = docs.length; i < len; i++){ + for(let i = 0, len = doc.length; i < len; i++){ if(i === len - 1){ - return this.handle_docs(job, docs[i], callback); + return this.handle_docs(job, doc[i], callback); } else{ - this.handle_docs(job, docs[i]); + this.handle_docs(job, doc[i]); } } } @@ -1014,23 +1113,37 @@ let tree = this.doc["id"]; let id; let tag; + let tag_key; + let tag_value; for(let i = 0; i < tree.length; i++){ - id = (id || docs)[tree[i]]; + id = (id || doc)[tree[i]]; } if(tags){ for(let i = 0; i < tags.length; i++){ - tag = (tag || docs)[tags[i]]; + tag_key = tags[i]; + + const tag_split = tag_key.split(":"); + + for(let i = 0; i < tag_split.length; i++){ + + tag_value = (tag_value || doc)[tag_split[i]]; + } + + tag_value = "@" + tag_value; } + + tag = this._tag[tag_key]; + tag = tag[tag_value] || (tag[tag_value] = []); } if(job === "remove"){ - delete this._doc["@" + id]; + delete this._doc[id]; for(let z = 0, length = index.length; z < length; z++){ @@ -1047,6 +1160,11 @@ tree = this.doc["field"]; + if(tag){ + + tag[tag.length] = doc; // tag[tag.length] = id; + } + for(let i = 0, len = tree.length; i < len; i++){ const branch = tree[i]; @@ -1054,20 +1172,23 @@ for(let x = 0; x < branch.length; x++){ - content = (content || docs)[branch[x]]; + content = (content || doc)[branch[x]]; } - this._doc["@" + id] = docs; + this._doc[id] = doc; const self = index[i]; - const fn = ( - job === "add" ? + let fn; - self.add - : - self.update - ); + if(job === "add"){ + + fn = self.add; + } + else{ + + fn = self.update; + } if(i === len - 1){ @@ -1199,7 +1320,7 @@ profile_start("remove"); } - for(let z = 0; z < (10 - (this.threshold || 0)); z++){ + for(let z = 0; z < (this.resolution - (this.threshold || 0)); z++){ remove_index(this._map[z], id); } @@ -1257,6 +1378,8 @@ } /** + * TODO: move fields to main index to provide pagination + * * @param {!string|Object|Array} query * @param {number|Function=} limit * @param {Function=} callback @@ -1287,9 +1410,10 @@ let _query = query; let threshold; - let boost; + //let boost; let where; let sort; + let cursor; let result = []; if(is_object(query) && (!SUPPORT_DOCUMENTS || !is_array(query))){ @@ -1308,11 +1432,12 @@ if(SUPPORT_DOCUMENTS){ - boost = query["boost"]; + //boost = query["boost"]; where = query["where"]; sort = query["sort"]; } + //cursor = this.paging && query["cursor"]; limit = query["limit"]; threshold = query["threshold"]; query = query["query"]; @@ -1347,6 +1472,11 @@ if(is_object(field)){ + if(sort){ + + _query["sort"] = null; + } + if(!is_array(field)){ field = [field]; @@ -1364,7 +1494,6 @@ const ref = doc_ref[field[i]]; // TODO: Support Field-Merging (return array before intersection and apply here) - result[i] = doc_idx[ref].search(_query); } @@ -1419,6 +1548,7 @@ "search": true, "limit": limit, + //"cursor": cursor, "threshold": threshold, "content": query }); @@ -1551,21 +1681,45 @@ } } - let ctx_map; + /* + if(SUPPORT_WHERE && where){ - if(!use_contextual || (ctx_map = this._ctx)[ctx_root]){ + const tags = this._tags; - let initial_z = 0; + if(tags){ - if(SUPPORT_DOCUMENTS && boost){ + for(let i = 0; i < tags.length; i++){ - threshold = (threshold || 1) / boost; + const current_tag = tags[i]; + const current_where = where[current_tag]; - if(boost < 0){ + if(!is_undefined(current_where)){ - initial_z = threshold; + check[check.length] = this._tag[current_tag]["@" + current_where]; + + delete where[current_tag]; + } + } + + if(get_keys(where).length === 0){ + + where = false; } } + } + */ + + let ctx_map; + + if(!use_contextual || (ctx_map = this._ctx)[ctx_root]){ + + let resolution = this.resolution; + + // TODO: boost on custom search is actually not possible, move to adding index instead + // if(SUPPORT_DOCUMENTS && boost){ + // + // threshold = (threshold || 1) / boost; + // } for(let a = (use_contextual ? 1 : 0); a < length; a++){ @@ -1589,7 +1743,7 @@ let map_value; - for(let z = initial_z; z < (10 - threshold); z++){ + for(let z = 0; z < (resolution - threshold); z++){ if((map_value = map[z][value])){ @@ -1639,7 +1793,7 @@ // Not handled by intersection: - result = intersect(check, limit, SUPPORT_DOCUMENTS && this._doc, SUPPORT_SUGGESTIONS && this.suggest); + result = intersect(check, limit, cursor, SUPPORT_DOCUMENTS && this._doc, SUPPORT_SUGGESTIONS && this.suggest); // Handled by intersection: @@ -1651,7 +1805,7 @@ result = this.where(where, null, limit, result); } - if(sort){ + if(SUPPORT_DOCUMENTS && sort){ result = merge_and_sort([result], sort); } @@ -1702,11 +1856,20 @@ let has_value; let tree; - if(is_string(key)){ + if(typeof key === "number"){ + + return [doc[key]]; + } + else if(is_string(key)){ + + if(is_undefined(value)){ + + return [doc[key]]; + } if(key === "id"){ - return [doc["@" + value]]; + return [doc[value]]; } //keys = [key]; @@ -1740,7 +1903,40 @@ if((keys_len === 1) && (keys[0] === "id")){ - return [doc["@" + key["id"]]]; + return [doc[key["id"]]]; + } + + const tags = this._tags; + + if(tags && !result){ + + for(let i = 0; i < tags.length; i++){ + + const current_tag = tags[i]; + const current_where = key[current_tag]; + + if(!is_undefined(current_where)){ + + result = this._tag[current_tag]["@" + current_where]; + //result = result.slice(0, limit && (limit < result.length) ? limit : result.length); + + if(--keys_len === 0){ + + return result; + } + + keys.splice(keys.indexOf(current_tag), 1); + + // TODO: delete from original reference? + delete key[current_tag]; + break; + } + } + + // for(let i = 0; i < result.length; i++){ + // + // result[i] = this._doc[result[i]]; + // } } tree = new Array(keys_len); @@ -1799,7 +1995,7 @@ } return results; - } + }; } if(SUPPORT_INFO){ @@ -1828,7 +2024,7 @@ words = 0, chars = 0; - for(let z = 0; z < (10 - (this.threshold || 0)); z++){ + for(let z = 0; z < (this.resolution - (this.threshold || 0)); z++){ keys = get_keys(this._map[z]); @@ -2225,36 +2421,8 @@ } : { "icase": global_encoder_icase - //"balance": global_encoder_balance }; - // Async Handler - - /* - const queue = SUPPORT_ASYNC ? (function(){ - - const stack = create_object(); - - return function(fn, delay, key){ - - const timer = stack[key]; - - if(timer){ - - clearTimeout(timer); - } - - return ( - - stack[key] = setTimeout(fn, delay) - ); - }; - - }()) : null; - */ - - // Flexi-Cache - const Cache = SUPPORT_CACHE ? (function(){ function CacheClass(limit){ @@ -2452,9 +2620,10 @@ * @param {number} partial_score * @param {number} context_score * @param {number} threshold + * @param {number} resolution */ - function add_index(map, dupes, value, id, partial_score, context_score, threshold){ + function add_index(map, dupes, value, id, partial_score, context_score, threshold, resolution){ /* if(index_blacklist[value]){ @@ -2472,7 +2641,7 @@ partial_score ? - (9 - (threshold || 6)) * context_score + (threshold || 6) * partial_score + (resolution - (threshold || (resolution / 1.5))) * context_score + (threshold || (resolution / 1.5)) * partial_score //(9 - (threshold || 4.5)) * context_score + (threshold || 4.5) * partial_score //4.5 * (context_score + partial_score) : @@ -2483,7 +2652,7 @@ if(score >= threshold){ - let arr = map[9 - ((score + 0.5) >> 0)]; + let arr = map[resolution - ((score + 0.5) >> 0)]; arr = arr[value] || (arr[value] = []); arr[arr.length] = id; @@ -2849,12 +3018,13 @@ /** * @param {!Array>} arrays * @param {number=} limit + * @param {number=} cursor * @param {Object=} docs * @param {boolean=} suggest * @returns {Array} */ - function intersect(arrays, limit, docs, suggest) { + function intersect(arrays, limit, cursor, docs, suggest) { let result = []; let suggestions; @@ -2885,8 +3055,6 @@ while(++z < length_z){ - // get each array one by one - let found = false; const is_final_loop = (z === (length_z - 1)); @@ -2911,10 +3079,21 @@ if(is_final_loop){ - result[count++] = docs ? docs[index] : tmp; + result[count++] = docs ? docs[tmp] : tmp; if(limit && (count === limit)){ + // if(cursor){ + // + // return { + // + // "current": cursor, + // "prev": null, + // "next": z + ":" + i, + // "result": result + // }; + // } + return result; } } @@ -2963,7 +3142,7 @@ for(i = 0, length = tmp.length; i < length; i++){ - result[count++] = docs ? docs["@" + tmp[i]] : tmp[i]; + result[count++] = docs ? docs[tmp[i]] : tmp[i]; if(limit && (count === limit)){ @@ -2991,7 +3170,7 @@ for(let i = 0; i < length; i++){ - result[i] = docs["@" + arr[i]]; + result[i] = docs[arr[i]]; } } else{ diff --git a/package.json b/package.json index d30bbb6..d5c5f65 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "flexsearch", - "version": "0.5.0", + "version": "0.5.1", "description": "Next-Generation full text search library with zero dependencies.", "homepage": "https://github.com/nextapps-de/flexsearch/", "author": "Thomas Wilkerling", @@ -33,7 +33,7 @@ "build-es5": "node compile RELEASE=es5 DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_PRESETS=true SUPPORT_SUGGESTIONS=true SUPPORT_SERIALIZE=true SUPPORT_INFO=true SUPPORT_DOCUMENTS=true SUPPORT_WHERE=true SUPPORT_LANG_DE=false SUPPORT_LANG_EN=false LANGUAGE_OUT=ECMASCRIPT5_STRICT", "build-node": "node compile RELEASE=node DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_PRESETS=true SUPPORT_SUGGESTIONS=true SUPPORT_SERIALIZE=true SUPPORT_INFO=true SUPPORT_DOCUMENTS=true SUPPORT_WHERE=true SUPPORT_LANG_DE=false SUPPORT_LANG_EN=false", "build-lang": "node compile RELEASE=lang", - "build-all": "npm run build && npm run build-light && npm run build-compact && npm run build-es5 && npm run build-node && npm run build-lang", + "build-all": "npm run build && npm run build-light && npm run build-compact && npm run build-es5 && npm run build-node", "test-production": "nyc --reporter=html --reporter=text mocha --timeout=3000 test --exit", "test-light": "nyc --reporter=html --reporter=text mocha --timeout=3000 test/ --exit", "test-develop": "nyc --reporter=html --reporter=text mocha --timeout=3000 --exit", @@ -58,7 +58,7 @@ "chai": "^4.2.0", "codacy-coverage": "^3.4.0", "coveralls": "^3.0.2", - "google-closure-compiler": "^20190124.0.0-nightly", + "google-closure-compiler": "^20190204.0.0-nightly", "mocha": "^5.2.0", "mocha-lcov-reporter": "^1.3.0", "mocha-phantomjs": "^4.1.0", diff --git a/test/benchmark-docs.html b/test/benchmark-docs.html new file mode 100644 index 0000000..8d89e77 --- /dev/null +++ b/test/benchmark-docs.html @@ -0,0 +1,555 @@ + + + + + Benchmark Presets + + + +

Document Index Benchmark Comparison

+

Indexed Text: Movie Documents

+
+
+
+ + + + + diff --git a/test/benchmark-presets.html b/test/benchmark-presets.html index 581d7de..62de25c 100644 --- a/test/benchmark-presets.html +++ b/test/benchmark-presets.html @@ -17,7 +17,6 @@

Presets Benchmark Comparison

-

Indexed Text: "Gulliver's Travels" (Swift Jonathan 1726)

@@ -26,30 +25,28 @@

Indexed Text: "Gulliver's Travels" (Swift Jonathan 1726)

diff --git a/test/benchmark-short.html b/test/benchmark-short.html new file mode 100644 index 0000000..165ab70 --- /dev/null +++ b/test/benchmark-short.html @@ -0,0 +1,463 @@ + + + + + Benchmark + + + +

Benchmark Comparison

+

Indexed Text: Movie Titles

+
+
+
+Test rules: 1. no cache allowed, 2. no async allowed, 3. should return at least 8 matches for the query "The Spirit", 4. result should be ordered by relevance + + + + + + + + + + + + + + diff --git a/test/benchmark.html b/test/benchmark.html index a021cba..0e506df 100644 --- a/test/benchmark.html +++ b/test/benchmark.html @@ -17,7 +17,6 @@

Benchmark Comparison

-

Indexed Text: "Gulliver's Travels" (Swift Jonathan 1726)

@@ -36,19 +35,16 @@

Indexed Text: "Gulliver's Travels" (Swift Jonathan 1726)

diff --git a/test/index.html b/test/index.html index 9c6b997..37a4301 100644 --- a/test/index.html +++ b/test/index.html @@ -22,6 +22,7 @@ var env = "min"; +