Test plan:
With ElasticSearch enabled,
* Perform a search using the default sort order
(i.e. 'relevance').
* Verify that no warnings are generated in
plack-intranet-error.log
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With ES and QueryAutoTruncate on, a search with punctuation surrounded
by spaces (like ISBD punctuation copied from other catalogue or
bibliography) returns no results. E.g.: in a search for "Maria Stuart ;
Die Jungfrau von Orleans / Schiller" (coded in record as 245 10 $a Maria
Stuart ; $b Die Jungfrau von Orleans / $c Schiller) -- both semicolon
and slash cause problems. One had to remove them manually to get
results which is not what is desired.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data
(http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1):
2. Make a search for the first ISBD zone taken verbatim from the first record:
E Street shuffle : the glory days of Bruce Springsteen & the E Street Band / Clinton Heylin
There should be no result.
3. Apply teh patch, restart plack.
4. Repeat the search. You should get the record (and onother one).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We can know add/remove homelibrary and holdinglibray from the config
page, this syspref is no longer needed for ES
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Koha with Zebra prevented auto truncation in some circumstances -- see
the first return for ccl inside C4::Search:: buildQuery, before applying
auto truncation, and setting $auto_truncation = 0 for some search
fields.
Koha with ElasticSearch applies auto truncation for all search fields,
not paying attention to these special cases when it should not be done.
This leads to various problems as described in bug 26508, 26608, etc.
The solution would be to prevent auto truncation for certain search
fields, above all – the identifiers. In addition, under no
circumstances should a search field other than of text type be truncated
(an attempt to truncate would generate an exception from ElasticSearch,
e.g. number_format_exception for integer).
Test plan
=========
0. Use the test data sample provided in the bug report. Comment 16 in
the ticket.
Scenario A (authority)
----------------------
1. Enable Elasticsearch engine.
2. Have like 100+ bibliographic records and properly linked authority
records.
3. Reindex ElasticSearch if needed.
4. Enable QueryAutoTruncate systempreference.
5. Find the authority record #1 and note the number of linked biblio
records.
6. Show the detail of authority #1 and compare the number of linked
biblio records.
7. If in the database there are authority records with ids =~ /^1/ (i.e.
10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get
two different numbers of linked records.
8. Also, as lists of linked biblio records (via link: Used in N
record(s) from results view and detail view) you will get two
different sets of biblio records. In particular, on the list generated
from detail view (authorities/detail.pl?authid=1) you will get biblio
records that are in fact not linked to the auth #1 (the list is
generated with &q=an:1).
8.99. Skip to scenario B and come back here after finishing to not have
to unapply the patches and restart services for nothing.
9. Apply this patch.
10. Counts and list of linked biblios should be ok.
Scenario B (analytics)
----------------------
1. Enable Elasticsearch engine.
2. Have three monographic bibliographic records with 001 = 1, 10, 100
(i.e. =~ /^1/).
3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896).
4. Enable QueryAutoTruncate and UseControlNumber systempreference.
6. Find the analytical record and click on the link generated from 773,
i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see
a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1).
7. From the biblio # 1 try to go to the analytice records (with Show analytics
link) - you should get 60+ records with from different host records (773 $t)
-- instead of one.
8. Apply this patch. Make sure that control-number and record-control-number
are included in ESPreventAutoTruncate syspref.
9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6
and at one analytical records when looking for analytical records in p. 7
(or more than one, but right, if you modified the test data set).
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1) Launch k-t-d with ES:
-- ktd --es7 pull
-- ktd --es7 up
2) Visit /cgi-bin/koha/authorities/authorities-home.pl and search for 'tim'
3) Edit that, add a dash '-' to 100$a, like 'Bunce, Tim - name'"
4) Do a authority search for 'tim - name', notice it errors out
5) Apply patch
6) Perform the same search, notice it doesn't error and returns the correct record
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Instead of typing the case sensitive Control-number each time.
4 strikes instead of 15 on your keyboard. Wow! Gain of 73%.
Test plan:
Copy ccl.properties to /etc/koha/zebradb, restart Zebra and
search for cnum=SOME_ID in opac or intranet.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The feature of searching within search results in librarian
interface has some problems with with elastic: 'Keyword' and
'Keyword as phrase' searches give no results. This is because
elastic is unaware of the search field 'kw' and 'kw' is not
transformed nor removed for limits.
Test plan
=========
1. Have a Koha instance with reasonable number of biblio records
1. Set SearchEngine to Elasticsearch
3. In librarian interface, perform a search that would give
you a list of results
4. With the new search box 'Search within results', having 'Keyword'
and 'Keyword as phrase' active, try to perform a search within
search results with a term that should give you some results
5. You should get no results
6. Apply the patch
7. Repeat steps 3 and 4
8. You should get expected results. Note the difference between
'Keyword' and 'Keyword as phrase' in results
9. Sign off
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch alters the code in the ES QueryBuilder. Reflecting
how things are handled in build_query_compat we clean the query,
but not the limits. In Zebra we simply recursivly call buildQuery,
but the ES query structure
This patchset adds a new ability to save searches on the staff client, and display them in the results
page on staff or opac as a new filter.
New filters can be added from the results page after a search, and there is an admin page for updating
deleting and renaming filters
There is a new permission to control management of these filters
New filters can be added that are not displayed along with facets, this allows for building custom links
using these filters to keep URLs shorter
Due to bug 30528 testing in ES is recommended
To test:
1 - Apply patches and update database and restart all
2 - Enable new system preference 'SavedSearchFilters'
3 - As superlibrarian perform a search in staff client, something broad like 'a'
4 - Note new 'Save search as filter' link on results page
5 - Click it, save search as new filter, check 'Staff client' visibility
6 - Perform another search
7 - Note the filter now appears above facets
8 - Click to it filter results
9 - Note results are limited by the new filter, and it is checked in the facets
10 - Confirm click the [x] removes the filter
11 - Go to administration->search filters
12 - Confirm the filter appears
13 - Edit and mark as OPAC visible
14 - Test OPAC to ensure it shows and can be applied/removed
15 - Copy URL with filter applied
16 - In adminsitration mark filter as not visible on staff or opac
17 - Confirm link above still works
18 - Create a new staff with catalogue and search filters permission
19 - Ensure they can access/save filters
20 - Remove filter permission and ensure they cannot
21 - Disable system preference
22 - Confirm links to search filters page are removed from admin home and admin sidebar
23 - Confirm filters do not appear on results and cannot be created
24 - Enable pref
25 - Create a filter
26 - From search filters page, click 'Edit search'
27 - Confirm you are taken to advanced search page letting you know which filter you are editing
28 - Confirm you can change searhc options and save
29 - Confirm you can perform the search from this page
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch updates the query building code to expand filters when searching
and pass them back as part of the cgi and descriptive search strings
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch allows subject headings from different thesaurus to co-exist in Koha.
Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):
Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"
MARC info:
650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
2 Source of heading or term sao
650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element feminism
2 Source of heading or term bnb
650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Idéhistoria
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Litteratur
Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.
Sponsored-by: Lund University Library, Sweden
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
1 - Enable ES and index
2 - Set expandedSearchOption to 'show'
3 - Advanced search on staff and opac using Year field on staff and Publication date range on OPAC for:
2005
2005-
-2005
<2005
>2005
2005-2010
4 - Confirm ranges don't work on staff side, but do on opac
5 - Apply patch
6 - Repeat searches - they work on both staff and opac
7 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Repeat previous tests with Elasticsearch engine
You will need to reindex and reset mappings to pickup the changes form the file
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When a query with "OR" is combined with a limit in Elasticsearch, the precedence is not preserved and the results are not correct.
To test:
1) Set SearchEngine to Elasticsearch
2) Index records in Elasticsearch
3) Do an advanced search
4) Select More options
5) Enter a value for the first Keyword (e.g. Novels)
6) Change "and" before the second Keyword to "or"
7) Enter another value for the second Keyword (e.g. Prose)
8) Limit the search (e.g. Item type Books)
9) Do the search
10) Observe that records with the first keyword are not in the results
11) Apply the patch
12) Repeat the search
13) Observe that results with both keywords are in the results
14) Sign off
Sponsored-by: Lund University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch includes the parameter:
include_type_name
on our ES calls
It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.
For ES8 we will need to remove the hardcoded type,
I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support
When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds quoting when handling branchcodes in searching in order
to prevent errors when branchcodes are reserved words in ES
To test:
0 - Be using Elasticsearch with Koha
1 - Add a new branch code:OR name:Orly
2 - Add an item to this branch
3 - Use advanced search to limit search to only Orly
4 - Oh really bad, the search fails!
5 - Apply patch
6 - Repeat search
7 - Oh really good, the search succeeds
8 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Starting in Elasticsearch 6.0 _key should be used instead of _term to order
buckets by their term
To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) If you observe an error about types, apply patch for bug 25669
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Observe if you get a deprecation warning about order key
5) Apply patch
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no deprecation warning about order key
8) Sign off
Sponsored-by: Lund University Library
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This expands the regex to handle this specific case
To test:
1 - Load record created for last patch
2 - Note analytics error
3 - Apply patch
4 - Restart and reload
5 - No more errorm also no Analytics link
6 - Add a 773$t to a record with title used before:
Digger does it all (not really!)
7 - reload the initial record
8 - See 'Show analytics' link
9 - Click the link
10 - You should end up on the record you added the 773 to
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
With all the work that's gone into improving the internal
_clean_search_term method I feel we should expose it publically as it's
going to be more widely helpful
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
if QueryAutoTruncate enabled we will have any special operators ruined
for example: "test [6 TO 7]" will be converted to "test* [6* TO* 7]"
so no reason to keep ranges when QueryAutoTruncate set to "enabled"
1) enable QueryAutoTruncate at your sysprefs.
2) perform a search using range, for example: "[1999 TO 2020]",
it shouldn't work the way it's supposed to.
3) apply the patch.
4) perform the same search with range, ensure that it works correctly.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch ensures that the behavior with
QueryRegexEscapeOptions set to values other than
"Escape" still will works as expected.
It does so by storing the contents of regexes
before escaping special characters and
then restores the contents of regexes back to how
it was before, ensuring that searching with regex is possible.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently having exclamation sign at the end of the query makes ES
search fail, and when you try to search for a book that has exclamation
sign in the tittle (something like "Words! words") won't show results
correctly as it tries to negate everything that is after exclamation
sign, making it impossible to search for books that have in in the title
This patch escapes exclamation signs if it's at the end of the query or
has a space after it, resolving both of the issues listed above.
To reproduce:
1) with ES enabled, search for the book with title that contains
exclamation sight at the end, like "book!", this search should result
in error.
2) do another search, but this time find/prepare beforehand book with a
title that has exclamation sign with a space after it,
e.g "exclamation! sign", it shouldn't find it as ES treats everything
after that exclamation sign as negation.
2) apply the patch.
3) perform searches from the steep one and two again.
Search from step one should no longer fail, while search from the step
two should find that book.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch screens square and curly brackets which have no special
language meaning.
To reproduce:
1) using ES, search for the book with title that contains
square and/or curly brackets, like "book [second edition]", which will
result in error.
2) apply the patch.
3) search for that book again, ensure that it works now.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently searches like: "book:", ":book" and "host-item:test:n"
cause internal server errors.
This patch adds additional regexes that remove the colons at the start
and end of the query, and another regex that screens all follow-up
colons that go after the first colon to avoid errors when searching for
"host-item:test:n".
To reproduce:
1) using ES, search for the book with title that contains
semicolon at the start or at the end of the line, separated with spaces,
this should cause internal server error.
2) try doing the same with something like "host-item:test:n", it should
result in error as well.
3) apply the patch.
4) repeat steps 1-2, ensure that it works now.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch removes the code from the search scripts into QueryBuilder
modules.
To test:
1 - Have a library group defined as a search group for both staff and opac
2 - Search on staff client and opac with that group limit and a single branch limit
3 - Note your results/counts
4 - Note the visuals of the search description
5 - Apply patch
6 - Repeat searches
7 - All should work as before
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the cni/Control-number-identifier index to enable
searches to use the 003 field.
Test plan
1/ Apply patch
2/ Re-index using updated configurations
3/ Confirm cni:number searches yield the expected results
4/ Signoff
Split-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In advanced search with Elasticsearch the limit on years range is actually using copydate :
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm in _fix_limit_special_cases() :
if ( $l =~ /^yr,st-numeric,ge=/ ) {
my ( $start, $end ) =
( $l =~ /^yr,st-numeric,ge=(.*) and yr,st-numeric,le=(.*)$/ );
next unless defined($start) && defined($end);
push @new_lim, "copydate:[$start TO $end]";
}
With Zebra it uses date-of-publication and also in Koha/SearchEngine/Elasticsearch/QueryBuilder.pm :
our %index_field_convert = (
(...)
'yr' => 'date-of-publication',
This patch uses %index_field_convert to perform 'yr' limit.
Test plan:
1) Apply patch
2) Use Elasticsearch searchengine
3) Go to advanced search with 'More options'
4) Perform a search with a year limit (value or range)
5) Check results are correct
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In Elasticsearch, query for biblios uses lenient=true.
This is also needed for authorities search.
In case a search field is defined as type numeric.
Test plan :
1) Use Elasticsearch searchengine
2) Define a search field 'local-number' as type 'Number'
3) Be sure to us 'local-number' in autorities mapping
4) Rebuild autorities
5) Performe a search for autorities with 'Search entire record' and
'contains' with term '123'
=> Without patch you get error :
[query_shard_exception] Can only use prefix queries on keyword and text fields - not on [local-number] which is of type [integer]
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
If we are searching on kw there is a leading colon at the beginning of
the generated query:
kw:foo becomes :foo
Note that it only happens when there is no other terms before.
Test plan:
0. Don't apply the patch
1. Search for kw:foo
2. Notice the error
Error: Unable to perform your search. Please try again.
and the logs say
Failed to parse query [(:foo*)]
3. Apply the patch
4. Repeat the search and notice that you know get:
"12 result(s) found for 'kw:foo'."
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In zebra we sort by callnumber using 8007 cn-sort
We should do the same in elasticsearch
To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
It'd be great if the Search Engine Configuration page would display
the various aliases (shortcuts) available : ti for title, sn for local-number, etc.
Patch changes Koha/SearchEngine/Elasticsearch/QueryBuilder.pm to move
hard-coded vars at the beging and adds a method to provide to %index_field_convert via a method.
Test plan :
1) Use Elasticsearch
2) Go to Administration > Search engine configuration (Elasticsearch)
3) Check you see new column 'Aliases' with for example ti for title.
4) Perform a search 'ti:<title>' and check you get results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We convert 'keyword' to '' as an index since we want it to search all fields
When we are searching 'as phrase', however, we should not drop the search type
To test:
1 - Enable IntranetCatalogPullDown
2 - Set searchEngine to Elasticsearch
3 - Perform a search for 'Keyword as phrase' for a phrase that does appear in a record
4 - You get the result
5 - Reverse the order of words in the phrase
6 - You still get a result?
7 - Apply patch
8 - Restart all the things
9 - Reversed search does not return record
10 - Correct order and search, correct record returned
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch prepares Koha to officially no longer support Elasticsearch 5.X
It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature
It updates the about page to add a deprecation warning if a site is running ES5
To test:
1 - Be running Koha with Elasticsearch 5.X
2 - Attempt to search
Error: Unable to perform your search. Please try again.
3 - Apply patch
4 - Update database
5 - Searching works
6 - Find syspref 'ElasticsearchCrossFields'
7 - Enable it
8 - Searching is now broken
9 - Check the about page
10 - you can now see the Elasticsearch version
11 - The systeminformation tab has a deprectaion warning
12 - Set SearchEngine preference to 'Zebra'
13 - View the about page - no warnings
14 - Test again with ES6 - searching should "work" with either pref setting
15 - There should be no warning on about pages
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the 'cross_fields' type to our searches:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-string-query.html#query-string-syntax
Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6
To test:
0 - Set QueryAutoTruncate to 'only if * is added'
1 - Find a record with a title and publisher
2 - Search for a word form the title and confirm the record is returned
3 - Search for a work from the title and the publisher's name
4 - The record is not returned
5 - Apply patch
6 - Repeat #3
7 - The record is returned
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Without this patch we get
Use of uninitialized value $3 in concatenation (.) or string at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 943.
This converts the | OR operator to two different regexes so that the
capture group variables will be defined in every case.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space
To test:
1 - Have Koha using ES
2 - Search for:
ti:chess AND chess
3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
4 - Search for:
ti:chess AND kw:chess
5 - No result
6 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
7 - Repeate search and check page source
8 - search_query has:
title:chess ANDchess
9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
title:chess AND chess
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 24567: (follow-up) Use dollar sign to refer to captures
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This takes care of more occurences of staff client and changes it to
staff interface, including in code comments.
To test:
- I think in this case careful code review is what we look for.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Andreas Jonsson
Janusz Kaczmarek
Jonathan Druart
Katrin Fischer
Mark Hofstetter
Marcel de Rooy
Nick Clemens
Frank Hansen
Tomas Cohen Arazi
Julian Maurice
Kevin Carnes
Fridolin Somers
Martin Renvoize
Petro Vashchuk
Joonas Kylmälä
Katrin Fischer