Test plan:
With ElasticSearch enabled,
* Perform a search using the default sort order
(i.e. 'relevance').
* Verify that no warnings are generated in
plack-intranet-error.log
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1. Apply patch
2. Rerun and make sure the error "Use of uninitialized value $sub6 in pattern match (m//) at /usr/share/koha/lib/Koha/SearchEngine/Elastricsearch.pm" is no longer present.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Initial testing done, it works well, needs an additional sign off from an external party
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The highlight logged-in library in facets feature does not work with
Elasticsearch. This is because facet_title_value contains an additional
" (count)" information and so gives negative results in comparisons made
in facets.inc (line 107).
Test plan:
==========
1. Have a test installation with Elastiscsearch.
2. Set system preference 'DisplayLibraryFacets' to 'both home and holding libraries'
3. Perform a seach that will find items of your logged-in library.
In KTD, search for 'branch:CPL'
4. Check the facet (home and holding libraries) of your logged-in
library is not highlighted
5. Apply the patch ; restart_all.
6. Repeat p. 3. Check the facet (home and holding libraries)--you
should see your logged-in branch highlighted now.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Like IncludeSeeFromInSearches adds see from heading in search (4xx),
new system preference IncludeSeeAlsoFromInSearches adds see also from heading (5xx).
Test plan:
Test on both Elasticsearch and Zebra:
1)
1.1) Create an authority record with heading, see from heading 4xx and see also from heading 5xx
1.2) Use this authority in a biblio record
2)
2.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Don't include"
2.3) Rebuild search engine
2.4) Perform a search on heading text => you find the biblio record
2.5) Perform a search on see from heading => you do not find the biblio record
2.6) Perform a search on see also from heading => you do not find the biblio record
3)
3.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Include"
3.2) Rebuild search engine
3.3) Perform a search on heading text => you find the biblio record
3.4) Perform a search on see from heading => you find the biblio record
3.5) Perform a search on see also from heading => you find the biblio record
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With ES and QueryAutoTruncate on, a search with punctuation surrounded
by spaces (like ISBD punctuation copied from other catalogue or
bibliography) returns no results. E.g.: in a search for "Maria Stuart ;
Die Jungfrau von Orleans / Schiller" (coded in record as 245 10 $a Maria
Stuart ; $b Die Jungfrau von Orleans / $c Schiller) -- both semicolon
and slash cause problems. One had to remove them manually to get
results which is not what is desired.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data
(http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1):
2. Make a search for the first ISBD zone taken verbatim from the first record:
E Street shuffle : the glory days of Bruce Springsteen & the E Street Band / Clinton Heylin
There should be no result.
3. Apply teh patch, restart plack.
4. Repeat the search. You should get the record (and onother one).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
Export the ES fields/mappings using
misc/search_tools/export_elasticsearch_mappings.pl > mappings.yaml
Copy this file to admin/searchengine/elasticsearch/mappings.yaml
Reset the mapping from the UI
Notice that the links to the AV categories have been preserved
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature adds the ability to link an ES facet to an authorised
value category, in order to display a description instead of the code.
Test plan:
1. Go to the ES mapping admin page
2. Notice the new "Authorized value category" column in the facets
config table
3. Create a new facet that is linked to a search field containing values
from an authorised value category. Pick this category in the list
4. Save
5. Reindex
6. Notice that the descriptions are now displayed (staff and OPAC) in
the search result page.
QA: See the related bug report for possible improvements. It's not
easy at all to deal with the "specials" here, a lot of changes must be
done in how we fetch/cache the AVs.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.
To test:
0 - Setup Koha with Elasticsearch
1 - Import the sample record on this report: "L'amour de l'art"
2 - Search for "amour de l'art" - no results
3 - Apply patch
4 - Reindex
5 - Search for "amour de l'art" - result!
6 - Search for "title:amour de l'art" - result!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can know add/remove homelibrary and holdinglibray from the config
page, this syspref is no longer needed for ES
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Koha with Zebra prevented auto truncation in some circumstances -- see
the first return for ccl inside C4::Search:: buildQuery, before applying
auto truncation, and setting $auto_truncation = 0 for some search
fields.
Koha with ElasticSearch applies auto truncation for all search fields,
not paying attention to these special cases when it should not be done.
This leads to various problems as described in bug 26508, 26608, etc.
The solution would be to prevent auto truncation for certain search
fields, above all – the identifiers. In addition, under no
circumstances should a search field other than of text type be truncated
(an attempt to truncate would generate an exception from ElasticSearch,
e.g. number_format_exception for integer).
Test plan
=========
0. Use the test data sample provided in the bug report. Comment 16 in
the ticket.
Scenario A (authority)
----------------------
1. Enable Elasticsearch engine.
2. Have like 100+ bibliographic records and properly linked authority
records.
3. Reindex ElasticSearch if needed.
4. Enable QueryAutoTruncate systempreference.
5. Find the authority record #1 and note the number of linked biblio
records.
6. Show the detail of authority #1 and compare the number of linked
biblio records.
7. If in the database there are authority records with ids =~ /^1/ (i.e.
10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get
two different numbers of linked records.
8. Also, as lists of linked biblio records (via link: Used in N
record(s) from results view and detail view) you will get two
different sets of biblio records. In particular, on the list generated
from detail view (authorities/detail.pl?authid=1) you will get biblio
records that are in fact not linked to the auth #1 (the list is
generated with &q=an:1).
8.99. Skip to scenario B and come back here after finishing to not have
to unapply the patches and restart services for nothing.
9. Apply this patch.
10. Counts and list of linked biblios should be ok.
Scenario B (analytics)
----------------------
1. Enable Elasticsearch engine.
2. Have three monographic bibliographic records with 001 = 1, 10, 100
(i.e. =~ /^1/).
3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896).
4. Enable QueryAutoTruncate and UseControlNumber systempreference.
6. Find the analytical record and click on the link generated from 773,
i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see
a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1).
7. From the biblio # 1 try to go to the analytice records (with Show analytics
link) - you should get 60+ records with from different host records (773 $t)
-- instead of one.
8. Apply this patch. Make sure that control-number and record-control-number
are included in ESPreventAutoTruncate syspref.
9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6
and at one analytical records when looking for analytical records in p. 7
(or more than one, but right, if you modified the test data set).
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/
This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml
If blank we default to 5000.
To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:
{
total => {
value => 1,
relation => 'gte'
}
}
Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)
Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220
We might need to find an alternative for future versions of
Elasticsearch
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
Tested with Search::Elasticsearch 6.80 and 7.717
Test plan:
1. Install Elasticsearch 8
(use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
before
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
1) Launch k-t-d with ES:
-- ktd --es7 pull
-- ktd --es7 up
2) Visit /cgi-bin/koha/authorities/authorities-home.pl and search for 'tim'
3) Edit that, add a dash '-' to 100$a, like 'Bunce, Tim - name'"
4) Do a authority search for 'tim - name', notice it errors out
5) Apply patch
6) Perform the same search, notice it doesn't error and returns the correct record
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Instead of typing the case sensitive Control-number each time.
4 strikes instead of 15 on your keyboard. Wow! Gain of 73%.
Test plan:
Copy ccl.properties to /etc/koha/zebradb, restart Zebra and
search for cnum=SOME_ID in opac or intranet.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
becomes as title__sort:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
instead of:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element. Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp. And this is obviously not
what is intended and results (in some cases) with bad sort order
when sorting by titles.
Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
you should have 245abp for title search field.
2. Insert / download / modify a record with 245 $b or $p,
an article at the beginning of the title, and correctly
set 2nd indicator. For instance:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
3. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It will be:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)
5. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It should
be now correct, i.e.:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The feature of searching within search results in librarian
interface has some problems with with elastic: 'Keyword' and
'Keyword as phrase' searches give no results. This is because
elastic is unaware of the search field 'kw' and 'kw' is not
transformed nor removed for limits.
Test plan
=========
1. Have a Koha instance with reasonable number of biblio records
1. Set SearchEngine to Elasticsearch
3. In librarian interface, perform a search that would give
you a list of results
4. With the new search box 'Search within results', having 'Keyword'
and 'Keyword as phrase' active, try to perform a search within
search results with a term that should give you some results
5. You should get no results
6. Apply the patch
7. Repeat steps 3 and 4
8. You should get expected results. Note the difference between
'Keyword' and 'Keyword as phrase' in results
9. Sign off
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch alters the code in the ES QueryBuilder. Reflecting
how things are handled in build_query_compat we clean the query,
but not the limits. In Zebra we simply recursivly call buildQuery,
but the ES query structure
This patchset adds a new ability to save searches on the staff client, and display them in the results
page on staff or opac as a new filter.
New filters can be added from the results page after a search, and there is an admin page for updating
deleting and renaming filters
There is a new permission to control management of these filters
New filters can be added that are not displayed along with facets, this allows for building custom links
using these filters to keep URLs shorter
Due to bug 30528 testing in ES is recommended
To test:
1 - Apply patches and update database and restart all
2 - Enable new system preference 'SavedSearchFilters'
3 - As superlibrarian perform a search in staff client, something broad like 'a'
4 - Note new 'Save search as filter' link on results page
5 - Click it, save search as new filter, check 'Staff client' visibility
6 - Perform another search
7 - Note the filter now appears above facets
8 - Click to it filter results
9 - Note results are limited by the new filter, and it is checked in the facets
10 - Confirm click the [x] removes the filter
11 - Go to administration->search filters
12 - Confirm the filter appears
13 - Edit and mark as OPAC visible
14 - Test OPAC to ensure it shows and can be applied/removed
15 - Copy URL with filter applied
16 - In adminsitration mark filter as not visible on staff or opac
17 - Confirm link above still works
18 - Create a new staff with catalogue and search filters permission
19 - Ensure they can access/save filters
20 - Remove filter permission and ensure they cannot
21 - Disable system preference
22 - Confirm links to search filters page are removed from admin home and admin sidebar
23 - Confirm filters do not appear on results and cannot be created
24 - Enable pref
25 - Create a filter
26 - From search filters page, click 'Edit search'
27 - Confirm you are taken to advanced search page letting you know which filter you are editing
28 - Confirm you can change searhc options and save
29 - Confirm you can perform the search from this page
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch updates the query building code to expand filters when searching
and pass them back as part of the cgi and descriptive search strings
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch strips the blank facets from the ES aggregations, this seems
to only affect language fields currently, but could affect any facetable field
that can contain blank values
To test:
1 - Have Koha running Elasticsearch with standard test data,
or add some records with blank values in the 008/35-37
2 - Search for 'a'
3 - Look at the language facets, they should appear normal
4 - Browse to:
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=a&sort_by=relevance_dsc&count=20&limit=ln:
5 - Note there is now an 'x' under the Languages facet - to remove a limit you cannot see
6 - Apply patch
7 - Reload
8 - Note the x is not there
9 - Inspect HTML, confirm no blank link under language facets
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch allows subject headings from different thesaurus to co-exist in Koha.
Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):
Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"
MARC info:
650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
2 Source of heading or term sao
650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element feminism
2 Source of heading or term bnb
650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Idéhistoria
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Litteratur
Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.
Sponsored-by: Lund University Library, Sweden
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In Zebra the availability limit code states:
'available' is defined as (items.onloan is NULL) and (items.itemlost = 0)
We should make ES follow the same behaviour so that changing the engine does not
change this.
To test:
1 - Find or create several records with two items each
2 - For those records setup items as:
a - one checked out, one available
b - both marked lost
c - both damaged
d - both withdrawn
e - both notforloan
3 - Enable ES and reindex to confirm everything is set
4 - Perform an advanced search to return those records and 'limit to available items'
5 - Confirm you see records b,c,d,e and records say '2 items, none available'
6 - Apply patch
7 - Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
8 - Perform an advanced search to return those records and 'limit to available items'
9 - Confirm you see records a,b,c,d and al but a say '2 items, none available'
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
1 - Enable ES and index
2 - Set expandedSearchOption to 'show'
3 - Advanced search on staff and opac using Year field on staff and Publication date range on OPAC for:
2005
2005-
-2005
<2005
>2005
2005-2010
4 - Confirm ranges don't work on staff side, but do on opac
5 - Apply patch
6 - Repeat searches - they work on both staff and opac
7 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Repeat previous tests with Elasticsearch engine
You will need to reindex and reset mappings to pickup the changes form the file
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Includes:
Bug 29697: (follow-up) Use flag embed_items
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We noticed that several characters will break Zebra queries. So search
terms need to be quoted for things to work. In this context, double
quotes inside search terms are problematic because double quotes are
what we use for quoting strings.
This patch makes the clean_search_term method escape double quotes.
To test:
1. Apply the unit tests patch
2. Run:
$ kshell
k$ prove t/Koha/SearchEngine/Zebra/QueryBuilder.t
=> FAIL: It doesn't work as it should!
3. Apply this patch
4. Repeat 2
=> SUCCESS: It does the job!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When a query with "OR" is combined with a limit in Elasticsearch, the precedence is not preserved and the results are not correct.
To test:
1) Set SearchEngine to Elasticsearch
2) Index records in Elasticsearch
3) Do an advanced search
4) Select More options
5) Enter a value for the first Keyword (e.g. Novels)
6) Change "and" before the second Keyword to "or"
7) Enter another value for the second Keyword (e.g. Prose)
8) Limit the search (e.g. Item type Books)
9) Do the search
10) Observe that records with the first keyword are not in the results
11) Apply the patch
12) Repeat the search
13) Observe that results with both keywords are in the results
14) Sign off
Sponsored-by: Lund University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch includes the parameter:
include_type_name
on our ES calls
It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.
For ES8 we will need to remove the hardcoded type,
I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support
When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds quoting when handling branchcodes in searching in order
to prevent errors when branchcodes are reserved words in ES
To test:
0 - Be using Elasticsearch with Koha
1 - Add a new branch code:OR name:Orly
2 - Add an item to this branch
3 - Use advanced search to limit search to only Orly
4 - Oh really bad, the search fails!
5 - Apply patch
6 - Repeat search
7 - Oh really good, the search succeeds
8 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In C4::AuthoritiesMarc::merge, we search all biblio records using an authority with Koha::Authorities->linked_biblionumbers().
This method uses simple_search_compat() with no results limit, even saying in comment :
44d6528b56/Koha/Authorities.pm (L80)
With Zebra searchengin it is correct.
But not with Elasticsearch, there is an hardcoded limit at 100 in case no limit is given :
44d6528b56/Koha/SearchEngine/Elasticsearch/Search.pm (L346)
This means authorities links are wrong after a merge or an edit in case the authority is used in more than 100 biblio records.
:(
I propose to fix by using by default the real server max given by Koha::SearchEngine::Elasticsearch::Search::max_result_window().
This will allow a huge limit nearly impossible to reach.
See Bug 30882 to how increase this limit.
Test plan :
1) Use Elasticsearch search engine
2) Use an authority id=111 linked to 200 biblio records
3) Perform a search 'an:111', you get 200 results
4) Create a new authority id=222 linked to 2 biblio records
5) Perform a search 'an:222', you get 2 results
6) Perform a merge of the two authorties, keeping id=222
7) Perform a search 'an:111'
without patch you get 100 results
with patch you get no results
8) Perform a search 'an:222'
without patch you get 102 results
with patch you get 202 results
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Andreas Jonsson
Catalyst Bug Squasher
Janusz Kaczmarek
Kyle M Hall
Fridolin Somers
Lari Strand
Jonathan Druart
Nick Clemens
Katrin Fischer
Mark Hofstetter
Julian Maurice
Marcel de Rooy
Frank Hansen
Tomas Cohen Arazi
Kevin Carnes