Koha with Zebra prevented auto truncation in some circumstances -- see
the first return for ccl inside C4::Search:: buildQuery, before applying
auto truncation, and setting $auto_truncation = 0 for some search
fields.
Koha with ElasticSearch applies auto truncation for all search fields,
not paying attention to these special cases when it should not be done.
This leads to various problems as described in bug 26508, 26608, etc.
The solution would be to prevent auto truncation for certain search
fields, above all – the identifiers. In addition, under no
circumstances should a search field other than of text type be truncated
(an attempt to truncate would generate an exception from ElasticSearch,
e.g. number_format_exception for integer).
Test plan
=========
0. Use the test data sample provided in the bug report. Comment 16 in
the ticket.
Scenario A (authority)
----------------------
1. Enable Elasticsearch engine.
2. Have like 100+ bibliographic records and properly linked authority
records.
3. Reindex ElasticSearch if needed.
4. Enable QueryAutoTruncate systempreference.
5. Find the authority record #1 and note the number of linked biblio
records.
6. Show the detail of authority #1 and compare the number of linked
biblio records.
7. If in the database there are authority records with ids =~ /^1/ (i.e.
10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get
two different numbers of linked records.
8. Also, as lists of linked biblio records (via link: Used in N
record(s) from results view and detail view) you will get two
different sets of biblio records. In particular, on the list generated
from detail view (authorities/detail.pl?authid=1) you will get biblio
records that are in fact not linked to the auth #1 (the list is
generated with &q=an:1).
8.99. Skip to scenario B and come back here after finishing to not have
to unapply the patches and restart services for nothing.
9. Apply this patch.
10. Counts and list of linked biblios should be ok.
Scenario B (analytics)
----------------------
1. Enable Elasticsearch engine.
2. Have three monographic bibliographic records with 001 = 1, 10, 100
(i.e. =~ /^1/).
3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896).
4. Enable QueryAutoTruncate and UseControlNumber systempreference.
6. Find the analytical record and click on the link generated from 773,
i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see
a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1).
7. From the biblio # 1 try to go to the analytice records (with Show analytics
link) - you should get 60+ records with from different host records (773 $t)
-- instead of one.
8. Apply this patch. Make sure that control-number and record-control-number
are included in ESPreventAutoTruncate syspref.
9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6
and at one analytical records when looking for analytical records in p. 7
(or more than one, but right, if you modified the test data set).
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/
This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml
If blank we default to 5000.
To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:
{
total => {
value => 1,
relation => 'gte'
}
}
Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)
Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220
We might need to find an alternative for future versions of
Elasticsearch
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
Tested with Search::Elasticsearch 6.80 and 7.717
Test plan:
1. Install Elasticsearch 8
(use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
before
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
1) Launch k-t-d with ES:
-- ktd --es7 pull
-- ktd --es7 up
2) Visit /cgi-bin/koha/authorities/authorities-home.pl and search for 'tim'
3) Edit that, add a dash '-' to 100$a, like 'Bunce, Tim - name'"
4) Do a authority search for 'tim - name', notice it errors out
5) Apply patch
6) Perform the same search, notice it doesn't error and returns the correct record
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Instead of typing the case sensitive Control-number each time.
4 strikes instead of 15 on your keyboard. Wow! Gain of 73%.
Test plan:
Copy ccl.properties to /etc/koha/zebradb, restart Zebra and
search for cnum=SOME_ID in opac or intranet.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
becomes as title__sort:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
instead of:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element. Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp. And this is obviously not
what is intended and results (in some cases) with bad sort order
when sorting by titles.
Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
you should have 245abp for title search field.
2. Insert / download / modify a record with 245 $b or $p,
an article at the beginning of the title, and correctly
set 2nd indicator. For instance:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
3. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It will be:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)
5. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It should
be now correct, i.e.:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The feature of searching within search results in librarian
interface has some problems with with elastic: 'Keyword' and
'Keyword as phrase' searches give no results. This is because
elastic is unaware of the search field 'kw' and 'kw' is not
transformed nor removed for limits.
Test plan
=========
1. Have a Koha instance with reasonable number of biblio records
1. Set SearchEngine to Elasticsearch
3. In librarian interface, perform a search that would give
you a list of results
4. With the new search box 'Search within results', having 'Keyword'
and 'Keyword as phrase' active, try to perform a search within
search results with a term that should give you some results
5. You should get no results
6. Apply the patch
7. Repeat steps 3 and 4
8. You should get expected results. Note the difference between
'Keyword' and 'Keyword as phrase' in results
9. Sign off
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch alters the code in the ES QueryBuilder. Reflecting
how things are handled in build_query_compat we clean the query,
but not the limits. In Zebra we simply recursivly call buildQuery,
but the ES query structure
This patchset adds a new ability to save searches on the staff client, and display them in the results
page on staff or opac as a new filter.
New filters can be added from the results page after a search, and there is an admin page for updating
deleting and renaming filters
There is a new permission to control management of these filters
New filters can be added that are not displayed along with facets, this allows for building custom links
using these filters to keep URLs shorter
Due to bug 30528 testing in ES is recommended
To test:
1 - Apply patches and update database and restart all
2 - Enable new system preference 'SavedSearchFilters'
3 - As superlibrarian perform a search in staff client, something broad like 'a'
4 - Note new 'Save search as filter' link on results page
5 - Click it, save search as new filter, check 'Staff client' visibility
6 - Perform another search
7 - Note the filter now appears above facets
8 - Click to it filter results
9 - Note results are limited by the new filter, and it is checked in the facets
10 - Confirm click the [x] removes the filter
11 - Go to administration->search filters
12 - Confirm the filter appears
13 - Edit and mark as OPAC visible
14 - Test OPAC to ensure it shows and can be applied/removed
15 - Copy URL with filter applied
16 - In adminsitration mark filter as not visible on staff or opac
17 - Confirm link above still works
18 - Create a new staff with catalogue and search filters permission
19 - Ensure they can access/save filters
20 - Remove filter permission and ensure they cannot
21 - Disable system preference
22 - Confirm links to search filters page are removed from admin home and admin sidebar
23 - Confirm filters do not appear on results and cannot be created
24 - Enable pref
25 - Create a filter
26 - From search filters page, click 'Edit search'
27 - Confirm you are taken to advanced search page letting you know which filter you are editing
28 - Confirm you can change searhc options and save
29 - Confirm you can perform the search from this page
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch updates the query building code to expand filters when searching
and pass them back as part of the cgi and descriptive search strings
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch strips the blank facets from the ES aggregations, this seems
to only affect language fields currently, but could affect any facetable field
that can contain blank values
To test:
1 - Have Koha running Elasticsearch with standard test data,
or add some records with blank values in the 008/35-37
2 - Search for 'a'
3 - Look at the language facets, they should appear normal
4 - Browse to:
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=a&sort_by=relevance_dsc&count=20&limit=ln:
5 - Note there is now an 'x' under the Languages facet - to remove a limit you cannot see
6 - Apply patch
7 - Reload
8 - Note the x is not there
9 - Inspect HTML, confirm no blank link under language facets
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch allows subject headings from different thesaurus to co-exist in Koha.
Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):
Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"
MARC info:
650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
2 Source of heading or term sao
650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element feminism
2 Source of heading or term bnb
650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Feminism
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Idéhistoria
650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
a Topical term or geographic name entry element Litteratur
Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.
Sponsored-by: Lund University Library, Sweden
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In Zebra the availability limit code states:
'available' is defined as (items.onloan is NULL) and (items.itemlost = 0)
We should make ES follow the same behaviour so that changing the engine does not
change this.
To test:
1 - Find or create several records with two items each
2 - For those records setup items as:
a - one checked out, one available
b - both marked lost
c - both damaged
d - both withdrawn
e - both notforloan
3 - Enable ES and reindex to confirm everything is set
4 - Perform an advanced search to return those records and 'limit to available items'
5 - Confirm you see records b,c,d,e and records say '2 items, none available'
6 - Apply patch
7 - Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
8 - Perform an advanced search to return those records and 'limit to available items'
9 - Confirm you see records a,b,c,d and al but a say '2 items, none available'
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
1 - Enable ES and index
2 - Set expandedSearchOption to 'show'
3 - Advanced search on staff and opac using Year field on staff and Publication date range on OPAC for:
2005
2005-
-2005
<2005
>2005
2005-2010
4 - Confirm ranges don't work on staff side, but do on opac
5 - Apply patch
6 - Repeat searches - they work on both staff and opac
7 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Repeat previous tests with Elasticsearch engine
You will need to reindex and reset mappings to pickup the changes form the file
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Includes:
Bug 29697: (follow-up) Use flag embed_items
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We noticed that several characters will break Zebra queries. So search
terms need to be quoted for things to work. In this context, double
quotes inside search terms are problematic because double quotes are
what we use for quoting strings.
This patch makes the clean_search_term method escape double quotes.
To test:
1. Apply the unit tests patch
2. Run:
$ kshell
k$ prove t/Koha/SearchEngine/Zebra/QueryBuilder.t
=> FAIL: It doesn't work as it should!
3. Apply this patch
4. Repeat 2
=> SUCCESS: It does the job!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When a query with "OR" is combined with a limit in Elasticsearch, the precedence is not preserved and the results are not correct.
To test:
1) Set SearchEngine to Elasticsearch
2) Index records in Elasticsearch
3) Do an advanced search
4) Select More options
5) Enter a value for the first Keyword (e.g. Novels)
6) Change "and" before the second Keyword to "or"
7) Enter another value for the second Keyword (e.g. Prose)
8) Limit the search (e.g. Item type Books)
9) Do the search
10) Observe that records with the first keyword are not in the results
11) Apply the patch
12) Repeat the search
13) Observe that results with both keywords are in the results
14) Sign off
Sponsored-by: Lund University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch includes the parameter:
include_type_name
on our ES calls
It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.
For ES8 we will need to remove the hardcoded type,
I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support
When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds quoting when handling branchcodes in searching in order
to prevent errors when branchcodes are reserved words in ES
To test:
0 - Be using Elasticsearch with Koha
1 - Add a new branch code:OR name:Orly
2 - Add an item to this branch
3 - Use advanced search to limit search to only Orly
4 - Oh really bad, the search fails!
5 - Apply patch
6 - Repeat search
7 - Oh really good, the search succeeds
8 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In C4::AuthoritiesMarc::merge, we search all biblio records using an authority with Koha::Authorities->linked_biblionumbers().
This method uses simple_search_compat() with no results limit, even saying in comment :
44d6528b56/Koha/Authorities.pm (L80)
With Zebra searchengin it is correct.
But not with Elasticsearch, there is an hardcoded limit at 100 in case no limit is given :
44d6528b56/Koha/SearchEngine/Elasticsearch/Search.pm (L346)
This means authorities links are wrong after a merge or an edit in case the authority is used in more than 100 biblio records.
:(
I propose to fix by using by default the real server max given by Koha::SearchEngine::Elasticsearch::Search::max_result_window().
This will allow a huge limit nearly impossible to reach.
See Bug 30882 to how increase this limit.
Test plan :
1) Use Elasticsearch search engine
2) Use an authority id=111 linked to 200 biblio records
3) Perform a search 'an:111', you get 200 results
4) Create a new authority id=222 linked to 2 biblio records
5) Perform a search 'an:222', you get 2 results
6) Perform a merge of the two authorties, keeping id=222
7) Perform a search 'an:111'
without patch you get 100 results
with patch you get no results
8) Perform a search 'an:222'
without patch you get 102 results
with patch you get 202 results
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Can't use an undefined value as an ARRAY reference at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 121.
Caught that when working on follow-up bug reports.
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch adds a background job submodule, UpdateElasticIndex, to deal
with async ES index update (not the deletion).
Using NYTProf (on a checkin):
Without
618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals.
With
521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals.
However there are some problems with this patch:
1. We don't want *all* the index update to be in the background_jobs
tabtle (we could add a filter on the list view)
2. We don't track the "progress" of the job as we are sending all the
records to Elastic. It is okish in my opinion but it must be noted.
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
In Elasticsearch 7 hits.total is now an object which is not always an exact
value. You can always get an exact total by adding a track_total_hits
parameter set to true when using the Elasticsearch search method
To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) If you observe an error about types, apply patch for bug 25669
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Observe that tests with count fail
5) Apply patch
6) Observe that tests with count pass
7) Sign off
Sponsored-by: Lund University Library
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Starting in Elasticsearch 6.0 _key should be used instead of _term to order
buckets by their term
To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) If you observe an error about types, apply patch for bug 25669
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Observe if you get a deprecation warning about order key
5) Apply patch
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no deprecation warning about order key
8) Sign off
Sponsored-by: Lund University Library
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
and some more...
There are lot of inconsistencies in our ->search calls. We could
simplify some of them, but not in this patch. Here we want to prevent
regressions as much as possible and so don't add unecessary changes.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
I need to add a 'request_timeout' entry to the ES configuration to fix some timeout problems on the sandboxes.
Instead of hard coding this new option it seems preferable to be
flexible and allow different options to be passed from the config file.
Test plan:
Add to the $KOHA_CONF, inside the elasticsearch section
<request_timeout>60</request_timeout>
Rebuild the ES index
koha-elasticsearch --rebuild kohadev
Change the value of the timeout to 1
Rebuild the index
It should fail (with a quite bad error 'Bad response received when
submitting request to Elasticsearch')
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This expands the regex to handle this specific case
To test:
1 - Load record created for last patch
2 - Note analytics error
3 - Apply patch
4 - Restart and reload
5 - No more errorm also no Analytics link
6 - Add a 773$t to a record with title used before:
Digger does it all (not really!)
7 - reload the initial record
8 - See 'Show analytics' link
9 - Click the link
10 - You should end up on the record you added the 773 to
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
1. Create an authority with at least a 1XX$a and a 4XX$a, for instance:
100 $a Foo
400 $a Bar
2. Create a biblio and add a link to this authority using the
cataloguing plugin
3. Disable syspref IncludeSeeFromInSearches
4. Reindex the biblio. You should be able to find it when searching
'Foo' but not when searching 'Bar'
5. Enable syspref IncludeSeeFromInSearches
6. Reindex the biblio. You should be able to find it when searching
'Foo' and also when searching 'Bar'
without the patch, 'Bar' doesn't yeld results. With it, it does.
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This bug adds a system preference to control ordering of facets and
adds the control to both Zebra and Elasticsearch
To test:
1 - Have a koha that can use both Zebra and ES
2 - Set 'displayFacetCount' to true
3 - Search in ES and Zebra
4 - Note facets in Zebra sorted alphabetically, ES by usage
5 - Apply patch, updatedatabase
6 - Search in ES and Zebra, facets are alphabetically sorted in both
7 - Find new syspref FacetOrder and set to 'by usage'
8 - Search in both engines, facets sorted by usage
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
- Create a biblio with title like "osteuropa:" or "osteuropa!"
- Go the this biblio detail pages (cgi-bin/koha/catalogue/detail.pl)
=> Error
- Apply bug 28316 and this one
- test again
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
With all the work that's gone into improving the internal
_clean_search_term method I feel we should expose it publically as it's
going to be more widely helpful
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
if QueryAutoTruncate enabled we will have any special operators ruined
for example: "test [6 TO 7]" will be converted to "test* [6* TO* 7]"
so no reason to keep ranges when QueryAutoTruncate set to "enabled"
1) enable QueryAutoTruncate at your sysprefs.
2) perform a search using range, for example: "[1999 TO 2020]",
it shouldn't work the way it's supposed to.
3) apply the patch.
4) perform the same search with range, ensure that it works correctly.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch ensures that the behavior with
QueryRegexEscapeOptions set to values other than
"Escape" still will works as expected.
It does so by storing the contents of regexes
before escaping special characters and
then restores the contents of regexes back to how
it was before, ensuring that searching with regex is possible.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently having exclamation sign at the end of the query makes ES
search fail, and when you try to search for a book that has exclamation
sign in the tittle (something like "Words! words") won't show results
correctly as it tries to negate everything that is after exclamation
sign, making it impossible to search for books that have in in the title
This patch escapes exclamation signs if it's at the end of the query or
has a space after it, resolving both of the issues listed above.
To reproduce:
1) with ES enabled, search for the book with title that contains
exclamation sight at the end, like "book!", this search should result
in error.
2) do another search, but this time find/prepare beforehand book with a
title that has exclamation sign with a space after it,
e.g "exclamation! sign", it shouldn't find it as ES treats everything
after that exclamation sign as negation.
2) apply the patch.
3) perform searches from the steep one and two again.
Search from step one should no longer fail, while search from the step
two should find that book.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch screens square and curly brackets which have no special
language meaning.
To reproduce:
1) using ES, search for the book with title that contains
square and/or curly brackets, like "book [second edition]", which will
result in error.
2) apply the patch.
3) search for that book again, ensure that it works now.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently searches like: "book:", ":book" and "host-item:test:n"
cause internal server errors.
This patch adds additional regexes that remove the colons at the start
and end of the query, and another regex that screens all follow-up
colons that go after the first colon to avoid errors when searching for
"host-item:test:n".
To reproduce:
1) using ES, search for the book with title that contains
semicolon at the start or at the end of the line, separated with spaces,
this should cause internal server error.
2) try doing the same with something like "host-item:test:n", it should
result in error as well.
3) apply the patch.
4) repeat steps 1-2, ensure that it works now.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch removes the code from the search scripts into QueryBuilder
modules.
To test:
1 - Have a library group defined as a search group for both staff and opac
2 - Search on staff client and opac with that group limit and a single branch limit
3 - Note your results/counts
4 - Note the visuals of the search description
5 - Apply patch
6 - Repeat searches
7 - All should work as before
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
There is existing error message in the code stating:
"Unable to understand your search query, please rephrase and try again."
which fits perfectly but because it looks for "ParseException" in the
warning output it doesn't show up on this page as it's actually
"parse_exception".
This patch makes that it's also checked if "parse_exception" is present
in the warning output.
Side note:
"ParseException" reaction code was added here:
e0f6c4dc Bug 12478: improve error reporting a bit
Search::Elasticsearch seems propagates clean ES JSON answer,
and in current ES version inside of $@ it contains "parse_exception"
string in dumped JSON answer ("'type' => 'parse_exception'").
Old seeked phrase "ParseException" wasn't reproduced, only in ES logs
("Caused by: org.apache.lucene.queryparser.classic.ParseException:
Cannot parse ..."). Check for both phrases won't complicate future
changes, but this note added for reference and code cleanup if needed.
To reproduce:
1) using ES search for something like "// ^ ! { } [ ] .. , <>" that
will for sure break the syntax of ES.
2) after the search query fails note that the error is
"Unable to perform your search. Please try again."
3) apply the patch
4) search for the same thing again
5) error message should be "Unable to understand your search query,
please rephrase and try again." now.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Andrew Nugged <nugged@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the cni/Control-number-identifier index to enable
searches to use the 003 field.
Test plan
1/ Apply patch
2/ Re-index using updated configurations
3/ Confirm cni:number searches yield the expected results
4/ Signoff
Split-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In advanced search with Elasticsearch the limit on years range is actually using copydate :
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm in _fix_limit_special_cases() :
if ( $l =~ /^yr,st-numeric,ge=/ ) {
my ( $start, $end ) =
( $l =~ /^yr,st-numeric,ge=(.*) and yr,st-numeric,le=(.*)$/ );
next unless defined($start) && defined($end);
push @new_lim, "copydate:[$start TO $end]";
}
With Zebra it uses date-of-publication and also in Koha/SearchEngine/Elasticsearch/QueryBuilder.pm :
our %index_field_convert = (
(...)
'yr' => 'date-of-publication',
This patch uses %index_field_convert to perform 'yr' limit.
Test plan:
1) Apply patch
2) Use Elasticsearch searchengine
3) Go to advanced search with 'More options'
4) Perform a search with a year limit (value or range)
5) Check results are correct
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In Elasticsearch, query for biblios uses lenient=true.
This is also needed for authorities search.
In case a search field is defined as type numeric.
Test plan :
1) Use Elasticsearch searchengine
2) Define a search field 'local-number' as type 'Number'
3) Be sure to us 'local-number' in autorities mapping
4) Rebuild autorities
5) Performe a search for autorities with 'Search entire record' and
'contains' with term '123'
=> Without patch you get error :
[query_shard_exception] Can only use prefix queries on keyword and text fields - not on [local-number] which is of type [integer]
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
'error' has special meaning in exceptions so naming the fields:
type, details
Rather than only dealing with a single exception type, we generically
get the ES exception info and pass it up.
I could not recreate timeout still, however, I simply restarted the
ES docker during commit stage to cause NoNodes exceptions
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This catches a timeout response from the ES server, logs this, and continues the indexing
To test:
1 - perl misc/search_tools/rebuild_elasticsearch.pl
2 - Make the ES server timeout (I don't have good instruction yet)
3 - Watch the job crash
4 - Apply patches
5 - perl misc/search_tools/rebuild_elasticsearch.pl
6 - Make the server timeout
7 - Note the job reports failed commit, and continues
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 26312: (follow-up) Reset buffers even if commit fails
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 26312: (follow-up) Fix whitespace and missing semicolon
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
If we are searching on kw there is a leading colon at the beginning of
the generated query:
kw:foo becomes :foo
Note that it only happens when there is no other terms before.
Test plan:
0. Don't apply the patch
1. Search for kw:foo
2. Notice the error
Error: Unable to perform your search. Please try again.
and the logs say
Failed to parse query [(:foo*)]
3. Apply the patch
4. Repeat the search and notice that you know get:
"12 result(s) found for 'kw:foo'."
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
koha-elastic --rebuild was failing with
Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Could not convert [marc_data.index] to boolean"
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Subdivision authorities are not used for linking, however, they are recognized by
C4::AuthoritiesMarc
While these records are not used for linking, they could provide reference and
should be allowed to exist in the catalog without breaking ES indexing
THis patch simply skips the step of parsing the authorities into the linking form
if the type contains '_SUBD'
To test:
1 - Import a subdivision authority record via Z39 or use the one attached to this bug
2 - perl misc/search_tools/rebuild_elasticsearch.pl -v -d
3 - Authority indexing dies:
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 202.
Use of uninitialized value $tag in regexp compilation at /usr/share/perl5/MARC/Record.pm line 206.
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 207.
Can't call method "tag" on an undefined value at /kohadevbox/koha/C4/Heading.pm line 71.
4 - Apply patches
5 - reindex
6 - Success!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In zebra we sort by callnumber using 8007 cn-sort
We should do the same in elasticsearch
To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
It'd be great if the Search Engine Configuration page would display
the various aliases (shortcuts) available : ti for title, sn for local-number, etc.
Patch changes Koha/SearchEngine/Elasticsearch/QueryBuilder.pm to move
hard-coded vars at the beging and adds a method to provide to %index_field_convert via a method.
Test plan :
1) Use Elasticsearch
2) Go to Administration > Search engine configuration (Elasticsearch)
3) Check you see new column 'Aliases' with for example ti for title.
4) Perform a search 'ti:<title>' and check you get results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We convert 'keyword' to '' as an index since we want it to search all fields
When we are searching 'as phrase', however, we should not drop the search type
To test:
1 - Enable IntranetCatalogPullDown
2 - Set searchEngine to Elasticsearch
3 - Perform a search for 'Keyword as phrase' for a phrase that does appear in a record
4 - You get the result
5 - Reverse the order of words in the phrase
6 - You still get a result?
7 - Apply patch
8 - Restart all the things
9 - Reversed search does not return record
10 - Correct order and search, correct record returned
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch prepares Koha to officially no longer support Elasticsearch 5.X
It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature
It updates the about page to add a deprecation warning if a site is running ES5
To test:
1 - Be running Koha with Elasticsearch 5.X
2 - Attempt to search
Error: Unable to perform your search. Please try again.
3 - Apply patch
4 - Update database
5 - Searching works
6 - Find syspref 'ElasticsearchCrossFields'
7 - Enable it
8 - Searching is now broken
9 - Check the about page
10 - you can now see the Elasticsearch version
11 - The systeminformation tab has a deprectaion warning
12 - Set SearchEngine preference to 'Zebra'
13 - View the about page - no warnings
14 - Test again with ES6 - searching should "work" with either pref setting
15 - There should be no warning on about pages
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the 'cross_fields' type to our searches:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-string-query.html#query-string-syntax
Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6
To test:
0 - Set QueryAutoTruncate to 'only if * is added'
1 - Find a record with a title and publisher
2 - Search for a word form the title and confirm the record is returned
3 - Search for a work from the title and the publisher's name
4 - The record is not returned
5 - Apply patch
6 - Repeat #3
7 - The record is returned
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
When both a record and record_id are passed to index_records the data should be passed through
to update_index. We missed copying over the record ids to the variable we use as a check.
To test:
1 - Set searchEngine system preference to Elasticsearch
2 - Reindex your db
3 - Search authorities
4 - Edit a record and add 'testwaffle' to the main heading
5 - Search authorities for 'testwaffle' - no results
6 - Apply patch
7 - Edit the record again, change 'testwaffle' to 'testpancake'
8 - Search authorities for 'testpancake' - result!
9 - Confirm imported authorities and authorities added via Z39 are correctly indexed
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Without this patch we get
Use of uninitialized value $3 in concatenation (.) or string at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 943.
This converts the | OR operator to two different regexes so that the
capture group variables will be defined in every case.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space
To test:
1 - Have Koha using ES
2 - Search for:
ti:chess AND chess
3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
4 - Search for:
ti:chess AND kw:chess
5 - No result
6 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
7 - Repeate search and check page source
8 - search_query has:
title:chess ANDchess
9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
title:chess AND chess
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 24567: (follow-up) Use dollar sign to refer to captures
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Apply patch
2 - ./installer/data/mysql/updatedatabase.pl
3 - Reset ES mapping: Administration->Search engine configuration , button at bottom of page
4 - 'issues' and 'title' mapping under 'search fields' should be mandatory and not editable
5 - On 'Bibliographic records' tab you should not be able to delete the single entry for issues
6 - You should be able to delete 'title' mappings, however, at the final one you should be stopped by javascript
7 - Bonus: force remove the last mapping from the page using developer tools - attempt to save and should be warned of missing mandatory mapping
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
For debugging purposes we may wish to see the requests and responses made to
Elasticsearch
To test:
1 - prove -v t/Koha/SearchEngine/Elasticsearch.t
2 - Set <trace_to>Stderr</trace_to> in koha-conf
3 - Restart all
4 - perl misc/search_tools/rebuild_elasticsearch.pl
5 - Note requests are shown
6 - Set
<trace_to>File</trace_to>
<trace_to>/var/log/koha/kohadev/plack-error.log</trace_to>
in koha-conf
7 - Restart all
8 - perl misc/search_tools/rebuild_elasticsearch.pl
9 - Check the plack log and see the ES requests
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently if you combine subfields in the marc mappings the subfields are indexed in the order
listed in the mapping.
i.e. 650(avxyz) in mapping
and in record:
650 $aHeading $zGeosubdiv $vFormsubdiv
is indexed as:
Heading Formsubdiv Geosubdiv
We should preserve the order and index as:
Heading Geosubdiv Formsubdiv
We can use built in function in Marc::Field to achieve this
To test:
1 - It is easy to find examples of this using authorities
2 - Find or create a record with subfields order azv
e.g. 150$aActresses$zUnited states$vBiography
3 - Add or have a second authority
e.g. 150$aActresses$vPortraits
4 - Set an authorities mapping for 'Heading' to 150(abgvxyz)
find at:
Administration->Search engine configuration (Elasticsearch)->Authorities tab
5 - Index the records in Elasticsearch
perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692
6 - View the first record in the ES index
curl es:9200/koha_kohadev_authorities/data/1692?pretty
7 - Note 'Heading' field is ordered as in the mapping
8 - Search authorities for 'contains' "act"
9 - Note the records sort incorrectly
10 - Apply patches
11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692
12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty
13 - Note the order is now preserved
14 - Search authorities for 'contains' "act"
15 - Note the records sort correctly
Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
JD amended patch: Fix
FAIL spelling
combind ==> combined
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Just formally needed. It is already loaded somewhere.
That is: Koha::SearchEngine::Elasticsearch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
2 - For example 'author' / 100a
100 _ _ ‡a
3 - Index that record into Elasticsearch 5.X:
perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
4 - Note error 'value must have length > 0'
5 - Edit mappings to set author 100a not suggestible
6 - perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
7 - Success
8 - Set field to suggestible again
9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
11 - Success!
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Doing the same change as previously (renaming biblionumber), but fixing
at the same the record fetch. If (theoretically) an authority is passed
without a record, it would have fetched a biblio record.
Test plan:
You need Elasticsearch here.
Replaced this line in AddAuthority:
$indexer->index_records( $authid, "specialUpdate", "authorityserver", $record );
by
$indexer->index_records( $authid, "specialUpdate", "authorityserver", undef );
And updated an authority record. Check if you can search for the change.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
JD amended patch: remove trailing whitespace
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
ModZebra:
The name is very misleading: we can index authid's too here.
And yes, it should not be in C4/Biblio too ;) A first step..
Adding the same change here in Koha/SearchEngine/Zebra/Indexer.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This is analogous to 26522, we shoudl skip record that cannot be retrieved for indexing
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
When batch editing, 2 reindex calls are sent to ES/Zebra.
We can easily avoid that reusing the skip_modzebra_update (renamed skip_record_index)
Additionally we should only send one request for biblio, and we should
only do it if we succeed
As the whole batch mod is in a transaction it is possible to fail in which case
Zebra queue is reset, but ES indexes have already been set
In addition to the skip param this patchset moves Zebra and Elasticsearch calls to
Indexer modules and introduces a generic Koha::SearchEngine::Indexer so that we don't
need to check the engine when calling for index
The new index_records routine takes an array so that we can reduce the calls to
the ES server.
The index_records routine for Zebra loops over ModZebra to avoid affecting current behaviour
Test plan:
General tests, under both search engines:
1 - Add a biblio and confirm it is searchable
2 - Edit the biblio and confirm changes are searchable
3 - Add an item, confirm it is searchable
4 - Delete an item, confirm it is not searchable
5 - Delete a biblio, confirm it is not searchable
6 - Add an authority and confirm it is searchable
7 - Delete an authority and confirm it is not searchable
Batch mod tests, under both search engines
1 - Have a bib with several items, none marked 'not for loan'
2 - Do a staff search that returns this biblio
3 - Items show as available
4 - Click on title to go to details page
5 - Edit->Item in a batch
6 - Set the not for loan status for all items
7 - Repeat your search
8 - Items show as not for loan
9 - Test batch deleting items
a - Test with a list of items, not deleting bibs
b - Test with a list of items, deleting bibs if no items remain where all items are only item on a biblio:
SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1)
c - Test with a list of items, deleting bibs if no items remain where some items are the only item on a biblio:
SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1,2)
10 - Confirm records are update/deleted as appropriate
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Have some records with uncertain dates in the 008
19uu, 195u, etc.
2 - Index them in Elasticsearch
3 - Do a search that will return them
4 - Sort results by publication/copyright date
5 - Note odd results
6 - Apply patch
7 - Reindex
8 - Sorting should be improved
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a "year" search field type. Fields with this type will only
retain values that looks like years, so invalid values such as
whitespace or word characters will not be indexed.
This for instance improves the behaviour when sorting by
"date-of-publication". If all values are indexed, records with
junk data instead of valid years will appear first among the search
results, drowning out more relevant hits. If assigning this field
the "year" type these records will instead always appear last,
regarless of sort order.
To test:
1) Have at least two biblios, one with a valid year in 008 (pos 7-10)
and another with an invalid one ("uuuu" for example)
2) Perform a wildcard search (*) and sort results by publication date.
3) The record with invalid year of pulication in 008 should appear first
4) Apply patch and run database updates
5) Reindex ElasticSearch
6) Perform the same search as in 2)
7) The record with the invalid year should now appear last
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In get_elasticsearch_params we set the conf to static if undefined,
but we never defined it
To test:
1 - Apply unit test patch
2 - prove -v t/Koha/SearchEngine/Elasticsearch.t
3 - It fails
4 - Apply this patch
5 - It succeeds
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>