'error' has special meaning in exceptions so naming the fields:
type, details
Rather than only dealing with a single exception type, we generically
get the ES exception info and pass it up.
I could not recreate timeout still, however, I simply restarted the
ES docker during commit stage to cause NoNodes exceptions
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit cc8787bf87)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 74284f4ddb)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
This catches a timeout response from the ES server, logs this, and continues the indexing
To test:
1 - perl misc/search_tools/rebuild_elasticsearch.pl
2 - Make the ES server timeout (I don't have good instruction yet)
3 - Watch the job crash
4 - Apply patches
5 - perl misc/search_tools/rebuild_elasticsearch.pl
6 - Make the server timeout
7 - Note the job reports failed commit, and continues
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 26312: (follow-up) Reset buffers even if commit fails
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 26312: (follow-up) Fix whitespace and missing semicolon
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 72e6999db7)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit d78690a0ab)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
If we are searching on kw there is a leading colon at the beginning of
the generated query:
kw:foo becomes :foo
Note that it only happens when there is no other terms before.
Test plan:
0. Don't apply the patch
1. Search for kw:foo
2. Notice the error
Error: Unable to perform your search. Please try again.
and the logs say
Failed to parse query [(:foo*)]
3. Apply the patch
4. Repeat the search and notice that you know get:
"12 result(s) found for 'kw:foo'."
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 1ff9160a62)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 988e119829)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Without this patch we get
Use of uninitialized value $3 in concatenation (.) or string at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 943.
This converts the | OR operator to two different regexes so that the
capture group variables will be defined in every case.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit ea7e83b271)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space
To test:
1 - Have Koha using ES
2 - Search for:
ti:chess AND chess
3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
4 - Search for:
ti:chess AND kw:chess
5 - No result
6 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
7 - Repeate search and check page source
8 - search_query has:
title:chess ANDchess
9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
title:chess AND chess
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 24567: (follow-up) Use dollar sign to refer to captures
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 6ce56f1134)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Subdivision authorities are not used for linking, however, they are recognized by
C4::AuthoritiesMarc
While these records are not used for linking, they could provide reference and
should be allowed to exist in the catalog without breaking ES indexing
THis patch simply skips the step of parsing the authorities into the linking form
if the type contains '_SUBD'
To test:
1 - Import a subdivision authority record via Z39 or use the one attached to this bug
2 - perl misc/search_tools/rebuild_elasticsearch.pl -v -d
3 - Authority indexing dies:
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 202.
Use of uninitialized value $tag in regexp compilation at /usr/share/perl5/MARC/Record.pm line 206.
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 207.
Can't call method "tag" on an undefined value at /kohadevbox/koha/C4/Heading.pm line 71.
4 - Apply patches
5 - reindex
6 - Success!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit d67fc5e010)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 8512ccb446)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
In zebra we sort by callnumber using 8007 cn-sort
We should do the same in elasticsearch
To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit a52a0f80b8)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 726da17134)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
We convert 'keyword' to '' as an index since we want it to search all fields
When we are searching 'as phrase', however, we should not drop the search type
To test:
1 - Enable IntranetCatalogPullDown
2 - Set searchEngine to Elasticsearch
3 - Perform a search for 'Keyword as phrase' for a phrase that does appear in a record
4 - You get the result
5 - Reverse the order of words in the phrase
6 - You still get a result?
7 - Apply patch
8 - Restart all the things
9 - Reversed search does not return record
10 - Correct order and search, correct record returned
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 9cfce5f010)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit d9b2c50463)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
This patch prepares Koha to officially no longer support Elasticsearch 5.X
It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature
It updates the about page to add a deprecation warning if a site is running ES5
To test:
1 - Be running Koha with Elasticsearch 5.X
2 - Attempt to search
Error: Unable to perform your search. Please try again.
3 - Apply patch
4 - Update database
5 - Searching works
6 - Find syspref 'ElasticsearchCrossFields'
7 - Enable it
8 - Searching is now broken
9 - Check the about page
10 - you can now see the Elasticsearch version
11 - The systeminformation tab has a deprectaion warning
12 - Set SearchEngine preference to 'Zebra'
13 - View the about page - no warnings
14 - Test again with ES6 - searching should "work" with either pref setting
15 - There should be no warning on about pages
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
This patch adds the 'cross_fields' type to our searches:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-string-query.html#query-string-syntax
Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6
To test:
0 - Set QueryAutoTruncate to 'only if * is added'
1 - Find a record with a title and publisher
2 - Search for a word form the title and confirm the record is returned
3 - Search for a work from the title and the publisher's name
4 - The record is not returned
5 - Apply patch
6 - Repeat #3
7 - The record is returned
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 03f15d13c5)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
When both a record and record_id are passed to index_records the data should be passed through
to update_index. We missed copying over the record ids to the variable we use as a check.
To test:
1 - Set searchEngine system preference to Elasticsearch
2 - Reindex your db
3 - Search authorities
4 - Edit a record and add 'testwaffle' to the main heading
5 - Search authorities for 'testwaffle' - no results
6 - Apply patch
7 - Edit the record again, change 'testwaffle' to 'testpancake'
8 - Search authorities for 'testpancake' - result!
9 - Confirm imported authorities and authorities added via Z39 are correctly indexed
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 26903: Unit tests
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Currently if you combine subfields in the marc mappings the subfields are indexed in the order
listed in the mapping.
i.e. 650(avxyz) in mapping
and in record:
650 $aHeading $zGeosubdiv $vFormsubdiv
is indexed as:
Heading Formsubdiv Geosubdiv
We should preserve the order and index as:
Heading Geosubdiv Formsubdiv
We can use built in function in Marc::Field to achieve this
To test:
1 - It is easy to find examples of this using authorities
2 - Find or create a record with subfields order azv
e.g. 150$aActresses$zUnited states$vBiography
3 - Add or have a second authority
e.g. 150$aActresses$vPortraits
4 - Set an authorities mapping for 'Heading' to 150(abgvxyz)
find at:
Administration->Search engine configuration (Elasticsearch)->Authorities tab
5 - Index the records in Elasticsearch
perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692
6 - View the first record in the ES index
curl es:9200/koha_kohadev_authorities/data/1692?pretty
7 - Note 'Heading' field is ordered as in the mapping
8 - Search authorities for 'contains' "act"
9 - Note the records sort incorrectly
10 - Apply patches
11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692
12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty
13 - Note the order is now preserved
14 - Search authorities for 'contains' "act"
15 - Note the records sort correctly
Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
JD amended patch: Fix
FAIL spelling
combind ==> combined
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 890fd6ff06)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Just formally needed. It is already loaded somewhere.
That is: Koha::SearchEngine::Elasticsearch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 6deb177576)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To test:
1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
2 - For example 'author' / 100a
100 _ _ ‡a
3 - Index that record into Elasticsearch 5.X:
perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
4 - Note error 'value must have length > 0'
5 - Edit mappings to set author 100a not suggestible
6 - perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
7 - Success
8 - Set field to suggestible again
9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
11 - Success!
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit eb3d0982ae)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Add a "year" search field type. Fields with this type will only
retain values that looks like years, so invalid values such as
whitespace or word characters will not be indexed.
This for instance improves the behaviour when sorting by
"date-of-publication". If all values are indexed, records with
junk data instead of valid years will appear first among the search
results, drowning out more relevant hits. If assigning this field
the "year" type these records will instead always appear last,
regarless of sort order.
To test:
1) Have at least two biblios, one with a valid year in 008 (pos 7-10)
and another with an invalid one ("uuuu" for example)
2) Perform a wildcard search (*) and sort results by publication date.
3) The record with invalid year of pulication in 008 should appear first
4) Apply patch and run database updates
5) Reindex ElasticSearch
6) Perform the same search as in 2)
7) The record with the invalid year should now appear last
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: Add database update script
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: Update tests
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: Add suppport for uncertain fields and ranges
To test:
1 - Have some records with uncertain dates in the 008
19uu, 195u, etc.
2 - Index them in Elasticsearch
3 - Do a search that will return them
4 - Sort results by publication/copyright date
5 - Note odd results
6 - Apply patch
7 - Reindex
8 - Sorting should be improved
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: Refactor using tokenize_callbacks
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: Simplify with new and imporved value_callbacks
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: (follow-up) Fix spelling
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: (follow-up) Add support for spaces as unknown characters
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 24807: (QA follow-up) Remove uneccessary tests
These tests fail now, the code expects a real response from ES in Indexer.pm
but these tests mock 'bulk' and so don't have the necessary fields.
We are testing the same code above and can just add the _id == biblionumber test
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
When batch editing, 2 reindex calls are sent to ES/Zebra.
We can easily avoid that reusing the skip_modzebra_update (renamed skip_record_index)
Additionally we should only send one request for biblio, and we should
only do it if we succeed
As the whole batch mod is in a transaction it is possible to fail in which case
Zebra queue is reset, but ES indexes have already been set
In addition to the skip param this patchset moves Zebra and Elasticsearch calls to
Indexer modules and introduces a generic Koha::SearchEngine::Indexer so that we don't
need to check the engine when calling for index
The new index_records routine takes an array so that we can reduce the calls to
the ES server.
The index_records routine for Zebra loops over ModZebra to avoid affecting current behaviour
Test plan:
General tests, under both search engines:
1 - Add a biblio and confirm it is searchable
2 - Edit the biblio and confirm changes are searchable
3 - Add an item, confirm it is searchable
4 - Delete an item, confirm it is not searchable
5 - Delete a biblio, confirm it is not searchable
6 - Add an authority and confirm it is searchable
7 - Delete an authority and confirm it is not searchable
Batch mod tests, under both search engines
1 - Have a bib with several items, none marked 'not for loan'
2 - Do a staff search that returns this biblio
3 - Items show as available
4 - Click on title to go to details page
5 - Edit->Item in a batch
6 - Set the not for loan status for all items
7 - Repeat your search
8 - Items show as not for loan
9 - Test batch deleting items
a - Test with a list of items, not deleting bibs
b - Test with a list of items, deleting bibs if no items remain where all items are only item on a biblio:
SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1)
c - Test with a list of items, deleting bibs if no items remain where some items are the only item on a biblio:
SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1,2)
10 - Confirm records are update/deleted as appropriate
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 25265: Rename skip_modzebra_update to skip_record_index
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 25265: Fix copy paste error for parameter
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 25265: (follow-up) Don't index malformed records
This is analogous to 26522, we shoudl skip record that cannot be retrieved for indexing
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 25265: (QA follow-up) Add shebang to Indexer.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 25265: (QA follow-up) Rename biblionumber in ModZebra, index_records
ModZebra:
The name is very misleading: we can index authid's too here.
And yes, it should not be in C4/Biblio too ;) A first step..
Adding the same change here in Koha/SearchEngine/Zebra/Indexer.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 25265: (QA follow-up) Check server type in Elasticsearch::index_records
Doing the same change as previously (renaming biblionumber), but fixing
at the same the record fetch. If (theoretically) an authority is passed
without a record, it would have fetched a biblio record.
Test plan:
You need Elasticsearch here.
Replaced this line in AddAuthority:
$indexer->index_records( $authid, "specialUpdate", "authorityserver", $record );
by
$indexer->index_records( $authid, "specialUpdate", "authorityserver", undef );
And updated an authority record. Check if you can search for the change.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We are guessing authtype code and inserting the heading built accorindg to C4::Heading
If we can't identify the auth type, we can format the heading. There is a record in the koha test
data that is missing the heading field so type cannot be idenfitied
This prevents us from dying n a record where we cannot identify the type.
Note: This code will also be triggered for custom authority types, higlighting
that they won't link because of hardcoded mappings in C4::Heading. We must tackle this
on a new bug
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 5a87514b2e)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit ce161fda9b)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
In get_elasticsearch_params we set the conf to static if undefined,
but we never defined it
To test:
1 - Apply unit test patch
2 - prove -v t/Koha/SearchEngine/Elasticsearch.t
3 - It fails
4 - Apply this patch
5 - It succeeds
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit b87dc492fe)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The code in build_query_compat contained a 'my' in the assigning of limits for the search in a conditional
This meant the limits were being set correctly during the conditional, but we blanked when passed to the rest
of the code. The effect was that the searches worked, however, the template params to repeat the search were
incomplete.
Removing the my ensures the same limits are applied during search and on re-sorting
To test:
1 - Enable Elasticsearch
2 - On OPAC perform advanced search, selecting only an itype, ccode, or LOC limit
3 - Attempt to sort results
4 - You are returned to Advanced search
5 - Apply patch
6 - Repeat
7 - It sorts!
Signed-off-by: Lisette Scheer <lisetteslatah@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 43dc505a52)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We assume all limits from advanced search to be a phrase and quote them
when doing this we should remove the phrase marker to avoid doulbe quoting
To test:
1 - Have koha using ES
2 - Go to advanced search
3 - Limit by a single itemtype that exists
4 - Get some results
5 - Limit by a different itemtype that exists
6 - Get some results
7 - Limit by both itemtypes
8 - Get only the results for the second itemtype
9 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
10 - Repeat search
11 - View page source and find 'search_query'
12 - See limit looks like itype:("("BK")" OR "("CR")")
13 - Apply patches
14 - Restart all the things
15 - Repeat search for both itemtypes
16 - Note results now include both types
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit a2b2426de8)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To test:
0 - Set DisplayLibraryFacets to 'both' and FacetMaxCount to 20
1 - Have more than 10 branches
2 - Have items in each of those branches
3 - Enable ES, set system preference SearchEngine to Elasticsearch
4 - Search for '*'
5 - Expand homebranch/holdingbranch facets
6 - Note you only get 10
7 - Apply patch
8 - Repeat search
9 - Now you get all your facets
Signed-off-by: Michael Springer <mspringer@mylakelibrary.org>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 4dd1672c1b)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To recreate:
1 - Have Koha using ES5 and Elasticsearch as search engine
2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
3 - Do a search in authorities using 'Search entire record' (abduction if using sample db)
4 - Note no results
5 - View the page source and find 'search_query'
6 - Note the uppercased fields
7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
8 - Note all fields lower-cased
9 - Apply patch
10 - Repeat search
11 - It works!
Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 2ab6018b6e)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Remove unused Elasticsearch parameters 'key_prefix' and 'request_timeout'.
Refactor so that get_elasticsearch_params only returns parameters used
by Search::Elasticsearch, add class accessor for index_name.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Replace remaining Catmandu dependant code with the Search::Elasticsearch
equivalent.
To test:
1) Apply patch
2) Run tests in t/Koha/SearchEngine/Elasticsearch.t, t/Koha/SearchEngine/ElasticSearch/*
and t/db_dependent/Koha/SearchEngine/*
3) All tests should pass
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Be using ES
2 - Find a unique record by searchign for title or seomthing
3 - Edit the record
4 - Repeat search and confirm you are returned to the details page
5 - Go to Tools->MARC modification templates
6 - Add a template and define an action
7 - Go to Tools->Batch record modification
8 - Select Enter a list of record numbers
9 - Enter the biblionumber of the record you searched for
10 - Use the marc modification template you setup
11 - Complete the modification
12 - Repeat your search
13 - note you get two results, both pointing to same biblionumber
14 - Check the ES index, note that you have two entries, one with the biblionumber, and the other with #.0, e.g.:
curl -XGET 'es:9200/koha_kohadev_biblios/data/14.0?pretty'
curl -XGET 'es:9200/koha_kohadev_biblios/data/14?pretty'
15 - Apply patch
16 - Delete and rebuild ES index
perl misc/search_tools/rebuild_elasticsearch.pl -d -b
17 - Repeat 2-12
18 - No duplication this time
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t
Or:
Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
2 - Reindex elasticsearch
3 - Check the ES count on the about page
4 - Check the count in the DB (SELECT count(*) FROM biblio)
5 - They don't match!
6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
7 - No errors indicated
8 - Apply patch
9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
There was an assumption in the ES code that match-heading mappings will appear in
a specified portion of the mappings array.
Certain mappings setups will not meet this assumption.
We need to move our searching up one level
The key seems to be having a mapping for a complete field, say 150, in both the
match-heading and another field as well as having mappings for ungrouped fields like
150a 150ab etc.
The unit test coverage should be sufficient for testing
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan :
Launching script several times and see that marc field is alway ordered increasing
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Strip initial characters from search fields in accordance with
nonfiling character indicators.
To test:
1) Apply patch
2) Run tests in t/Koha/SearchEngine/Elasticsearch.t
3) All tests should pass
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In the advanced search form, you can enable several limits using syspref
AdvancedSearchTypes (namely itemtypes, shelving locations, collections)
When used, the resulting query parts end up being joined with OR, even
if the field is different. That means that if you pick "Book" under
itemtypes tab, and "Fiction" under collection tab, it will search
"itype:BOOK OR ccode:FIC". It should be AND.
For instance, if you select:
Itemtypes:
✓ Book
✓ DVD
Location:
✓ Child
✓ Adult
it should search:
itype:(Book OR DVD) AND location:(Child OR Adult)
Test plan:
0. Do not apply the patch yet
1. Enable elasticsearch
2. Set syspref AdvancedSearchTypes = 'itemtypes|loc|ccode'
3. Create a new itemtype and a new authorised value for categories LOC
and CCODE
4. Create a biblio with the new itemtype, another biblio with the new
location, another biblio with the new collection, and again another
biblio with the new itemtype, location and collection
5. Verify that you can find these new biblio records using only the
"advanced search types" in the advanced search form
6. In the advanced search form, pick all 3 limits (itemtype, location,
collection) and verify that it returns the 4 records.
7. Apply the patch
8. Repeat step 6, it should now return only the biblio that satisfies
all criteria
9. Verify that if you select more than one
{itemtype|location|collection} it still returns results that
satisfies any selected criteria
10. prove t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Not only should terms from facets/limits be grouped, but order should probably matter
We build a different facet for "Dillinger Girl" or "Girl Dillinger"
The parens are possibly overkill now, but I think it makes it very clear and does not hurt
To test:
1 - Follow original plan
2 - Create a new record with 'Girl Dillinger' as author
3 - Search for 'Dill*'
4 - You get all three records (and maybe others that match)
5 - Limit by 'Girl Dillinger' - you get two records
6 - Same for 'Dillinger Girl'
7 - Apply patch
8 - Limits/facets for 'Dillinger Girl' and 'Girl Dillinger' now match a isngle record
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
- Use Elasticsearch 6 (you'll need Bug 18969),
- create a biblio (#1) with "Dillinger Girl" in author and what you
want in title,
- create another biblio (#2) with the word "girl" in the title and
"Dillinger Escaplan" as author
- reindex
- search * and refine on "Dillinger Girl"
- Ko => Biblio #1 and #2 appear
- Apply this patch,
- search * and refine on "Dillinger Girl"
- Ok => anly biblio #1 appears
- use Elasticsearch 5 again
- check for no search regression
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
There is no need to specify the `client` param here because
'5_0::Direct' is the default value for Search::Elasticsearch 5.x and it
breaks when Search::Elasticsearch 6.x is installed
Test plan:
1. Apply the patch
2. Make sure you have Search::Elasticsearch 5.x
3. Reload starman
4. Verify that search works
5. Install Search::Elasticsearch 6.x
6. Reload starman
7. Verify that search works
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 9978 should have fixed them all, but some were missing.
We want all the license statements part of Koha to be identical, and
using the GPLv3 statement.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This is an interface for quick and efficient browsing through records.
It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.
To Test:
1 - Apply patches
2 - Update database (updatedatabase on kohadevbox)
3 - Compile the CSS
https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client
yarn build --view=opac on kohadevbox
4 - Enable the new syspref OpacBrowseSearch
5 - Have ES running and some records in it
SearchEngine syspref set to Elasticsearch
6 - Browse to opac home, click 'Browse search' link
for your site)
7 - Test searching for author, title, and subject
8 - Verify that results are returned in expected order
9 - Experiment with fuzziness
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html#fuzziness
Options are: exact (0 edits), fuzzy (1 edit), very fuzzy (2 edits)
10 - Click any result and verify specific titles are correct
11 - Click through title to record and verify it is the correct record
12 - Test that disabling pref removes the link on the opac home
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Nick Clemens
Jonathan Druart
Joonas Kylmälä
Marcel de Rooy
David Gustafsson
Frédéric Demians
Fridolin Somers
Alex Arnaud
Julian Maurice
Robin Sheat