Strip initial characters from search fields in accordance with
nonfiling character indicators.
To test:
1) Apply patch
2) Run tests in t/Koha/SearchEngine/Elasticsearch.t
3) All tests should pass
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In the advanced search form, you can enable several limits using syspref
AdvancedSearchTypes (namely itemtypes, shelving locations, collections)
When used, the resulting query parts end up being joined with OR, even
if the field is different. That means that if you pick "Book" under
itemtypes tab, and "Fiction" under collection tab, it will search
"itype:BOOK OR ccode:FIC". It should be AND.
For instance, if you select:
Itemtypes:
✓ Book
✓ DVD
Location:
✓ Child
✓ Adult
it should search:
itype:(Book OR DVD) AND location:(Child OR Adult)
Test plan:
0. Do not apply the patch yet
1. Enable elasticsearch
2. Set syspref AdvancedSearchTypes = 'itemtypes|loc|ccode'
3. Create a new itemtype and a new authorised value for categories LOC
and CCODE
4. Create a biblio with the new itemtype, another biblio with the new
location, another biblio with the new collection, and again another
biblio with the new itemtype, location and collection
5. Verify that you can find these new biblio records using only the
"advanced search types" in the advanced search form
6. In the advanced search form, pick all 3 limits (itemtype, location,
collection) and verify that it returns the 4 records.
7. Apply the patch
8. Repeat step 6, it should now return only the biblio that satisfies
all criteria
9. Verify that if you select more than one
{itemtype|location|collection} it still returns results that
satisfies any selected criteria
10. prove t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Not only should terms from facets/limits be grouped, but order should probably matter
We build a different facet for "Dillinger Girl" or "Girl Dillinger"
The parens are possibly overkill now, but I think it makes it very clear and does not hurt
To test:
1 - Follow original plan
2 - Create a new record with 'Girl Dillinger' as author
3 - Search for 'Dill*'
4 - You get all three records (and maybe others that match)
5 - Limit by 'Girl Dillinger' - you get two records
6 - Same for 'Dillinger Girl'
7 - Apply patch
8 - Limits/facets for 'Dillinger Girl' and 'Girl Dillinger' now match a isngle record
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
- Use Elasticsearch 6 (you'll need Bug 18969),
- create a biblio (#1) with "Dillinger Girl" in author and what you
want in title,
- create another biblio (#2) with the word "girl" in the title and
"Dillinger Escaplan" as author
- reindex
- search * and refine on "Dillinger Girl"
- Ko => Biblio #1 and #2 appear
- Apply this patch,
- search * and refine on "Dillinger Girl"
- Ok => anly biblio #1 appears
- use Elasticsearch 5 again
- check for no search regression
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
There is no need to specify the `client` param here because
'5_0::Direct' is the default value for Search::Elasticsearch 5.x and it
breaks when Search::Elasticsearch 6.x is installed
Test plan:
1. Apply the patch
2. Make sure you have Search::Elasticsearch 5.x
3. Reload starman
4. Verify that search works
5. Install Search::Elasticsearch 6.x
6. Reload starman
7. Verify that search works
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 9978 should have fixed them all, but some were missing.
We want all the license statements part of Koha to be identical, and
using the GPLv3 statement.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This is an interface for quick and efficient browsing through records.
It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.
To Test:
1 - Apply patches
2 - Update database (updatedatabase on kohadevbox)
3 - Compile the CSS
https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client
yarn build --view=opac on kohadevbox
4 - Enable the new syspref OpacBrowseSearch
5 - Have ES running and some records in it
SearchEngine syspref set to Elasticsearch
6 - Browse to opac home, click 'Browse search' link
for your site)
7 - Test searching for author, title, and subject
8 - Verify that results are returned in expected order
9 - Experiment with fuzziness
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html#fuzziness
Options are: exact (0 edits), fuzzy (1 edit), very fuzzy (2 edits)
10 - Click any result and verify specific titles are correct
11 - Click through title to record and verify it is the correct record
12 - Test that disabling pref removes the link on the opac home
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Be using Elasticsearch
2 - Check that field 150 is mapped to 'Match-heading' or add it (the subfields don't matter)
3 - Add a topic term authority record like "150 $aCats$vFiction"
4 - Add a 650 with $aCats and $vFiction and $e depicted to a bibliographic record
5 - Run the linker for the bib
perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v
6 - Confirm the record is not correctly linked to the record
7 - Apply patch
8 - Reindex authorities for ES
perl misc/search_tools/rebuild_elasticsearch.pl -v -d -a
9 - Run the linker and confirm record is correctly linked
perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
A search made on Shelving location tab in advanced search gives 0 result.
Test plan :
Without patch:
- in advanced search, choose a location in shelving location tab and start search (opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&do=Search&limit=mc-loc)
- 0 result
- check that location exist by doing a simple search and filtering on the location facet
- have some results
With patch:
- in advanced search, choose a location in shelving location tab and start search (opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&do=Search&limit=mc-loc)
- found results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Enable ES
2 - Enable OpacSuppression
3 - Suppress a bib in staff client
Edit 942n to be 1
4 - Search the opac for anything
5 - Tail the plack logs and note a deprecation warning
6 - Apply patch
7 - Restart all the things
8 - Repeat search on opac
9 - No error/warn in logs
10 - Record is correctly suppressed
Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
When executing
Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings()
against populated search engine mappings tables, one gets this
exception:
DBD::mysql::st execute failed: Duplicate entry '388-73' for key
'PRIMARY' [for Statement \"INSERT INTO `search_marc_to_field` ( `facet`,
`search_field_id`, `search_marc_map_id`, `sort`, `suggestible`) VALUES (
?, ?, ?, ?, ? )\" with ParamValues: 0='', 1='73', 2='388', 3=undef,
4=''] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line
1834.\nDBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry
'388-73' for key 'PRIMARY' at /home/koha/Koha/Koha/SearchField.pm line
38"
This patch fixes this, by first deleting all entries, then adding the
default ones.
If a subroutine says reset_elasticsearch_mappings(), I expect it to
reset the elasticsearch mappings without errors. When writing tests it
is better to call that one subroutine reset_elasticsearch_mappings
instead of the bunch of code in
admin/searchengine/elasticsearch/mappings.pl?op=reset_confirmed
Also this promotes code reuse.
TEST PLAN:
a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings'
b. Run misc/devel/populate_db.pl on an already populated DB and you eventually
face this problem.
After this patch, step a. works.
This patch is adapted from the original patch made by Olli-Antti
Kivilahti <olli-antti.kivilahti@jns.fi>.
Signed-off-by: Johanna Raisa <johanna.raisa@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - In staff or OPAC using ES, search for "biblionumber:3" or any existing biblionumber
2 - No results
3 - Apply patch, restart all the things
4 - Search again
5 - You go to the biblionumber
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
The deletion subroutine expects to get a string, this is fine in most places, but when
we sort our records for batch deletion they are cast to numbers and the delete is not processed.
This patch quotes the biblionumbers as strings before passing them to Catmandu
To aid in detecting orphaned records you can use the patches on bug 22831
To test:
1 - Have or create a bib
2 - Go to Tools->Batch record deletion
3 - Click 'Enter a list of record numbers'
4 - Enter the biblionumber'
5 - Confirm deletion
6 - Search for the record, it still appears
7 - Apply patch, restart all the things
8 - Repeat 1-6
9 - This time the record is deleted
10 - Test other methods of record deletion to confirm the record is removed
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Note: I also remove warnings for undefined operation in this patch, is a trivial fix
To test:
prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Export your authorities via Tools->Export data
2 - Define a record matching rule in Admin->Record matchign rules
Use index: LC-card-number
field: 010$a
3 - Stage the exported records for import and use the rule created above for matching
4 - The process does not complete
5 - Check intranet error logs - exception on unknown marclist
6 - Apply patch
7 - Repeat
8 - Success!
9 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Adds support for using the "scan indexes" action in advanced search by using faceting with a prefix filter. Requires that the field be set as facetable for anything to be found.
Test plan:
1. Apply patch
2. Go to advanced search and click "More options"
3. Select author as the search field, enter a last name and check "Scan indexes"
4. Perform search and observe the result list resembling scan results
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.
Additionally, sort fields based on 'sum' should also use sum in building the value to sort on
To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Elasticsearch indexing uses 999$c to store record id by deleting the all field first !
So you can not store anything in field 999, even in UNIMARC and even in authorities records.
Looks like it is quick fix code added to start Elasticsearch use.
This behavior is disturbing and very strange for UNIMARC flavour.
This patch corrects by defining record ids mandatory in Koha::SearchEngine::Elasticsearch::Indexer::update_index().
This ids array is actually always given (except in UT).
I think it is useless to allow adding a record without its id.
Test plan :
1) Use Elasticsearch as SearchEngine
2) Create a subfield 999$z in default framework
3) Create a record with default framework
4) Enter a random string (never used in catalog) like "tototata" in 999$z
5) In Search engine configuration, define search field "subject" for 999$z
6) Rebuild record : misc/search_tools/rebuild_elasticsearch.pl -b -bn <biblionumber> -v
7) Search for the random string => You get a result
8) Optionnaly look at records in ES : <es server>:9200/<es index name>/data/<biblionumber>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan
- Set up a elasticsearch 6 instance to work with Koha,
- you may need to make koha works with ES 6 (see bug 20589),
- make a search and limit it to available items only,
=> no result
- Apply this patch,
- make a search and limit it to available items only,
- you should get some results
- Do the same with ES 5.x
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
Search main heading ($a only)
Search main heading
Default operator beeing 'contains'.
Actually with Elasticsearch those search give no results.
Example with heading :
200
$a Casaubon
$b Isaac
$f 1559-1614
Call to Elasticsearch :
"query" : {
"bool" : {
"must" : [
{
"query_string" : {
"query" : "Casaubon*",
"default_field" : "heading-main",
}
},
{
"query_string" : {
"query" : "(Isaac*) AND (1559-1614*)",
"default_field" : "heading"
}
}
]
}
},
"sort" : [
{
"heading__sort.phrase" : "asc"
}
]
}
Patch adds to "query_string" :
analyze_wildcard : true.
Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Add "QueryRegexEscapeOption" system preference to provide option to escape
Elasticsearch regexp delimiters (/) within queries, or alternativly to
unescape escaped slashes (\/) while escaping unescaped slashes, in
effect making "\/" the new regexp delimiter.
How to test:
1) Run tests in ./t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
2) All tests should succeed
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Maksim Sen <maksim.sen@inlibro.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Prevent Elasticsearch from throwing an exception when the mappings
include uppercase subfield identifier.
Though forbidden in MARC21, It doesn't seem to be the case for Unimarc
or ISO 2709 as a whole. It is also possible to define such a subfield
identifier in the Koha MARC Framework and in the Elasticsearch
mappings.
Test plan:
1. add a 995Z subfield identifier in Elasticsearch mappings
(Administration -> Search engine configuration)
2. run $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
3. see it throw: “Invalid MARC field expression: 995Z”
4. apply patch
5. rerun $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
6. sababa
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Prevent Koha::SearchEngine::Elasticsearch from dying when trying to run
Koha::Exceptions::Elasticsearch::MARCFieldExprParseError->throw()
Plan test:
1. Go to Administration -> Searchengine configuration, and add a mapping
for 995Z, whatever the search field (I did it with local-classification),
save
2. run $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
3. see it die asking for Koha::Exceptions::Elasticsearch
4. apply patch
5. rerun $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
6. see it throw: Invalid MARC field expression: 995Z
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.
How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.
Sponsorded-by: Gothenburg Univesity Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In cataloguing, the use of tag editor opens authorities finder with a limit on specific authorities type.
This limit is missing with Elasticsearch.
This patch adds in query a "filter" on "term" which is the most performant way because there must be no ranking computed on this part.
Test plan :
1) Use Elasticsearch
2) Create an autority of type author (NP in UNIMARC) with heading "Tolkien"
3) Create an autority of type subject-author (SAUT in UNIMARC) with heading "Tolkien"
4) Create a biblio record
5) Use tag editor on a author field (700 in UNIMARC)
6) Seach for "Tolkien" in $a
without patch : you see the 2 authorities in results
with patch : you see only the correct authority in results (NP in UNIMARC)
7) Check search in authorities-home.pl is still OK
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching.
Test plan:
1. Test that searching and indexing works with the patch without any changes.
2. Switch to array format and index some records.
3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes.
4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries
5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping.
Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing.
Test plan:
1. Reset mappings and reindex biblios.
2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass.
3. Try that all of the following year range type work in publication date search and year limit in advanced search:
yyyy
yyyy-yyyy
-yyyy
yyyy-
Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
- apply this patch,
- edit admin/searchengine/elasticsearch/mappings.yaml
to add weight for some fields,
- go to admin > search engine configuration,
- reset your mappings (add ?op=reset&i_know_what_i_am_doing=1 to url),
- check that weights you've added are set
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch removes the constraint of only passing 5 facets to the template unless the list is expanded, in fact, it removes the 'expanded' attribute from Search.pm
Now that all facets are passed to page it adds a 'show more' link at the bottom of lists and allows user to expand or collapse any facet set without reloading page.
Updated tests included.
To test:
1 - Perform an OPAC search that returns more than 5 of any given facet type
2 - Click the "Show more" link on the facets and see that the search is reloaded
3 - Apply patch
4 - Repeat search
5 - Note that you can click "Show more" without reloading page
6 - Test that page load is not greatly affected
7 - Ensure that all facet links function normally
8 - Ensure that facets are the same a prior to patch
9 - Repeat for staff client
10 - Prove t/Search.t
NOTE: This patch makes it much easier to see that there is an existing issue with marking the "active" facet. Ending punctuation seems to confuse the matcher.
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
You get no results when searching with an hyphen + with * in query string (or with preference QueryAutoTruncate) :
ie /cgi-bin/koha/opac-search.pl?q=saints-anges*
Looks like query-string by default does not compute wildcards, see analyze_wildcard in :
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
Test plan :
1) Use Elasticsearch
2) Create a record with "saints-anges"
3) Search for "saints-anges" => you get results
4) Search for "saints-anges*" => you get results
5) Search for "saints-ang*" => you get results
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Arthur Bousquet <arthur.bousquet@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
'Static' is the default value of Search::Elasticsearch and for a good
reason : it works in most cases, unlike the 'Sniff' option
To test:
1 - Apply patch
2 - Edit koha-conf.xml
3 - Add '<cxn_pool>Static</cxn_pool>' to the elasticsearch stanza
4 - Restart all the things!
5 - Reindex ES, it works
6 - Set SearchEngine to ES, try searching
7 - It works!
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plan:
After each of the following steps, verify that the results and query description looks ok.
1. Do a simple keyword query
2. Change sort order
3. Click a facet
4. Do an advanced search with title and author
5. Change sort order
6. Click a facet
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marjorie <marjorie.barry-vila@collecto.ca>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.
Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass: prove -v t/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plain:
1. Reindex both authorities and biblios with the -d flag to recreate the index, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v -d"
2. Check that authorities only contains authority-specific fields in Elasticsearch e.g. by fetching http://localhost:9200/koha_dev_authorities/_mappings
3. Reindex both without the -d flag and verify it works, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v"
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plan:
1. Do an advanced search for
Title = new
AND
Title = york
2. Verify that the results match an advanced search for:
Title = new york
3. Verify that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch still pass
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code
To test:
1 - Make sure you have records with differing languages (in the MARC21 008
field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Without this we could not recreate mappings as there are no defaults for
the additional fields - this restores behavoiur from before patches
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch adds a new section 'Facet order' in the Biblio tab of the
'Search engine configuration' admin page of the Elastic mappings.
The idea is to let the librarians define the facet to display and order
them as their needs.
The ergonomic is not perfect and I am open to any suggestions.
Test plan:
Move up and down the field list to order the facets
Hide/show some facets
Rebuild index
At the OPAC and the staff interface you should see the changes on the
search result page.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Default settings are appended in get_elasticsearch_params so that there's a central place for them. Defaults can be overridden in koha-conf.xml.
Test plan:
1. Verify that searches still work with the patch applied.
2. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>Static</cxn_pool>
3. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>FOO</cxn_pool>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.
Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Escapes unquoted colons that have whitespace on either side. Removed unbalanced quotes.
Test plan:
1. Make sure the test case described in the bug works
2. Make sure tests pass:
prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Björn Nylén <bjorn.nylen(at)ub.lu.se>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Keep authority-number as alias and change field name
from alias to real field in hard coded Elasticsearch query
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch lower cases the sort by fields to normalize checking them and adjusts
some existing tests to meet the new expectations.
The regex for splitting terms has been moved into a subroutine so that adjustment was made
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.
Sponsored-by: Gothenburg University Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
- Makes token splitting work like in biblio search
- Only adds right hand truncation since that's what Zebra also does
- Only adds truncation if the token is not a phrase and ends in a word character
- Adds tests to existing and new QueryBuilder functions
Test plan:
1. Create an authority record for "Duck, Donald"
2. Try contains type authority searches with the following terms and make sure they find the record:
Duck, Donald
donald duck
don duck
"Duck, Donald"
3. Make sure the following search works but does not find anything:
"Duck, Don"
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Moved db_dependent tests to the proper directory in the hierarchy, fixed tests to check correct results now that the authority query builder works better and added several new tests to cover changed functionality.
Sponsored-by: National Library of Finland
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Also optimize it so it's actually usable.
Test plan:
1. To test it properly you need biblio and authority data. You might get away with enabling AutoCreateAuthorities and BiblioAddsAuthorities so that authorities are created in the process. Another option would be to import authorities first e.g. from LoC.
2. Make sure the authority index has been properly created with "misc/search_tools/rebuild_elastic_search.pl -a -d"
3. Run "misc/link_bibs_to_authorities.pl -v -l" twice and observe the results.
Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
https://bugs.koha-community.org/show_bug.cgi?id=20244
Test plan:
1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
https://bugs.koha-community.org/show_bug.cgi?id=20244
Test plan:
1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Add missing pods, remove obsolete syspref and add test for serialization format for records exceeding max record size
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Add persistent per index "index status" state to provide useful
user feedback when update of Elasticsearch server mappings fails
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Default to base64 encoded binary MARC with MARCXML
fallback if record exceeds maximum size
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Implement optimized indexing for Elasticsearch
How to test:
1) Time a full elasticsearch re-index without this patch by running the
rebuild_elastic_search.pl with the -d flag:
`koha-shell <instance_name> -c "time rebuild_elastic_search.pl -d"`.
2) Apply this patch.
3) Time a full re-index again, it should be about twice at fast (for a
couple of thousand biblios, with fewer biblios results may be more
unpredictable).
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
It is true by default on ES 5 and has been removed on ES 6
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
The idea is the following: if some search field(s) are weighted in
search engine config page, Koha will query ES on all fields plus those with
the coresponding weights. Else, search is done on the entire record with
no weighting. The advanced search page is unaffected by these changes
Test plan (having Koha working with Elasticsearch):
- apply this patch
- have some weights defined for various fields
- try searches from the search bar and from the advanced search page
- confirm weighting affects the relevancy (in expected ways)
e.g.
1. search for 'a' from advanced search, note results
2. give 'title' a weight
3. search for 'a' using the simple search bar
4. results with 'a' in the title should now be more relevant
- confirm search results on advanced search page are unaffected
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.
To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plan:
- Try a simple search (OPAC or staff) like "title:foo",
- click on a facet: badaboum!
- apply this patch,
- retry
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
- Context: Koha working with elasticsearch,
- Apply BZ 19582 (that make auth_finder.pl works again),
- apply this patch,
- go to cgi-bin/koha/authorities/auth_finder.pl,
- type at least 3 chararcters in a search field and
check that the autocomplete returns results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
To test:
1 - Do some authority searches in Zebra
2 - Switch to ES and repeat, results will vary and some may fail
3 - Apply patch and dependencies
4 - Reindex ES
5 - Repeat searches, they should suceed and results should be similar to
Zebra
6 - Slight differences are okay, but results should (mostly) meet
expectations
A few notes:
We add a 'normalizer' to ensure we get a single token from the heading
indexes, this makes 'starts with' work as expcted
We switch to 'AND' for fields searched from cataloging editor - this
matches Zebra results
We force the '__sort' fields for sorting - if sorting looks wrong try
reducing the heading field to a single subfield - this will need to be
addressed on a future bug (multiple subfields create an array, ES sorts
those randomly)
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This avoid hardcoding '10000' in two different places and allow users to
adjust this setting.
Also, this patch fixes a bug when the search return less than 10000
results
Test plan:
1. Do a search that returns 10000+ records.
2. Note the warning above the pagination buttons
3. Go to the last page, no error
4. Change the ES setting:
curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \
'{"index": {"max_result_window": 20000}}'
5. Do another search that returns more than 10000 but less than 20000
6. Note that the warning does not show up
7. Go to the last page, still no error
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
"return" statement with explicit "undef" at line 245, column 5. See
page 199 of PBP. (Severity: 5)
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 20073: Move Elasticsearch yaml files back to admin directory
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This reverts commit f489d2034b.
This commit breaks the install process when using debian packages.
Reverting as we are very close to the 18.05.00 release
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
"return" statement with explicit "undef" at line 245, column 5. See
page 199 of PBP. (Severity: 5)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Stolen from bug 20244
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This option in combination with append seems to add null entries in the indexes
for fields that don't exist. These fields then affect sorting. Removing it solves
the issue
To test:
1 - Have an ES setup
2 - Catalog and index some records
3 - View a record in ES:
curl 'localhost:9200/koha_kohadev_biblios/data/4?pretty=yes'
4 - Note there are 'null' entries in the indexes, save this result for
comparison
5 - Apply patch
6 - Reindex, amking sure to delete existing indexes
perl misc/search_tools/rebuild_elastic_search.pl -d
7 - View the record again
8 - Note the null entries are removed
9 - Note indices are otherwise intact
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
TO test:
0 - Have a koha with ES enabled and some auth records
1 - backup your db
2 - sudo koha-mysql <<instance>>
3 - UPDATE auth_types SET authtypecode="PERSO_NAMB" WHERE
authtypecode="PERSO_NAME" (or adjust as necessary)
4 - Search for some authorities where a PERSO_NAME record will be
returned
5 - KA_BOOM!
6 - Apply patch
7 - Results are returned, they do not display a code, viewing record you
won't see code
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We likely shoudln't pass through an uncoverted sort order for now, but
it does allow us to look ahead to implementing the orders directly so
seems a good option to have.
Either this patch should be used, or the commented out tests should be
removed
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently sort order is extracted from sort condition by splitting the
field, instead use regular expression to extract the last part preceded
by underscore.
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Enable Zebra
2 - Perform an auth search
3 - note results
4 - Enable ES
5 - Repeat search, note (likely) diff results
6 - Open a record in cataloging and use the button to launch auth search
7 - Perform same search as above, note results match for eiher engine
selected
8 - NOTE: Disbale sorting for ES search - this will be dealt with in
another report
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
0 - Apply Unit test patch only
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - Should fail
3 - Apply this patch
4 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
5 - should pass
6 - search for 'Local-number.raw:"4"' (or a vlid biblionumber)
7 - should get expected result
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a new statuc function to Koha::SearchEngine::ElasticSearch
which is instended to replace most of get_elasticsearch_params. This function
reads the configuration from C4::Context->config('elasticsearch') and raises
relevant exceptions when mandatory entries are missing.
Its behaviour is covered by tests.
To test:
- Run:
$ kshell
k$ prove t/Koha/SearchEngine/Elasticsearch.t
=> SUCCESS: Tests pass!
- Sign off :-D
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
David Gustafsson
Julian Maurice
Nick Clemens
Alex Arnaud
Jonathan Druart
Robin Sheat
Fridolin Somers
Joonas Kylmälä
Ere Maijala
Nicolas Legrand
Alex Arnaud
Josef Moravec
root
Martin Renvoize
Tomas Cohen Arazi