Tests verify that whole_record and weighted_fields parameters are
correctly included in the query_cgi string returned by build_query_compat(),
ensuring these parameters are preserved in facet links and pagination.
To test:
1. Apply this patch
2. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: The values are not being passed around
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
0. Apply this patch and startup ktd with elastic (e.g `--search-engine es7`)
1. prove t/db_dependent/check_sysprefs.t
=> pass
2. DB cli> SELECT COUNT(*) FROM systempreferences;
3. prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4. DB cli> SELECT COUNT(*) FROM systempreferences;
=> No change, still the same number of sysprefs
5. prove t/db_dependent/check_sysprefs.t
=> pass
6. Revert this patch or checkout main
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
8. DB cli> SELECT COUNT(*) FROM systempreferences;
=> One syspref crated (ElasticsearchIndexStatus_mydb)
9. prove t/db_dependent/check_sysprefs.t
=> fail
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We currently have a sysprefs named "Elasticsearch*" and some name only "ES*". This patch unifies the names to always start with "Elasticsearch".
Test plan:
* Start koha-testing-docker, open the staff interface and go to Administration.
* Enter "Elasticsearch" into the System Preference search box and click "search"
* You will find a bunch of sysprefs
* Now search for "ESPreventAutoTruncate", which should find one match
* Apply the patch
* run (in ktd) the script `updatedatabase`
* You should see output like
DEV atomic update /kohadevbox/koha/installer/data/mysql/atomicupdate/bug_39526_unify_syspref_variable_names_for_elasticsearch.pl [09:35:03]: Bug 39526 - Unify system preference variable names for Elasticsearch
Renamed system preference 'ES..' to 'Elasticsearch..'
* Go again to to sysprefs and search again for "Elasticsearch"
* you should now find "ElasticsearchPreventAutoTruncate"
* Search for "ESPreventAutoTruncate", which should show "No system preferences matched your search for .." (and the list of ALL sysprefs)
Please note that the qa script explains about installer/data/mysql/mandatory/sysprefs.sql, but the last line is not touched by this patch, so not sure why this happens.
Sponsored-by: HKS3
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
DBIx::Class::Storage::DBI::insert(): Missing value for primary key column 'filter' on SearchMarcToField - perhaps you forgot to set its 'is_auto_increment' attribute during add_columns()? Treating 'filter' implicitly as an autoinc, and attempting value retrieval at /kohadevbox/koha/Koha/SearchField.pm line 42
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML. at t/db_dependent/Koha/SearchEngine/Elasticsearch.t line 854.
=> This one is expected: we have generated a record with large fields
item-level_itypes set but no itemtype set for item (1568) at /kohadevbox/koha/Koha/Schema/Result/Item.pm line 1100.
item-level_itypes set but no itemtype set for item (1568) at /kohadevbox/koha/Koha/Schema/Result/Item.pm line 1100.
=> Those 2 ones can be removed by create a biblio and items using
build_sample_biblio and build_sample_item
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Use of uninitialized value in string comparison (cmp) at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 536.
I think this is the correct fix. We generate a random av category that
is used in the data, so $label get undef:
521 $label = $authorised_values{$t};
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
NB, some previous test had to be adjusted to make it posible to call
search_auth_compat in _search and properly interpret its results.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When building queries for matchign Zebra uses OR to combine searches - Elasticsearch
has been effectively using 'AND' - this patch checks whether and_or has been passed
and follows the variable when present.
To test:
1 - Have an import file of auths with multiple 035
2 - Setup an authority matching rule
Search index: other-control-number
Score: 1000
Tag: 035
Subfields: a
3 - Make sure Koha is using Elasticsearch
4 - Stage file without matching, import file
5 - Stage file again, using your matching rule
6 - Note all records match
7 - Setup Cataloging->Marc modification template
8 - Add an action to delete first 035 field
9 - Run it against the records you just imported
You can use a report:
SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported)
10 - Change matching rule in batch to none, then back to your rule
11 - No matches!
12 - Apply patch, restart all
13 - Set matchign rule to none, then back to your rule
14 - Matches!
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Works as advertised!
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
to keep the syspref names consistent with the existings values. Also "ES" is not as self-documenting as "Elasticsearch".
No code change, so the test plan stays the same.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Adjust existing and add new
Signed-off-by: Matthias Meusburger <matthias.meusburger@biblibre.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Set search engine to ES
2 - Search the catalog for "*"
3 - Note results
4 - Edit first result
5 - Repeat search, results are reordered
6 - Apply patch
7 - Search for "*"
8 - Highest biblionumbers are returned first
9 - Edit first result, repeat search
10 - Order does not change
11 - Confirm sorting by author, title, etc passes the same trials
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1) Apply patch
2) Make sure tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t
pass
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This has been generated.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.
There should be no behavior change, however, queries to ES should be simpler
To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
MARC records with over 99999 bytes are invalid by spec, and when you use
UTF-8 encoded characters in your MARC records, there is the potential
to generate fatal errors in MARC::File::USMARC when it runs
"marc_to_utf8" from "MARC::File::Encode" during its "decode" operation.
That is, if you MARC::File::USMARC->encode a MARC record
with over 99999 bytes (including a number of UTF-8 bytes), there
is the potential when you run MARC::File:USMARC->decode on that same
data that you'll generate a fatal exception.
The main patch in bug 38913 wraps the function doing the decode,
so that a bad record doesn't crash processing.
Without the patch, this unit test will fail. With the patch, this
unit test will pass.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The patch was corrupted, and was rewritten as the original one by
Hammat
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added unit tests.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We add a level of bracket nesting. I checked with Nick that this
shouldn't cause any issues in the queries and we agreed it made sense so
I've gone ahead and updated the tests to reflect the change.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
NB in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t,
I had to modify four existing test by changing Local-number:123456 to
Personal-name:donald -- since Local-number should never be
auto truncated according to the submitted patch.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/
This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml
If blank we default to 5000.
To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Removed skip_record_index => 1 from automatic_renewals.pl. See BZ.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Mentored-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
A unit test has been added to show the pass-through of skip_record_index from AddRenewal() to the Item->store() method.
Test plan:
1) In the shell run prove -vv t/db_dependent/Koha/SearchEngine/Indexer.t
2) Observe Subtest 3: Test AddRenewal indexer call
3) All four tests should pass
Mentored-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch alters the code in the ES QueryBuilder. Reflecting
how things are handled in build_query_compat we clean the query,
but not the limits. In Zebra we simply recursivly call buildQuery,
but the ES query structure
This patchset adds a new ability to save searches on the staff client, and display them in the results
page on staff or opac as a new filter.
New filters can be added from the results page after a search, and there is an admin page for updating
deleting and renaming filters
There is a new permission to control management of these filters
New filters can be added that are not displayed along with facets, this allows for building custom links
using these filters to keep URLs shorter
Due to bug 30528 testing in ES is recommended
To test:
1 - Apply patches and update database and restart all
2 - Enable new system preference 'SavedSearchFilters'
3 - As superlibrarian perform a search in staff client, something broad like 'a'
4 - Note new 'Save search as filter' link on results page
5 - Click it, save search as new filter, check 'Staff client' visibility
6 - Perform another search
7 - Note the filter now appears above facets
8 - Click to it filter results
9 - Note results are limited by the new filter, and it is checked in the facets
10 - Confirm click the [x] removes the filter
11 - Go to administration->search filters
12 - Confirm the filter appears
13 - Edit and mark as OPAC visible
14 - Test OPAC to ensure it shows and can be applied/removed
15 - Copy URL with filter applied
16 - In adminsitration mark filter as not visible on staff or opac
17 - Confirm link above still works
18 - Create a new staff with catalogue and search filters permission
19 - Ensure they can access/save filters
20 - Remove filter permission and ensure they cannot
21 - Disable system preference
22 - Confirm links to search filters page are removed from admin home and admin sidebar
23 - Confirm filters do not appear on results and cannot be created
24 - Enable pref
25 - Create a filter
26 - From search filters page, click 'Edit search'
27 - Confirm you are taken to advanced search page letting you know which filter you are editing
28 - Confirm you can change searhc options and save
29 - Confirm you can perform the search from this page
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Sponsored-by: Lund University Library, Sweden
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Tomás Cohen Arazi
Jonathan Druart
Thomas Klausner
Janusz Kaczmarek
Nick Clemens
David Gustafsson
Koha Development Team
David Cook
Olivier V
Matt Blenkinsop
Martin Renvoize
Katrin Fischer
Frank Hansen
Julian Maurice