Reports may consume too much resources of the db. To avoid that a new config param
'report_sql_max_statement_time_seconds' was created, which sets the maximum time a report statement may
take to execute in seconds. If it takes longer it will be aborted.
reflowed the code, so that is is executed after is_sql_valid, and put
the code in a subroutine
Test plan:
0. if you run this test in ktd you should check if you are running
mysql or mariadb, and then be able to switch database
1. create a new report by going to
More > Reports > New SQL Report
(/cgi-bin/koha/reports/guided_reports.pl?op=add_form_sql)
enter the following SQL (which only purpose is to run long)
select ExtractValue(bm1.metadata,'//controlfield[@tag="001"]')
from biblio_metadata bm1
join biblio_metadata bm2 on
ExtractValue(bm1.metadata,'//controlfield[@tag="001"]') =
ExtractValue(bm2.metadata,'//datafield[@tag="773"]/subfield[@code="w"]')
give the Report a name eg "extract_long"
2. run the report
3. report should be executed and return several 1000 lines on ktd
4. apply patch
5. add a the parameter to koha-conf.xml to the <config> </config> part
<report_sql_max_statement_time_seconds>0.001</report_sql_max_statement_time_seconds>
OR use the following statemtn
sed -i '/<biblioserver>/a <report_sql_max_statement_time_seconds>0.001<\/report_sql_max_statement_time_seconds>' /etc/koha/sites/kohadev/koha-conf.xml
6. restart koha
koha-plack --restart kohadev
7. go back to the report page and rerun report which now should lead to an error
The following error was encountered:
The database returned the following error:
Query execution was interrupted, maximum statement execution time exceeded
Please check the log for further details.
8. in koha-conf.xml set
<report_sql_max_statement_time_seconds>20</report_sql_max_statement_time_seconds>
9. restart koha
10. rerun query, which now should execute as before
11. stop ktd, and restart with the "other" database
DB_IMAGE=mysql:8.0 ktd up
DB_IMAGE=mariadb:10.5 ktd up
12. repeat from step 1 (omit applying the patch)
13. Sign off
14. Thx
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This silences warnings about undefinved values in the cache key
This seems to come from some calls in _build_languages_arrayref where we might not
have all variants of a language - the call could be prevented, but I am not sure
what the effects would be, so I simply convert undef (null value) to "undef" (string)
and use that as the cache key
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This should have no visible effect, just internal caching
To test:
1 - Do a search, check the language facets
2 - Apply patch, restart all
3 - Do a search, check the language facets
4 - No change
5 - Sign off
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
1. Apply the patch
2. Load the OPAC main page (opac-main.pl)
3. Inspect the search form
→ Confirm role="search" is present
4. Use a screen reader (or browser accessibility tree)
→ Verify a "search" landmark is announced
5. Ensure only one search landmark is present on the page
6. Perform a catalog search
→ Confirm search functionality is unchanged
Signed-off-by: Laura Woodward <lwoodard@artsbma.org>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Add x-koha-embed parameter to GET /deleted/biblios/{biblio_id} and use
$c->objects->to_api() to properly handle embedding items.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The restore functionality is a cataloguing operation and should live
under the editcatalogue permission module (bit 9) rather than tools
(bit 13), consistent with where it appears in the UI.
Update all references:
- userpermissions.sql and atomicupdate: module_bit 13 → 9
- Swagger x-koha-authorization: tools → editcatalogue
- CGI flagsrequired: tools → editcatalogue
- Templates: CAN_user_tools_records_restore → CAN_user_editcatalogue_records_restore
- C4::Auth::get_cataloguing_page_permissions: remove from tools
sub-perms list (editcatalogue '*' already covers it)
- Tests: module_bit 13 → 9
Signed-off-by: Michaela <michaela.sieber@kit.edu>
Sponsored-by: ByWater Solutions
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
- Fix permission description to explicitly mention bibliographic records
and items (not just 'records')
- Fix date filter to use correct DB column per table: deletedbiblio uses
'timestamp', deleteditems uses 'deleted_on'
- Remove leftover CAN_user_editcatalogue_edit_catalogue condition from
the Batch editing section in cat-menu.inc; a user with only
edit_catalogue would see an empty section heading
- Wrap biblio and item restore operations in DB transactions so that a
failed store cannot leave a record inserted without being removed from
the deleted table (or vice-versa)
- Catch Koha::Exceptions::Object::DuplicateID in the item restore
controller and return a human-readable 409 instead of a generic 500
- When 'Restore bibliographic record' is clicked in the deleted-bib
warning modal, fetch the biblio and open the full restore modal with
the originally clicked item pre-selected, rather than restoring
immediately without item selection
Signed-off-by: Michaela <michaela.sieber@kit.edu>
Sponsored-by: ByWater Solutions
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This adds RESTORE action for biblios and items to the action log
Signed-off-by: Michaela <michaela.sieber@kit.edu>
Sponsored-by: ByWater Solutions
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
TEST PLAN: Item/Biblio Undelete Feature
SETUP:
- Apply patches and update database
- Give test user "Restore deleted records" permission under Cataloging
permissions
- Delete at least one bibliographic record with items attached
- Delete at least one standalone item
TESTING:
Navigate to Cataloging:
- Go to Cataloging home page
- Click "Restore deleted records" link (should be visible with
permission)
Test Restore Records Page:
- Verify two tables appear: "Deleted bibliographic records" and
"Deleted items"
- Verify deleted records appear in tables with biblio ID, title,
barcode, deleted date
Test Date Filtering:
- In left sidebar, enter date range in "Deleted from" and "Deleted to"
fields
- Click "Apply" button - tables should filter by date range
- Click "Clear" button - filter should reset
Test Item Restore:
- Find a deleted item in the items table (with existing biblio)
- Click "Restore" button in Actions column
- Verify success message appears and item disappears from table
- Search catalog for the restored item - confirm it's back
Test Biblio Warning Modal:
- Find a deleted item whose biblio is also deleted
- Click "Restore" button on the item
- Modal appears warning "bibliographic record is deleted"
- Click "Restore biblio" - biblio restores, modal closes
- Click "Restore" on the item again - item now restores successfully
- Verify both records are back in catalog
Test Biblio with Items Restore:
- Find deleted biblio with multiple deleted items
- Click "Restore" button for the biblio
- Modal shows checkboxes for each deleted item
- Select some items (not all), click "Restore with selected items"
- Verify biblio and selected items restore
- Unselected items remain in deleted items table
Permission Testing:
- Remove "Restore deleted records" permission from user
- Verify "Restore deleted records" link no longer appears in
Cataloging home
Signed-off-by: Michaela <michaela.sieber@kit.edu>
Sponsored-by: ByWater Solutions
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To test:
1. Go to the Serials module and create or edit a subscription.
2. Search for a vendor to add. Note that only the vendor name and Select column are visible.
3. Apply the patch and refresh the vendor search result page.
4. Notice that the vendor ID column is now visible.
5. Sign off and have a great day! :D
Signed-off-by: Emmanuel Bétemps <e.betemps@gmail.com>
Signed-off-by: Angela Berrett <angela.berrett@familysearch.org>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Koha's XSLT display path now calls a plugin hook so plugins can
extend the record processor filter list before MARC is transformed.
This enables prototype punctuation filters (and similar) to run
as plugins without core changes.
Test plan:
1) Enable plugins in koha-conf.xml.
2) Install the TestPunctuation plugin from https://github.com/HKS3/koha-test-punctuation
3) Enable the plugin in Koha admin.
4) View a biblio in staff or OPAC with XSLT display enabled and confirm
the filter runs (e.g., 245$a gets a trailing period if missing).
Signed-off-by: Alexander Wagner <alexander.wagner@desy.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Images of covers must preserve transparency of png-images. This is
useful for example for the covers of CD/DVD disks.
Plan test :
1. Apply the patch.
2. In 'Koha administration > System preferences', set
'LocalCoverImages' to 'show' and OPACLocalCoverImages to 'show'.
3. Create a new bibligraphical record
4. In the record details select tab Images and click on upload
5. Click on « Drop files here » and select the image file to upload then
click «Process images»
6. Search for the record created on step 3
7. Click on the record to view detail page (detail.pl)
8. Click on the image to view it fully
---> Notice the background is transparent
9. Save the image and open it
---> Notice the background is transparent
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
On the OPAC, news should rather be ordered by the publication date than
by the creation date (~ id): a user want to see last published news
first and not last created news, regardless of their publication date
TEST PLAN:
1 - Create 3 news, have the 3 news published in an order different from
their id (for example news 1 => 2020, news 2 => 2025, news 3 => 2023)
2 - Go on http://youropac/cgi-bin/koha/opac-main.pl -> news are ordered
by ID
APPLY PATCH
3 - Repeat 2 -> news are ordered by Date
Signed-off-by: Mercury <mercury@coffee.geek.nz>
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
When merging (or updating) authorities, the biblios using the auths might also be updated. If this happens, Koha used to create one background job for each biblio. If an auth is used in a lot of biblios, this creates a lot of background jobs, each to index one single biblio.
This patch fixes that by collecting the affected biblios and putting all of them into one reindex background job.
Test plan:
* Apply the patch
* run the unit test t/db_dependent/Authority/Merge.t
* the test should now have 16 tests in total, with a new test #15 output like this:
1..3
ok 1 - 3 bibliographic record updated while merging two authorities
ok 2 - we have only one more background job
ok 3 - we have the correct biblio ids in the job to be indexed
ok 15 - merge with ModBiblio only creates one indexer background job for multiple biblios
ok 16 - no warnings
Manual test plan (rather cumbersome):
* Start KTD with ElasticSearch
* Create an PERSON authority (eg "100a Test")
* Create three biblios which use that authority in eg 100
* Access the DB to get the count of background jobs
* `ktd --dbshell`
* `select count(*) from background_jobs;`
* the count should be the same as the number of biblios you created (eg 3). But whatever it is, remeber it
* Edit the authority (change 100 to "Test new")
* check if the change is reflected in the linked biblios (100a = "Test new")
* Check the Background jobs in the DB again:
* `select count(*) from background_jobs;`
* should be three more than previously (eg 6)
* You can use this SQL to verify that the last three jobs each contain one biblionumber (which is what we want to fix):
* `select json_extract(data,'$.record_ids'),enqueued_on from background_jobs order by enqueued_on desc limit 3;`
+-----------------------------------+---------------------+
| json_extract(data,'$.record_ids') | enqueued_on |
+-----------------------------------+---------------------+
| [2299] | 2026-03-25 13:19:34 |
| [2300] | 2026-03-25 13:19:34 |
| [2301] | 2026-03-25 13:19:34 |
+-----------------------------------+---------------------+
* Now apply the patch and run restart_all
* Edit the authority again, setting 100a to "Test new patched".
* The biblios should change accordingly
* Check the Background jobs in the DB again:
* `select count(*) from background_jobs;`
* This should now be only 1 more then the last time (so eg 7)
* Check the details of the background jobs:
* `select json_extract(data,'$.record_ids'),enqueued_on from background_jobs order by enqueued_on desc limit 3;`
* The most recent entry should now contain three items in one row, and the following two should be the same as in the previous query (i.e. the old jobs from the edit before the patch)
+-----------------------------------+---------------------+
| json_extract(data,'$.record_ids') | enqueued_on |
+-----------------------------------+---------------------+
| [2299, 2300, 2301] | 2026-03-25 13:29:17 |
| [2300] | 2026-03-25 13:19:34 |
| [2299] | 2026-03-25 13:19:34 |
+-----------------------------------+---------------------+
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
MARC 21 authority records often have 024 $a (Other Standard
Identifier) - https://www.loc.gov/marc/authority/ad024.html,
but the field 024 is missing from ES search fields and the record
cannot be found with these identifiers.
Test plan
=========
1. Have a ktd instance with Elasticsearch.
2. Add a 024 field to an authority record, i.e. for Alatriste, Gustavo:
024 7# $a 86971927 $7 viaf
3. In Authorities, make an 'Entire record' search for the added
identifier. You should get no results.
4. Apply the patch, reindex with:
sudo koha-elasticsearch --rebuild -r -d -a kohadev
5. Repeat p. 3. You should now be able to find the record.
Sponsored-by: Pontificia Università di San Tommaso d'Aquino (Angelicum)
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
- bug_number was set to 40853 instead of 21820 (probably copy-paste error)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Previously, Koha always added items to the Zebra queue when indexing via Elasticsearch. This adds a new system preference ElasticsearchEnableZebraQueue that allows disabling this behavior.
Unit Test:
* run the unit test t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
* there should be a new subtest like this:
1..5
ok 1 - We start with an empty zebra queue
ok 2 - new entry in zebraqueue for auth
ok 3 - new entry in zebraqueue for biblio
ok 4 - no new entry in zebraqueue for auth
ok 5 - no new entry in zebraqueue for biblio
ok 5 - syspref ElasticsearchEnableZebraQueue
Manual Test Plan:
* Start ktd with Elasticsearch enabled
* As there is no interface to the zebraqueue, you need to connect to the DB and use SQL directly
* ktd --dbshell
* `select count(*) from zebraqueue;`
* remember that number
* Edit any biblio or auth
* Check the zebraqueue again via SQL
* There should be one more entry
* Now apply the patch
* Then got to Sysprefs and set ElasticsearchEnableZebraQueue to `Disable`
* Edit another biblio or auth
* has to be a different one, because Koha will prevent the same ID to be added again to zebraqueue
* Again check zebraqueue via SQL
* There should be *no* new entry
QA Note:
the qa tool reports
[FAIL] Koha/SearchEngine/Elasticsearch/Indexer.pm
FAIL pod_coverage
sub _get_record has no pod coverage.
But this patch does not touch this code, so I guess this fail is caused by a stricter qa tool
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds an 'Export mappings' button to the Elasticsearch
mappings configuration page. It triggers a YAML download of the
current search field mappings using the new API endpoint.
To test:
1. Apply this patch (and the previous API endpoint patch)
2. Navigate to Administration > Search engine configuration
=> SUCCESS: An 'Export mappings' button appears next to 'Reset mappings'
3. Click 'Export mappings'
=> SUCCESS: A 'search_mappings.yaml' file is downloaded
4. Verify the YAML file contains the expected mappings structure
=> SUCCESS: File contains search fields grouped by index
5. Sign off :-D
Signed-off-by: Laurence <laurence.rault@biblibre.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new API endpoint to retrieve the search field
mappings configuration (search fields and their MARC mappings, grouped
by index).
The endpoint supports content negotiation:
- application/json (default)
- application/yaml (for export/import compatibility)
An optional 'type' query parameter allows filtering mappings by MARC
type (marc21 or unimarc).
Implementation notes:
- New controller: Koha::REST::V1::Config::SearchMappings
- Reuses Koha::SearchEngine::Elasticsearch::raw_elasticsearch_mappings
- Registers 'application/yaml' MIME type in Koha::REST::V1
- Permission: manage_search_engine_config
To test:
1. Apply this patch
2. Rebuild the API bundle:
$ yarn api:bundle
3. Restart Plack
4. Run:
$ prove t/db_dependent/api/v1/config_search_mappings.t
=> SUCCESS: Tests pass
5. Test manually:
$ curl -u user:pass /api/v1/config/search_mappings
=> SUCCESS: Returns JSON with search field mappings
$ curl -u user:pass -H 'Accept: application/yaml' \
/api/v1/config/search_mappings
=> SUCCESS: Returns YAML output
$ curl -u user:pass /api/v1/config/search_mappings?type=marc21
=> SUCCESS: Only marc21 mappings are returned
6. Sign off :-D
Signed-off-by: Laurence <laurence.rault@biblibre.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
koha-elasticsearch only allows the parameters to be specified on the
command line, otherwise it'll fall back to values hard-coded in the
script. This allows default parameters to be defined in
/etc/default/koha-common when installed with the Debian packages.
It also copies the current defaults into the defaults file.
Test plan, comprehensive:
1. You will need a package setup with elasticsearch, and some records.
2. Build a package (or copy the files modified in this patch over ones
you might already have installed)
3. Modify some ES_* values in /etc/default/koha-community to your
liking. 'ES_VERBOSE' might be a good one.
4. Do an elasticsearch reindex and observe that your parameter defaults
are respected.
Test plan, simple:
1. With a git checkout:
* `sudo ln -s .../koha/debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh`
* `sudo ln -s .../koha/debian/koha-common.default /etc/default/koha-common`
2. Modify `.../koha/debian/scripts/koha-elasticsearch` around line 200
so that it prints out the variables, and see that it properly respects
modifying the defaults and/or setting the command line arguments.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Refactor marc_record_to_documents to avoid code duplicaiton.
To test:
1) Verify tests in t/db_dependant/Koha/SearchEngine/Elasticsearch.t all
pass
2) Apply patch
3) Veriy tests in t/db_dependant/Koha/SearchEngine/Elasticsearch.t
still pass
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Lucas Gass
Kyle M Hall
Mark Hofstetter
Nick Clemens
Laura_Escamilla
Jacob O'Mara
Martin Renvoize
Owen Leonard
Hammat Wele
Baptiste
Thomas Klausner
Janusz Kaczmarek
Paul Derscheid
Tomás Cohen Arazi
Robin Sheat
David Gustafsson