gitea
/
Koha-clms
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Koha-clms/Koha
History
Tomas Cohen Arazi ce4522db14
Bug 38101: Make ES indexer split big fields into chunks 
This patch makes the `_process_mappings()` method split the index values
in the event of them being bigger than the allowed 32766 bytes size.

To test:
1. Have KTD running with ES:
   $ ktd --proxy --es7 up -d
2. Perform a search
3. Pick the first result for edition
4. Find a cool Wiki page with lots of paragraphs
5. Copy all of the paragraphs and put them on a 500$a field for the record.
6. Repeat 2
=> FAIL: The record is not found
7. Reindex manually:
   $ ktd --shell
  k$ perl misc/search_tools/rebuild_elasticsearch.pl --biblios --where "biblionumber=3"  -v -v
=> FAIL: You get something like:
```
[22229] Committing final records...
One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
[22229] There were errors during indexing
Record #3 Document contains at least one immense term in field="note.raw" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[10, 109, 117, 115, 116, 97, 102, 97, 32, 102, 117, 101, 32, 101, 108, 32, 115, 101, 103, 117, 110, 100, 111, 32, 104, 105, 106, 111, 32, 100]...', original message: bytes can be at most 32766 in length; got 32771 (illegal_argument_exception) : max_bytes_length_exceeded_exception (bytes can be at most 32766 in length; got 32771)
[22229] Total 1 records indexed
```
8. Apply this patch
9. Repeat 7
=> SUCCESS: No error!
10. Repeat 2
=> SUCCESS: The record is indexed and can be found!
11. Sign off :-D

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 6998f5b7e1)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
 		2025-02-04 21:50:22 +01:00
..
Account Bug 22421: (QA follow-up) Update API mapping and specs 2024-11-08 17:06:06 +01:00
Acquisition Bug 38011: Add a foreign key between vendors and subscriptions 2024-11-11 17:43:06 +01:00
App Bug 36598: (QA follow-up): tidy up code 2024-10-18 12:07:50 +02:00
ArticleRequest
…
Auth Bug 37157: Fix Malformed UTF-8 character in JSON string before decode_json 2024-06-27 11:50:06 +02:00
Authority
…
BackgroundJob Bug 34355: (QA follow-up) Make record staging code DRY 2024-11-11 17:43:18 +01:00
Biblio Bug 31224: Move item field removal up to $metadata->record 2024-11-08 11:23:44 +01:00
Cache Bug 29623: (QA follow-up) Add POD to Koha::Cache::Lite 2022-06-14 08:26:59 -03:00
Cash Bug 36481: (QA follow-up) Rename branch_default to library_default 2024-05-01 13:51:41 +01:00
Charges
…
Checkouts Bug 33362: Allow return claims to be resolved even if the issue has since been delete from the database 2023-05-24 11:21:31 -03:00
Club Bug 29523: (QA follow-up) Fix failing test in club holds 2023-10-27 16:40:43 -03:00
CodeList/Unimarc Bug 30941: (QA follow-up) Only export what is needed 2022-08-19 11:00:42 -03:00
Config Bug 32478: Remove Koha::Config::SysPref->find since bypasses cache 2023-06-07 15:25:18 -03:00
Course Bug 33568: Course reserves 2024-04-05 15:34:52 +02:00
Database Bug 28633: (QA follow-up) Fix column description 2024-11-11 14:12:12 +01:00
DateTime/Format Bug 36432: Remove circular dependency from Koha::Object 2024-04-04 15:24:32 +02:00
ERM Bug 33339: (Follow up) Fix more cases of formula=empty 2024-11-14 10:46:31 +01:00
Edifact Bug 35724: (QA follow-up) Also account for port in FTP 2024-04-19 18:31:37 +02:00
Exceptions Bug 37510: Make Koha::Object->delete throw Koha::Exception 2024-08-16 17:33:46 +02:00
Exporter Bug 31286: (QA follow-up): tidy up code 2024-03-25 09:53:24 +01:00
ExternalContent Bug 33697: Remove RecordedBooks (rbdigital) integration 2023-06-05 15:36:11 -03:00
Filter/MARC Bug 34481: Tidy altered code 2024-06-27 14:04:50 +02:00
Hold Bug 28966: (QA follow-up) Changes for Koha objects 2023-07-25 16:25:39 -03:00
ILL Bug 36221: (QA follow-up): Fix translation entry concat 2024-11-13 17:00:50 +01:00
Import Bug 37184: Special character encoding problem when importing MARC file from the Acquisitions module 2024-10-29 16:14:33 +01:00
Indexer
…
Installer Bug 38299: Make filehandler optional for colored outputs 2024-11-18 17:44:37 +01:00
Item Bug 33568: Display library names instead of codes for transfers 2024-04-05 15:34:56 +02:00
Library Bug 6796: Add library_hours table and set opening hours for library 2024-04-12 11:26:03 +02:00
List
…
MetadataRecord Bug 35345: Add --where option to rebuild_elasticsearch.pl 2024-04-22 12:39:26 +02:00
Middleware Bug 36560: (QA follow-up) Tidy code 2024-10-25 16:41:29 +02:00
Misc
…
Notice Bug 23295: (follow-up) Add 'NOTICE_FAILURE_SUSPENSION' as a new system restriction type 2024-10-24 10:45:39 +02:00
Number Bug 37511: Location of the currency symbol (before or *after*) the amount 2024-10-30 11:57:26 +01:00
OAI Bug 38605: Fix Harvester.pm use of days parameter 2024-12-12 14:04:56 +01:00
Object Bug 36836: (QA follow-up) Tiny fixes to terminology and capitalization 2024-10-24 13:38:23 +02:00
Objects Bug 35287: (QA follow-up): Squash 2024-10-24 13:38:22 +02:00
Old Bug 38678: Fix wrong attribute mapping in Koha::Old::Biblio 2025-02-04 21:40:55 +01:00
Patron Bug 28633: Add preferred name field to patrons 2024-11-11 14:12:09 +01:00
Patrons Bug 33339: Prevent Formula Injection (CSV Injection) in CSV files 2024-11-14 10:46:31 +01:00
Plugins Bug 35536: Add RemovePlugins calls in plugin unit tests 2024-03-22 12:26:39 +01:00
Policy Bug 30825: Move holds_control_library to Koha::Policy::Holds 2023-09-22 12:35:46 -03:00
Preservation Bug 33547: (QA follow-up) Tidy 2023-10-23 11:33:53 -03:00
REST Bug 38466: Fix error messages in KBART import 2024-12-12 14:04:06 +01:00
RecordProcessor
…
Result
…
SMS
…
SMTP Bug 29523: (QA follow-up) Catch remaining SUPER::to_api cases 2023-10-27 16:40:42 -03:00
Schema Bug 38522: DBIC schema update 2024-12-13 12:24:44 +01:00
SearchEngine Bug 38101: Make ES indexer split big fields into chunks 2025-02-04 21:50:22 +01:00
Serial Bug 33568: enumchron/volinfo 2024-04-05 15:34:47 +02:00
Sitemapper Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml 2024-10-30 14:34:52 +01:00
Subscription
…
SuggestionEngine Bug 31069: Remove ExplodedTerms dependency on templates 2022-07-18 12:37:35 -03:00
Tags
…
Template/Plugin Bug 38758: Add strftime as TT virtual method 2024-12-30 12:22:31 +01:00
Ticket Bug 35657: Add support for assignee_id to ticket_updates endpoint 2024-05-02 15:03:35 +02:00
UI Bug 33292: Add a checkbox to refund a lost fee on return claim 2024-10-24 13:38:14 +02:00
Util Bug 36976: Warning 'Argument "" isn't numeric in numeric' in log when merging bibliographic records 2024-06-28 13:49:24 +02:00
XSLT Bug 12758: Add new module call in Koha::XSLT::Base 2022-11-09 09:52:25 -03:00
Z3950Responder Bug 36996: Add z3950Status system preference 2024-07-23 16:53:03 +01:00
pdfformat Bug 33393: tidy layout1page.pm 2024-04-19 19:54:46 +02:00
Account.pm Bug 22421: (follow-up): Check issue during add_credit 2024-11-08 12:42:07 +01:00
ActionLog.pm
…
ActionLogs.pm
…
AdditionalContent.pm Bug 36875: (follow-up) Modify query in translated_content 2024-06-20 17:55:39 +02:00
AdditionalContents.pm Bug 37684: Direct link to expired news are broken 2024-11-07 17:32:38 +01:00
AdditionalContentsLocalization.pm Bug 31383: (QA follow-up) Tidy 2023-10-20 14:44:03 -03:00
AdditionalContentsLocalizations.pm Bug 31383: Create a parent-child DB relation for additional content 2023-10-20 14:43:56 -03:00
AdditionalField.pm Bug 37577: Add additional fields for ERM packages 2024-11-08 16:09:19 +01:00
AdditionalFieldValue.pm
…
AdditionalFieldValues.pm
…
AdditionalFields.pm
…
AdvancedEditorMacro.pm
…
AdvancedEditorMacros.pm
…
ApiKey.pm
…
ApiKeys.pm
…
ArticleRequest.pm Bug 35030: Add 'article' to patron activity triggers 2023-10-24 10:05:26 -03:00
ArticleRequests.pm
…
AudioAlert.pm
…
AudioAlerts.pm Bug 29859: Use iterator instead of as_list 2022-02-09 15:36:23 -10:00
AuthUtils.pm Bug 33341: Address some perlcritic errors in 5.36 2023-03-28 14:50:33 +02:00
AuthorisedValue.pm Bug 34920: Ensure 'description' property is an empty string if 'lib' is null 2024-07-01 18:55:48 +02:00
AuthorisedValueCategories.pm
…
AuthorisedValueCategory.pm Bug 32997: Add REST API endpoint to list authorised values for multiple given categories 2023-03-02 12:00:15 -03:00
AuthorisedValues.pm Bug 35579: Cache authorised value lookup by MARC field 2023-12-28 12:16:09 +01:00
Authorities.pm Bug 33083: Add a generic class to handle methods for collections of records 2023-03-13 15:11:59 -03:00
Authority.pm Bug 29948: OPACAuthorIdentifiersAndInformation 2024-05-13 14:00:10 +02:00
BackgroundJob.pm Bug 35655: (follow-up) Fix perlcritic error 2024-09-30 13:52:21 -03:00
BackgroundJobs.pm Revert "Bug 36217: Fix background jobs page's include_last_hour filter" 2024-10-18 14:26:01 +02:00
Biblio.pm Bug 31224: EmbedItems takes an arrayref 2024-11-07 16:29:24 +01:00
BiblioFramework.pm
…
BiblioFrameworks.pm
…
BiblioUtils.pm Bug 31224: Update instances of metadata->record 2024-11-07 16:29:23 +01:00
Biblioitem.pm Bug 32711: Add 'public_read_list' appropriately 2023-07-24 13:58:25 -03:00
Biblioitems.pm
…
Biblios.pm Bug 33974: (follow-up) biblionumber column needs special handling 2023-07-03 14:51:46 -03:00
Booking.pm Bug 38175: (follow-up) Quiet warnings in tests 2024-11-04 13:12:12 +00:00
Bookings.pm Bug 38175: (QA follow-up) Ensure we handle cancelled in clash detection 2024-10-24 18:27:24 +02:00
Cache.pm
…
Caches.pm
…
Calendar.pm Bug 27249: Prevent infinite loop when searching for an open day 2023-10-17 14:45:31 -03:00
Checkout.pm Bug 33641: (follow-up) Add field to api spec 2024-11-15 09:13:19 +01:00
Checkouts.pm Bug 33887: (QA follow-up) Perltidy and fix some formatting issues 2023-10-25 12:09:53 -03:00
CirculationRule.pm Bug 32894: Remove wrong caching from Koha:: methods - simple 2023-06-07 16:44:03 -03:00
CirculationRules.pm Bug 34440: Add lead and trail time to circulation rules 2024-08-30 16:41:26 +02:00
Cities.pm
…
City.pm
…
ClassSortRule.pm
…
ClassSortRules.pm
…
ClassSource.pm Bug 30920: (follow-up) Flush on delete 2023-03-13 15:11:44 -03:00
ClassSources.pm
…
ClassSplitRule.pm
…
ClassSplitRules.pm
…
Club.pm Bug 29844: Fix ->search occurrences 2022-02-09 15:36:23 -10:00
Clubs.pm Bug 29844: Simplify Koha::Clubs->get_enrollable 2022-02-09 15:36:23 -10:00
Config.pm Bug 31393: Koha::Config: handle the special case for 'content' attribute 2023-10-30 09:02:17 -03:00
CookieManager.pm Bug 31250: (QA follow-up) The future will be easier 2022-09-12 15:34:11 -03:00
Course.pm Bug 28762: Use Koha::Course in course-details controller 2024-07-23 16:04:05 +01:00
Courses.pm
…
CoverImage.pm
…
CoverImages.pm
…
CsvProfile.pm
…
CsvProfiles.pm
…
CurbsidePickup.pm Bug 32496: Reduce unnecessary unblessings of objects in Circulation.pm 2023-09-22 10:52:39 -03:00
CurbsidePickupIssue.pm Bug 30650: Add some useful modules and tests 2022-07-29 14:59:57 -03:00
CurbsidePickupIssues.pm Bug 30650: Add some useful modules and tests 2022-07-29 14:59:57 -03:00
CurbsidePickupOpeningSlot.pm Bug 30650: Add some useful modules and tests 2022-07-29 14:59:57 -03:00
CurbsidePickupOpeningSlots.pm Bug 30650: Add some useful modules and tests 2022-07-29 14:59:57 -03:00
CurbsidePickupPolicies.pm Bug 30650: Add some useful modules and tests 2022-07-29 14:59:57 -03:00
CurbsidePickupPolicy.pm Bug 32891: Fix slot selection in last hour 2023-02-21 10:22:28 -03:00
CurbsidePickups.pm Bug 30650: Add filter_by_scheduled_today 2022-07-29 15:00:47 -03:00
Database.pm Bug 36026: Use only configured TLS options for database connection 2024-11-11 15:35:36 +01:00
DateUtils.pm Bug 36432: Remove circular dependency from Koha::Object 2024-04-04 15:24:32 +02:00
Desk.pm Bug 36480: Add GET /libraries/:library_id/desks 2024-04-26 18:06:10 +02:00
Desks.pm
…
EDI.pm Bug 38271: (QA follow-up) Tidy code 2024-10-29 08:46:39 +01:00
Edifact.pm
…
Email.pm Bug 31627: (follow-up) Update -Id to -ID 2024-05-10 14:11:50 +02:00
Encryption.pm Bug 33934: Add more detail to 'No encryption_key in koha-conf.xml' 2023-06-07 15:25:18 -03:00
Exception.pm Bug 29857: (QA follow-up) Add license 2022-03-01 22:47:47 -10:00
Exceptions.pm Bug 28869: Implement authorised_value_categories.is_integer_only 2024-05-17 10:36:23 +02:00
ExternalContent.pm
…
FrameworkPlugin.pm Bug 37146: Prevent path traversal by validating input 2024-08-01 17:26:37 +02:00
Hold.pm Bug 30648: Add new field to REST API 2024-10-29 08:46:44 +01:00
Holds.pm Bug 32529: Add Koha::Holds->filter_by_found 2023-01-05 09:21:10 -03:00
I18N.pm Bug 35174: Do not explode if po dir does not exist 2023-11-16 08:56:36 +01:00
ImportBatch.pm Bug 34355: (QA follow-up) Permissions fixes 2024-11-11 17:43:18 +01:00
ImportBatchProfile.pm
…
ImportBatchProfiles.pm
…
ImportBatches.pm
…
Installer.pm Bug 37818: Remove 'XXX' from the installer code 2024-09-13 13:54:17 +02:00
Item.pm Bug 36182: Show vendor name if acquisition_source is a vendor_id 2024-11-12 17:50:15 +01:00
ItemType.pm Bug 35906: Add bookable option on itemtype 2024-10-25 15:34:40 +02:00
ItemTypes.pm
…
Items.pm Bug 35906: (QA follow-up) Refactor (slightly) to reduce complexity in Koha/Item(s).pm, catalogue/updateitem 2024-10-25 15:34:44 +02:00
KeyboardShortcut.pm
…
KeyboardShortcuts.pm
…
Libraries.pm
…
Library.pm Bug 37850: (QA follow-up): Update illemail REST API attribute name 2024-11-12 17:50:16 +01:00
Linktracker.pm
…
Localization.pm Bug 30920: (follow-up) Flush on delete 2023-03-13 15:11:44 -03:00
Localizations.pm
…
Logger.pm Bug 35907: Add ability to log all custom report runs with or without query 2024-05-10 16:45:45 +02:00
Manual.pm Bug 35463: More preservation manual links 2023-12-20 15:13:45 +01:00
MarcOrder.pm Bug 38326: (QA follow-up) Tidy 2024-11-19 16:35:13 +01:00
MarcOrderAccount.pm Bug 34355: Add account creation 2024-11-11 17:43:12 +01:00
MarcOrderAccounts.pm Bug 34355: Add account creation 2024-11-11 17:43:12 +01:00
MarcOverlayRule.pm
…
MarcOverlayRules.pm
…
MarcSubfieldStructure.pm
…
MarcSubfieldStructures.pm
…
MetaSearcher.pm Bug 30813: (QA follow-up) Adjust three use statements 2022-06-08 11:40:32 -03:00
MetadataIterator.pm
…
MetadataRecord.pm Bug 33749: Use TrimFields instead of stripWhitespaceChars 2023-11-03 12:08:06 -03:00
MoreUtils.pm
…
OAIServer.pm Bug 35659: (more QA follow-ups) 2024-10-28 14:01:29 +01:00
OAIServers.pm Bug 35659: (more QA follow-ups) 2024-10-28 14:01:29 +01:00
OAuth.pm
…
OAuthAccessToken.pm
…
OAuthAccessTokens.pm
…
Object.pm Bug 38342: Pass the value to exception to avoid warnings 2024-11-07 18:49:40 +01:00
Objects.pm Bug 36873: Make Koha::Objects->delete pass parameters to the underlying method 2024-07-01 18:55:43 +02:00
OverdueRule.pm Bug 29145: (QA follow-up) Use objects instead of straight SQL queries 2023-10-03 09:24:41 -04:00
OverdueRules.pm Bug 29145: (QA follow-up) Use objects instead of straight SQL queries 2023-10-03 09:24:41 -04:00
Patron.pm Bug 36454: (QA follow-up) Use native is_expired instead of redundant JS calculation 2024-11-18 17:04:25 +01:00
Patrons.pm Bug 37845: Remove C4::Members::DeleteExpiredOpacRegistrations 2024-10-21 15:10:31 +02:00
Plugins.pm Bug 30897: Remove recursive parent identification loop 2024-04-11 16:53:43 +02:00
ProblemReport.pm
…
ProblemReports.pm
…
PseudonymizedTransaction.pm Bug 34611: Add pseudonymize routine to Koha::Statistics 2024-03-19 10:25:05 +01:00
PseudonymizedTransactions.pm
…
Quote.pm
…
Quotes.pm
…
Rating.pm
…
Ratings.pm
…
Recall.pm Bug 23781: SMS notices and messaging preferences for recalls 2024-07-23 16:38:08 +01:00
Recalls.pm Bug 32696: Recalls can inadvertently extend the due date 2024-08-19 17:29:54 +02:00
RecordProcessor.pm
…
RecordSource.pm Bug 37513: Add Koha::RecordSource->usage_count 2024-08-27 12:14:24 +02:00
RecordSources.pm Bug 35919: Add Koha::RecordSource(s) classes 2024-04-26 17:06:03 +02:00
Report.pm Bug 32413: Fix repeated param names for JSON report 2024-10-30 07:47:48 +01:00
Reports.pm
…
Review.pm
…
Reviews.pm
…
Schema.pm
…
Script.pm Bug 36367: Remove _new_userenv 2024-06-14 11:17:17 +01:00
SearchEngine.pm
…
SearchField.pm Bug 20388: (QA follow-up) Tidy 2024-04-16 17:59:47 +02:00
SearchFields.pm
…
SearchFilter.pm Bug 37333: Group search terms in a saved filter 2024-07-18 17:53:11 +02:00
SearchFilters.pm Bug 17170: DB Updates 2022-10-21 11:15:13 -03:00
SearchMarcMap.pm
…
SearchMarcMaps.pm
…
Serial.pm
…
Serials.pm
…
Session.pm Bug 36098: Default to 'file' if pref does not exist 2024-03-01 10:55:51 +01:00
SharedContent.pm Bug 27490: Update systempreference 'language' to 'StaffInterfaceLanguages' 2024-09-13 13:54:25 +02:00
SimpleMARC.pm Bug 36515: (QA follow-up) Tidy code 2024-08-02 19:00:05 +02:00
Sitemapper.pm Bug 33871: Use Koha object in Sitemapper, add optional filter 2023-09-01 11:23:25 -03:00
Statistic.pm Bug 37540: Ensure datetime is a string before enqueueing pseudonymization 2024-08-01 18:22:27 +02:00
Statistics.pm
…
StockRotationItem.pm Bug 28294: Remove updateWrongTransfer 2024-11-07 18:49:36 +01:00
StockRotationItems.pm
…
StockRotationRota.pm
…
StockRotationRotas.pm
…
StockRotationStage.pm
…
StockRotationStages.pm
…
Subscription.pm
…
Subscriptions.pm
…
Suggestion.pm Bug 35597: Move modification logging to Koha::Suggestion 2024-06-13 14:08:33 +01:00
SuggestionEngine.pm
…
Suggestions.pm Bug 33238: Fix bad suggestions query (Bug 29886 follow-up) 2023-04-17 09:39:43 -03:00
Tag.pm
…
Tags.pm
…
TemplateUtils.pm Bug 33030: Add standardized subroutine for processing Template Toolkit syntax outside of notices 2023-07-14 16:52:05 -03:00
Ticket.pm Bug 36758: (follow-up) Fix unit tests 2024-08-12 14:01:08 +02:00
Tickets.pm Bug 31028: Add new Koha::Object(s) classes 2023-03-06 11:23:17 -03:00
Token.pm Bug 36102: If CSRF check fails, try with anonymous 2024-03-01 11:00:34 +01:00
TrackedLink.pm
…
TrackedLinks.pm
…
UploadedFile.pm Bug 29156: File missing warning in Koha::UploadedFile for permanent files only 2022-01-21 21:34:51 -10:00
UploadedFiles.pm
…
Uploader.pm Bug: 36818 Escape characters in file names uploaded 2024-06-20 17:55:37 +02:00
Virtualshelf.pm Bug 28965: (QA follow-up) Fix misc qa issues 2024-11-08 16:09:22 +01:00
Virtualshelfcontent.pm
…
Virtualshelfcontents.pm
…
Virtualshelfshare.pm Bug 32894: Make ->sharee looks like others 2023-06-07 16:44:05 -03:00
Virtualshelfshares.pm
…
Virtualshelves.pm Bug 28965: (QA follow-up) Fix misc qa issues 2024-11-08 16:09:22 +01:00
Z3950Responder.pm
…
Z3950Server.pm
…
Z3950Servers.pm
…