If PatronSelfRegistration we should ignore (i.e. do nothing) if
--del-exp-selfreg is passed to cleanup_database.pl
It prevents the script to warn that the pref is not set.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
(cherry picked from commit d331c7aba8)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
Signed-off-by: Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz>
(cherry picked from commit 768f2a4882)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
After bug 35659 that added the OAI harvester to Koha the -d switch
for deleting authorities before importing with bulkmarcimport.pl
was broken and resulted in a FK error.
Switching from truncate to DELETE FROM like we already did for items,
biblio etc. fixes the issue.
To test:
- Make sure you have some authorities in your database
- Export some into a file for importing
- Run bulkmarcimport -a -d --file <filename>
- Verify you get the a DB FK constraint error.
- Apply patch
- Repeat steps, verify the script now works as expected again
* Existing authorities are deleted
* File is imported
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 59d07337de)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit eb6241644a)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The singular form is wrong and only exists in textual context.
This is a result of git grep | xargs sed.
Test plan:
Read the patch.
git grep BorrowerLog
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Aleisha Amohia <aleisha@catalyst.net.nz>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 4a267d0e9b)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit f59d8c46e3)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Cronjob cleanup_database.pl usage is wrong :
Usage: $0 [-h|--help] [--confirm] (...) [--bg-days DAYS [--bg-type TYPE] ]
Those args has been renamed jobs-days and jobs-type.
Also some args are missing.
There are too many args, I propose to remove them from there, one need to read the full help.
Test plan:
Run command line `misc/cronjobs/cleanup_database.pl -h`
You see first line:
Usage: misc/cronjobs/cleanup_database.pl
-h --help prints this help message, and exits, ignoring all
...
Signed-off-by: Anke Bruns <anke.bruns@gwdg.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 3aefbaeaf6)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit d46e9d25a9)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
If the holds reminder cronjob would send a reminder on a holiday,when holidays are taken into account, it also will send it on the subsequent holidays and the first open day.
This bugfix adds a check to the cron when --holidays is set to skip sending the notices for that library if today is a holiday.
To Test:
Before applying:
1. Create a hold for a patron (I used patron 23529000035676 and 39999000000252)
2. Ensure your patron has an email address set and the hold reminder email selected.
3. Set the hold to waiting by checking it and confirming the hold.
4. In Tools->Calendar add a holiday to yesterday and a seperate holiday to today.
5. On the command line (the date should be set to 5 days ago for my commands to work and any dates adjusted accordingly)
koha-mysql kohadev
update reserves set waitingdate="2025-09-25"
exit
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays --date 2025-09-29
6. Observe that 1 notice would have sent to your patron
7. On the command line
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays
8. Observe that 1 notice would have sent to your patron
9. On the command line
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays --date 2025-10-01
10. Observe that 1 notice would have sent to your patron
11. Apply patches
12. Repeat the last command from step 5 and step 7
13. Observe that no notice should be sent for your patron
14. Repeat step 7
15. Observe that a notice should be sent for your patron.
16. Run
perl misc/cronjobs/holds/holds_reminder.pl -days 5 -t -v
17. Confirm the notice should be sent for your patron
18. Sign off
Signed-off-by: Katie Bliss <KEBliss@dmpl.org>
Bug 35700: (QA follow-up) fix POD and $today
This patch replaces the inncorrectly removed POD note and replaces $today with the more correct $date_to_run for checking if we should be skipping sending the message on the day.
Amended-by: Jonathan Druart
Squashed and adjusted commit title
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit b895eb9b7d)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit 00ee0cc163)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Executing marc_ordering_process.pl will fail to run with the following errors:
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 21: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 23: marc_ordering_process.pl: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 25: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: --confirm]: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: --verbose]: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: ./marc_ordering_process.pl: No such file or directory
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 29: or,: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 31: 0: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 33: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 35: This: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 36: If: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 38: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 40: =over: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 42: syntax error near unexpected token `newline'
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 42: `=item B<-v|--verbose>'
This is due to an inadvertent newline before the shebang of the script.
Test Plan:
1) Run ./misc/cronjobs/marc_ordering_process.pl
2) Note the error
3) Apply this patch
4) Run ./misc/cronjobs/marc_ordering_process.pl
5) No error!
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit a067423f3f)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit edec2009dc)
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
The es_indexer_daemon.pl was missing handling for the case where
Koha::BackgroundJob->connect() returns undef without throwing
an exception (which happens when connection fails silently).
This adds the missing warning for that case, matching the behavior
in background_jobs_worker.pl.
Test plan:
1. Set JobsNotificationMethod to 'STOMP'
2. Configure invalid STOMP settings that cause connect() to return undef
3. Run es_indexer_daemon.pl - should show connection warning
4. Verify jobs are still processed via database polling
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 9ca0e183b5)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit f39c370980)
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
The background job workers are showing STOMP/RabbitMQ connection error
warnings even when the JobsNotificationMethod system preference was set
to 'polling', which doesn't require a message broker connection.
This patch adds checks for the JobsNotificationMethod preference before
displaying connection warnings, so warnings only appear when STOMP is
actually expected to be available.
Changes:
- misc/workers/background_jobs_worker.pl: Check syspref before warning
- misc/workers/es_indexer_daemon.pl: Check syspref before warning
Test plan:
1. Set JobsNotificationMethod to 'polling'
2. Run background_jobs_worker.pl - should not show STOMP warnings
3. Run es_indexer_daemon.pl - should not show STOMP warnings
4. Set JobsNotificationMethod to 'STOMP' without RabbitMQ running
5. Run workers - should show appropriate STOMP connection warnings
6. Sign off :-D
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 4ff1f4ca9c)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit c9f4566aa6)
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Bug 27486 renamed system preference "delimiter" into "CSVDelimiter"
This patch just replaces the commented code with a TODO text
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 269ad710b2)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit e744f001ae)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This change checks for HTTP(S) URLs and skips the shell when using
system() so that shell metacharacters aren't interpreted.
Test plan:
0. Apply the patch
1. Insert the following into the Babeltheque_url_update system preference:
$(whoami > /tmp/rce.log)
2. koha-shell kohadev
3. perl misc/cronjobs/social_data/update_social_data.pl
4. Note the script says the following:
$(whoami > /tmp/rce.log) is not a HTTP URL. Aborting.
5. Note also that /tmp/rce.log is not created
6. Change Babeltheque_url_update syspref to something like the following:
http://localhost:8080
7. perl misc/cronjobs/social_data/update_social_data.pl
8. Note that the wget succeeds but the bunzip2 fails (as expected) since
we haven't fetched a bzipped file
9. Try to create a payload that passes the HTTP URL test with shell
metacharacters and note that they're not interpetted and instead
just used as part of a URL string.
Signed-off-by: Jake Deery <jake.deery@openfifth.co.uk>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 7bb61b3114be14d2d18d03f855385d863f0bac60)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch fixes an identified RCE in the update_social_data.pl
social_data, which when executed, allows an attacker to run arbitrary
code on the system's shell.
TO TEST:
a) In the Babeltheque_url_update syspref, wrap any bash in $(), and then
run the update_social_data.pl script.
*) For example, $(whoami >
/tmp/rce.log) will output to a file the username the perl script is
running as.
APPLY PATCH
b) Run the update_social_data.pl script again. The string is now
meta-escaped, and the RCE no longer works.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit a26836bac54183c82fd5934e07c3c7306012998a)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly
rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job.
Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search
so we don't fetch any to pseudonymize that should not be.
To test:
1 - Generate some entries in statistics (checkin, checkout, renew, etc)
2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings)
3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
4 - Statistics pseudonymized
5 - Check the background jobs table, note jobs were generated
6 - Apply patch
7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
8 - You get same count as before, and are warned of existing pseudonymized transactions
9 - enter 'Y' to continue
10 - Check the pseudonymized_transacations table
11 - Confirm same rows were generated twice
12 - Check the background jobs
13 - Confirm there were no new jobs generated (for pseudonymization)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 342c1d5993)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
It will be a commented line in the debian template and added also
in the crontab.example file.
As mentioned on BZ, the comment can easily be removed automatically
during Docker builds or startups etc.
Test plan:
Just read the patch. Nothing changes here actually.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 2d93ec69bd)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The translation script explodes without meaningful error when
"incorrect" structure is found (edit)
Found when working on bug 38714, the translation script can break with
an error that does not directly point to the problematic string
[13:00:23] Error: Command failed: misc/translator/xgettext.pl --charset=UTF-8 -F -o /tmp/koha-2rwDEm/Koha-opac-bootstrap.pot -f /tmp/koha-2rwDEm/files
Can't use an undefined value as a HASH reference at misc/translator/xgettext.pl line 131.
at ChildProcess.exithandler (node:child_process:422:12)
at ChildProcess.emit (node:events:517:28)
at ChildProcess.emit (node:domain:552:15)
at maybeClose (node:internal/child_process:1098:16)
at ChildProcess._handle.onexit (node:internal/child_process:303:5)
at Process.callbackTrampoline (node:internal/async_hooks:128:17)
[13:00:23] 'po:update' errored after 992 ms
Test plan:
0. Use main and confirm that `gulp po:update --lang LANG` returns
successfully
1. Apply the first patch
=> The error is
Can't use an undefined value as a HASH reference at misc/translator/xgettext.pl line 131, <$INPUT> line 80.
2. Apply this patch
=> The error is now:
Incorrect structure found at ./koha-tmpl/intranet-tmpl/prog/en/modules/admin/branches.tt:230: '<option value="*" ' at misc/translator/xgettext.pl line 124, <$INPUT> line 100.
Which is way more interesting to fix the problem as we know exactly
where the problem comes from
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 3a5d8d51ca)
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
This patch makes the script provide links to the staff interface instead
of the OPAC. This way, the links can just be picked to access the
problematic records for fixing them.
To test:
1. Run:
$ ktd --shell
k$ perl misc/maintenance/compare_es_to_db.pl
=> FAIL: Generated links point to the OPAC
2. Apply this patch
3. Repeat 1
=> SUCCESS: Generated links are useful as they point to the staff
interface
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit cc0aa072dd)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
->connect does not raise an exception, so we didn't reached the catch
block.
Adjusting the code but I am not really happy with it, but more changes
will derail the original bug
Now we log:
Cannot connect to broker (Access refused for user 'guest') at /kohadevbox/koha/Koha/BackgroundJob.pm line 96.
Cannot connect to the message broker, the jobs will be processed anyway at /kohadevbox/koha/misc/workers/background_jobs_worker.pl line 100.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 8dfb19a45a)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This change adds error handling for ERROR frames to background_jobs_worker.pl.
A previous patch added it to es_indexer_daemon.pl which is a different
background jobs worker.
To test ERROR handling for background_jobs_worker.pl:
1. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
<password>bad</password>
</message_broker>
2. sudo koha-worker --restart --quiet kohadev
3. View the errors in the logs:
tail -f /var/log/koha/kohadev/worker-output.log
tail -f /var/log/koha/kohadev/worker-error.log
To test MESSAGE handling for background_jobs_worker.pl:
1. Ensure RabbitMQ connection details are correct
2. Go to Batch item modification and add the following barcodes:
39999000001310
39999000004571
3. Add a z - Public Note of the following and click Save:
THEBACKGROUNDWORKERWORKED
4. Search the catalogue using THEBACKGROUNDWORKERWORKED in the top
catalogue search
5. Note that you get 2 results in your search
6. Look in the logs just to make sure there are no new messages there
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 4dc4a232c4)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This change causes es_indexer_daemon.pl to die on ERROR frames instead
of spinning in an infinite loop. It also adds a content-type STOMP frame header
to help consumers know what data type is in the MESSAGE frame body.
To test ERROR handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch
2. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
<password>bad</password>
</message_broker>
3. sudo koha-es-indexer --restart kohadev
4. View the errors in the logs:
tail -f /var/log/koha/kohadev/es-indexer-output.log
tail -f /var/log/koha/kohadev/es-indexer-error.log
To test MESSAGE handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch (and ensure RabbitMQ connection details are correct)
2. Change the title of a bib record to something new and unique
3. Try to search for the bib using the new title
4. Note you can find it
5. Look in the logs just to make sure there are no messages there
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 2c1314ead6)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.
The only change is that it will now pass a LIMIT <batch_size> to the
query.
To test:
0. Have KTD started with ES (skip if you have an instance running):
$ ktd --es7 up -d
1. Stop the indexer and rabbit:
$ ktd --shell
k$ sudo koha-es-indexer --stop kohadev
k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D
[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 20b600d450)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
From https://css-tricks.com/snippets/css/make-pre-text-wrap/
/* Browser specific (not valid) styles to make preformatted text wrap */
pre {
white-space: pre-wrap; /* css-3 */
white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
}
... I think it's now safe to remove the backward compatibility versions and simply keep pre-wrap.
Test plan (from bug 921)0:
* place a hold on a title with a very very long title or author
* mark it as "waiting for pickup" by doing a checkin
* go to command line interface, and run
* misc/cronjobs/gather_print_notices.pl /tmp
* misc/cronjobs/printoverdues.sh /tmp
* if your title/author is long enough, it's splitted on 2 lines
with this patch (it is not before this patch)
It will also remove the warnings from t/db_dependent/Patron/Borrower_Discharge.t
WARNING: Ignored `white-space: -moz-pre-wrap` at 6:14, invalid value.
WARNING: Ignored `white-space: -o-pre-wrap` at 7:14, invalid value.
WARNING: Ignored `word-wrap: break-work` at 8:14, invalid value.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 03cf21fd92)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit 7f05cfc408)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 8f90ac9d4e)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit 3042fa93d8)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
For creating this patch I looked at all parameters defined in the
handler mapping, removed duplicated ones and tracked down which ones
really were parameters in `GetOptions`. This included tracking down
variables that were later assigned other values and things like that.
The mapping is used to display the right parameter name if missing a
mandatory one.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 46cc88acc3)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit ec396bfdd4)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Bug 31143 created a script in cronjob and then moved it to maintenance. However, it is still referred as in cronjob in help strings. This patch updates the hint.
No test needed I guess. Only changing a string
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 34886a9b5b)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit 5ffdf17098)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch corrects a few spelling mistakes and missing grammer and adds
a little more detail to the '--date', '--email' and '--frombranch'
options.
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit f6fd9f2e90)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
(cherry picked from commit b54a6574f2)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
If using ES, the script will collect the newly created records and biblionumbers in atwo arrays (@search_engine_record, @search_engine_record_ids) and pass that to the indexer when $commitnum records are read. The arrays are cleaned afterwards.
BUT: The records and biblionumbers were always stored, even if we're not using Elasticsearch. And as the cleaning of those arrays only happens if we are in fact using ES, all the data is added but never cleaned.
This patch fixes this by only storing record and biblionumber if we have an $indexer (i.e. if we're using Elasticsearch).
I have no idea though how/if the records are indexed when using Zebra.
After applying this patch and importing a large enough file without using Elasticsearch, the RAM usage also stays more or less constant.
@Jan, please verify!
Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 132593c8a8)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
So, I reviewed the code, which has a bunch of issues:
* there's a call to `next RECORD`, but no `RECORD` loop label.
* $record_number is incremented twice
* and the calls to txn_begin is moved from the outside the loop into each record, starting one transaction per record. Not sure how mysql handles this, but postgres would not like it.
This patch fixes those issues:
After I fixed the first two issues, I ran the patched script on a file containing 75 authorities (from GND), and not one was added to the DB, presuambly because nested transactions don't work in mysql?
So I also moved the BEGIN transaction outside of the main loop and started a new one after each commit, and added a final commit after to loop.
Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 01d36c5e8b)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Include checking for invalid marc data in main loop to avoid
loading all marc data into memory at once.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit ba0979a4c9)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Fridolin Somers
Baptiste Wojtkowski
Jonathan Druart
Jacob O'Mara
Tomás Cohen Arazi
Katrin Fischer
Marcel de Rooy
Lisette Scheer
Kyle M Hall
Nick Clemens
David Cook
Jake Deery
Paul Derscheid
Martin Renvoize
Thomas Klausner
David Gustafsson