Commit Graph

3146 Commits

Author SHA1 Message Date
Jonathan Druart 4a7a8fbef1
Bug 38664: The tidy script - fix race condition
There was a race condition, edit_file trying to open it while it was not
fully written by prettier (?) This is a dirty fix but it seems to do the
trick.

The error was:
edit_file 'koha-tmpl/intranet-tmpl/prog/en/includes/cat-search.inc' - write_file: No such file or directory at misc/devel/tidy.pl line 149.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-11 14:58:13 +01:00
Jonathan Druart 0cdaf9935f
Bug 38664: The tidy script - tidy tt twice
For an unknown reason the script is not idempotent (!)
Decided to run it twice for now, and have a look at this problem later.
It might be related to the length of the line

Version 0

<div>
    <div class="page-section clearfix">
        <div class="listgroup">
            <h4>
                <span>Statuses</span>
                [% IF foo %]
                    [% IF ( ITEM_DAT.withdrawn ) %]<span>Withdrawn</span>[% END %]
                    [% IF ITEM_DAT.bookable == 1 %] Yes [% ELSIF ITEM_DAT.bookable == 0 %] No [% ELSE %] Follow item type [% END %]

                    [% IF ITEM_DAT.bookable == 1 %] [% t('Yes') %] [% ELSIF ITEM_DAT.bookable == 0 %] [% t('No') %] [% ELSE %] [% t('Follow item type') %] [% END %]
                [% END %]
            </h4>
        </div>
    </div>
</div>

Version 1
<div>
    <div class="page-section clearfix">
        <div class="listgroup">
            <h4>
                <span>Statuses</span>
                [% IF foo %]
                    [% IF ( ITEM_DAT.withdrawn ) %]<span
                        >Withdrawn</span
                    >[% END %]
                    [% IF ITEM_DAT.bookable == 1 %]Yes[% ELSIF ITEM_DAT.bookable == 0 %]No[% ELSE %]Follow
                    item type[% END %]

                    [% IF ITEM_DAT.bookable == 1 %][% t('Yes') %][% ELSIF ITEM_DAT.bookable == 0 %][% t('No') %][% ELSE %][% t('Follow item type') %][% END %]
                [% END %]
            </h4>
        </div>
    </div>
</div>

Version 2
div>
    <div class="page-section clearfix">
        <div class="listgroup">
            <h4>
                <span>Statuses</span>
                [% IF foo %]
                    [% IF ( ITEM_DAT.withdrawn ) %]
                        <span>Withdrawn</span>
                    [% END %]
                    [% IF ITEM_DAT.bookable == 1 %]
                        Yes
                    [% ELSIF ITEM_DAT.bookable == 0 %]
                        No
                    [% ELSE %]
                        Follow item type
                    [% END %]

                    [% IF ITEM_DAT.bookable == 1 %][% t('Yes') %][% ELSIF ITEM_DAT.bookable == 0 %][% t('No') %][% ELSE %][% t('Follow item type') %][% END %]
                [% END %]
            </h4>
        </div>
    </div>
</div>

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-11 14:58:12 +01:00
Jonathan Druart 2c6f9d31b6
Bug 38664: The tidy script
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-11 14:58:12 +01:00
Jonathan Druart a9c0a70f67
Bug 38664: The auto-rebase script
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-11 14:58:11 +01:00
Matt Blenkinsop 3d3ccd4624
Bug 30300: (QA follow-up) Move logic to queue_notice
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-03 14:26:54 +01:00
Matt Blenkinsop ae68d05b59
Bug 30300: Update the cronjob to respect patron messaging preferences
Test plan:
1) Apply patch and run updatedatabase
2) In system preferences set MembershipExpiryDaysNotice to 5
3) Choose a patron and edit their expiry date to today's date
4) Ensure that patron has a primary email address
5) Run perl misc/cronjobs/membership_expiry.pl -c -v -before 10 -after 10
6) Check the patron's messages, there should be no expiry notice queued
7) Edit the patron's messaging preferences and select the checkbox for email in the row for 'Patron expiry'
8) Repeat step 5
9) There should be a patron expiry message in the patron's message queue
10) Sign off!

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-03 14:26:54 +01:00
Nicolas Hunstein d8c5e63ec7
Bug 31270: Terminology - use "self-registration"
Standardize on the use of "self-registration" (and variations)
for consistency in the OPAC and staff interface, instead of
"self registration".

Test plan:
1. Check the codebase for occurences of "self reg":
   . git grep "self reg"
   . git grep "selfreg"
   . git grep "Self reg"
   . git grep "Selfreg"
2. Note the occurences, excluding code comments, previous
   database updates, and previous release notes.
3. Search the system preferences for "self reg" and "selfreg".
4. Note the occurences in the results for the system preference
   descriptions (excluding system preference names) for:
   - PatronSelfRegistrationDefaultCategory
   - PatronSelfRegistrationEmailMustBeUnique
   - PatronSelfRegistrationLibraryList
   - PatronSelfRegistrationPrefillForm
5. Note the system preferences category heading for the OPAC
   - "Self registration and modification".
6. Apply the patch.
7. Repeat steps 1 to 5. There should now be no occurences
   (excluding code comments, previous database updates, and
   previous release notes).

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-31 15:57:36 +01:00
Jonathan Druart 3a5d8d51ca
Bug 38900: Provide info about the problematic strings
The translation script explodes without meaningful error when
"incorrect" structure is found (edit)

Found when working on bug 38714, the translation script can break with
an error that does not directly point to the problematic string

[13:00:23] Error: Command failed: misc/translator/xgettext.pl --charset=UTF-8 -F -o /tmp/koha-2rwDEm/Koha-opac-bootstrap.pot -f /tmp/koha-2rwDEm/files
Can't use an undefined value as a HASH reference at misc/translator/xgettext.pl line 131.

    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:517:28)
    at ChildProcess.emit (node:domain:552:15)
    at maybeClose (node:internal/child_process:1098:16)
    at ChildProcess._handle.onexit (node:internal/child_process:303:5)
    at Process.callbackTrampoline (node:internal/async_hooks:128:17)
[13:00:23] 'po:update' errored after 992 ms

Test plan:
0. Use main and confirm that `gulp po:update --lang LANG` returns
   successfully
1. Apply the first patch
=> The error is
Can't use an undefined value as a HASH reference at misc/translator/xgettext.pl line 131, <$INPUT> line 80.
2. Apply this patch
=> The error is now:
Incorrect structure found at ./koha-tmpl/intranet-tmpl/prog/en/modules/admin/branches.tt:230: '<option value="*" ' at misc/translator/xgettext.pl line 124, <$INPUT> line 100.

Which is way more interesting to fix the problem as we know exactly
where the problem comes from

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-30 18:55:20 +01:00
Jonathan Druart c26a989c64
Bug 38104: Always update when hea cronjob is run
If the cronjob is configured to run every month we don't need to add
another test on the delay in the script itself.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-24 16:14:31 +01:00
phette23 97c2437bb3
Bug 29238: Cataloging cron jobs are not logged
Testing plan:

1. set CronjobLog to "Log"
2. Run the three cron jobs
   `perl misc/link_bibs_to_authorities.pl; perl misc/cronjobs/merge_authorities.pl -b; perl misc/migration_tools/remove_unused_authorities.pl --confirm`
3. Visit the log viewer /cgi-bin/koha/tools/viewlog.pl, select the "Cron jobs" module, & note it's empty
4. apply the patch
5. Repeat steps 2 & 3 but note the cron logs aren't empty

squash

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-24 16:14:30 +01:00
Andrew Fuerste Henry 562dd9e2c9
Bug 38894: Use HomeOrHoldingBranch with longoverdue cronjob
To test:
1: Add 4 items, 2 with homebranch = Library A and 2 with homebranch = Library B
2: Logged into Library A, check out 1 Library A item and 1 Library B item, both with due dates in the past enough for the longoverdue cron
3: Set yourself to Library B, check out 1 Library A item and 1 Library B item, both with due dates in the past enough for the longoverdue cron
4: Run longoverdue cron with --verbose, without --confirm, and without --Library. see that it would act on all 4 items
5: Set CircControl to "the library the item is from"
6: Set HomeOrHoldingBranch to "the item's home library"
7: Run longoverdue cron with --verbose and --library="A" (where A is the code for your Library A), wihout --confirm
8: See the cron wants to act on the items with homebranch = A
9: Set HomeOrHoldingBranch to "the item's holding library"
10: Run longoverdue cron with --verbose and --library="A" (where A is the code for your Library A), wihout --confirm
11: See the cron is still trying to act on the two items with homebranch = A
12: Apply patch, restart all
13: Run longoverdue cron with --verbose and --library="A" (where A is the code for your Library A), wihout --confirm
14: See the cron is now trying to act on the two items with holdingbranch = A
15: Set HomeOrHoldingBranch to "the item's home library"
16: Run longoverdue cron with --verbose and --library="A" (where A is the code for your Library A), wihout --confirm
17: See the cron wants to act on the items with homebranch = A

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-24 12:02:14 +01:00
Kyle M Hall 576621a6df
Bug 37448: Add script to allow developers to quickly generate large amounts of data for development and testing
Test Plan:
1) Apply this patch
2) Run: misc/devel/create_test_data.pl -n 99 -s Borrower -d surname=Hall  -d zipcode=111111
3) Search patrons' for the name "Hall"
4) Note there are 99 Hall's in your results!

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Bug 37448: Make use of build_sample_biblio and build_sample_item

The script is brilliant, but for biblios and items we should make use of build_sample_biblio and build_sample_item or this data does not get indexed + linked tables rows get missed

To test, before this patch, run:
1) misc/devel/create_test_data.pl -n 5 -s Biblio -d title=Test
  Notice the 'Test' biblio is created on the database, but doesnt show on searches, and accessing it directly through URL throws a 500 error (because metadata does not exist for the biblio)
2) Apply this patch. Repeat the step above. Notice it now shows on searches and visiting the biblio directly shows no errors

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 37448: (QA follow-up) Tidy script

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-10 19:19:26 +01:00
Martin Renvoize 9e894ee790
Bug 38857: (QA follow-up) Fix perltidy errors
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-10 19:19:23 +01:00
Nick Clemens ed04d5b0fa
Bug 38857: Update cronjobs to log run before processing parameters
A few notes:
- I grabbed this list by looking for things in misc/ that had cronlogaction
- I didn't touch scripts that aren't in the cronjobs directory, except
  fix_invalid_dates where I follow import_patrons and only log if confirmed
  since they aren't crons, I figure this seems reasonable
- purge_suggestions I moved the logggin up and changed the addition of
  effective days to a verbose message - this seems more consistent

To test:
 1 - Enable CronjobLog
 2 - On command line:
     perl misc/cronjobs/cleanup_database.pl --old_reservers 550
 3 - Error on command line
 4 - In staff client, go to Tools->Log viewer, check 'Cronjobs' and submit
 5 - No entries
 6 - Apply patch
 7 - Repeat and see run is logged with wrong parameter
 8 - Fix parameter and confirm
 9 - Spot check other files until you are satisfied
10 - Sign it off!

Signed-off-by: William Lavoie <william.lavoie@inLibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-10 19:19:23 +01:00
Kyle M Hall 7fb78e4a5d
Bug 22415: (QA follow-up) Fix misc qa issues
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-10 19:19:14 +01:00
Martin Renvoize 7f6c186831
Bug 22415: Update EDI to use Koha::Logger
This patch updates the EDI code to use the Koha::Logger wrapper in
preference to the bare Log::Log4perl module.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-01-10 19:19:13 +01:00
Pedro Amorim 7834d25fb4
Bug 36039: (QA follow-up): Improve audit messages
Improve messages clarity.
Show success message if no schema diff was found from audit.
Code refactor to ensure the title+message shown in UI is the same as the one shown in CLI and are all translatable.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-27 15:19:05 +01:00
Martin Renvoize 534fba2691
Bug 36039: (follow-up) Move view logic to script/template
This patch updates the module as suggested by David to handle just the
diff code and leaves the display logic up to the templates and script.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: JesseM <jesse@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-27 15:19:04 +01:00
Pedro Amorim 09bb85409d
Bug 36039: Move audit_database script logic to a dedicated module
Test plan #1 ensure same script functionality (same as bug 34064):
0. Apply patch
1. vi ./installer/data/mysql/kohastructure.sql
2. Comment out some columns, change NULL status, or whatever you like
3. perl misc/maintenance/audit_database.pl \
    --filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql
4. Note that the output includes SQL commands to change the database
to match the new kohastructure.sql

5a. Try using koha-foreach and note that the database name appears above
the database comparison
5b. koha-foreach "perl misc/maintenance/audit_database.pl \
    --filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql"

Test plan #2:
1. After doing test plan #1, access the new 'database audit' tab, visit:
<staff_url>/cgi-bin/koha/about.pl?tab=database
2. Ensure the output is the same as test plan #1

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: JesseM <jesse@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-27 15:19:03 +01:00
Tomas Cohen Arazi 18c3e26db4
Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-23 12:20:39 +01:00
Tomas Cohen Arazi f50978c256
Bug 38726: (follow-up) Some more {Mm}arc ordering cases
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-23 12:20:36 +01:00
Nick Clemens e758492d99
Bug 38382: Fresh connection when connexion CSRF token expires
This patch simply generates a new user agent when reauthenticating when the CSRF token
for the session has expired.

To test:
1 - Edit Koha/Token.pm, line 275
from:    { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS * 3600 ) },
to:      { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS ) },
2 - This will make your CSRF tokens expire after 8 seconds
3 - vim /etc/koha/sites/kohadev/connexion.cnf
4 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
5 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
6 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
7 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
8 - Successful request
9 - Wait 10 seconds (more than 8 anyways)
10 - Repeat 7
11 - Unsuccessful request - Invalid CSRF token
12 - Ctrl+C to stop the connexion daemon
13 - Apply patch
14 - repeat 6
15 - repeat 7, success
16 - Wait more than 8 seconds
17 - repeat 7, success!
18 - repeat again until you are satisfied
19 - Sign off!

WNC amended and tidied

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-20 18:33:25 +01:00
Katrin Fischer 66e186e3f8
24.11.00: Add missing manual contributors to release notes
This is a late addition due to problems with the release notes
script at time of release.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-03 14:36:23 +01:00
Katrin Fischer 7b617ebceb
24.11.00: Add release notes
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-25 20:38:56 +01:00
Alex Buckley 438c62e8f1
Bug 36766: (follow-up) Set ReplytoDefault as the preferred from_address of sftp_file.pl
Default to KohaAdminEmailAddress if ReplytoDefault is not set

Test plan:
1. Enter a valid email address into ReplytoDefault and
   KohaAdminEmailAddress sysprefs

2. Apply patch and restart services

3. Generate a file on your Koha server. Place it in /tmp/test.mrc

4. Manually connect to the SFTP remote host and accept the hostkey

5. Enter Koha shell

6. Run the sftp_file.pl command-line script
./sftp_file.pl --host <your remote hostname> --user <your user> --pass <your password> --upload_dir <your upload directory> --file /tmp/test.mrc --email <your email>

7. Confirm the SFTP was successful

8. Confirm that the email that you received has a from address matching the
   ReplytoDefault syspref

9. Empty out the ReplytoDefault syspref and repeat steps 5 and 6

10. Confirm that the new email you received has a from address matching
    the KohaAdminEmailAddress syspref

Sponsored-by: Catalyst IT
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-22 10:22:37 +01:00
Matthias Meusburger 4bfb4da587
Bug 35659: (QA follow-up) Fix DateTime and cronlogaction messages
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-19 16:59:55 +01:00
Nick Clemens 1ba5e028cb
Bug 37790: Add skip indexing and holds queue options and verbosity to update localuse script
This patch skips record indexing and real time holds queue updates when updating the localuse field
from statistics. A note is added to the script that the user should reindex if the localuse field is mapped.

Additionally a verbose option is added to the script, and doubled use of GetOptions is removed.

Lastly, a check is added to confirm the items value is being changed before the value is stored.

To test:
 1 - Enable the real time holds queue
 2 - Enable Elasticsearch
 3 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm
 4 - Note all items are touched and reported
 5 - Check the background jobs table - there are many jobs generated
 6 - Apply patch
 7 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm
 8 - Note 0 items are reported updated, no new background jobs
 9 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm --verbose
10 - No items reported
11 - Update some items in the DB
     UPDATE items SET localuse = 99 WHERE itemnumber LIKE '%9';
12 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm
13 - Only the number of items changed above reported
14 - UPDATE items SET localuse = 99 WHERE itemnumber LIKE '%9';
15 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm --verbose
16 - Each item changed reported, and the amounts, and the total items updated.

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-18 17:04:32 +01:00
Wainui Witika-Park 769b394292
Bug 31143: (follow-up) Add description to fix_invalid_dates.pl
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-14 16:47:04 +01:00
Kyle M Hall 6b96763992
Bug 33339: Prevent Formula Injection (CSV Injection) in CSV files
The system is vulnerable to Formula Injection attacks as the data
stored within the database and exported as CSV/Excel is not being
sanitized or validated against implanted formula payloads

This patch modifies all uses of Text::CSV and derived classes to pass
the "formula" parameter with value of "empty" which replaces formulas
by empty string.

Test Plan:
1) Apply this patch
2) For guided_reports.pl, attempt to export CSV where you've set a column to a formula somehow
   ( such as "=1+3" )
3) Export that CSV file
4) Note the formula has not been exported
5) Repeat this plan for the remaining scripts that export CSV files
   where users can define the outputted data

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Fixed two conflicts. I have tested that this works as advertised on:
- Reports (Download > Comma separated text (.csv)) [Text::CSV::Encoded]
- Circulation > Overdues > Download file of all overdues [Text::CSV_XS]
- misc/export_borrowers.pl [Text::CSV]
This covers all modules used, and both GUI and command line.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Change none to empty in the commit message ! None is the default,
doing nothing. Empty clears the formulas.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-14 10:46:31 +01:00
Jan Kissig cfb841f9e6
Bug 23426: (follow-up) Enhance sip_cli_emulator.pl for test plan
This follow up enhances the sip_cli_emulator.pl to use start-item and/or end-item as params. With these new params the original test plan can be extended:

Test plan:
a) create several manual invoices for patron 23529000035676 : http://localhost:8081/cgi-bin/koha/members/maninvoice.pl?borrowernumber=19
b) run
     perl misc/sip_cli_emulator.pl -a 127.0.0.1 -p 6001 -su term1 -sp term1   -l CPL --patron 23529000035676 -m patron_information -s "   Y      " --start-item=1 --end-item=2
   to get fine 1 and 2 or
     perl misc/sip_cli_emulator.pl -a 127.0.0.1 -p 6001 -su term1 -sp term1   -l CPL --patron 23529000035676 -m patron_information -s "   Y      " --start-item=3 --end-item=3
   to retrieve fine 3

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-13 17:00:48 +01:00
Katrin Fischer 30bec5bb98
Bug 34355: (QA follow-up) Terminology and capitalization
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:22 +01:00
Nick Clemens fc97c91493
Bug 34355: (QA follow-up) Remove debugging comment from archiving command
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:20 +01:00
Matt Blenkinsop 3ee7c3ccbf
Bug 34355: (QA follow-up) Respect AcqCreateItem syspref
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:19 +01:00
Matt Blenkinsop 8c7418da95
Bug 34355: (QA follow-up) Add a delete flag to the cronjob
This patch adds a 'delete' flag to the cronjob to select whether a file should be deleted or archived once processed

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:16 +01:00
Matt Blenkinsop 6b85d06c19
Bug 34355: Add match_field and match_value to allow multiple accounts in the same file directory
Signed-off-by: Andrew Fuerste Henry <andrewfh@dubcolib.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:14 +01:00
Matt Blenkinsop 42ab7761f4
Bug 34355: Add cronjob and required methods
This patch adds a cronjob to take a marc file from the Koha file directory system and import it before creating order lines

Test plan:

This test plan involves running a cronjob, not sure how easy this is in the sandboxes

1) In system preferences, click Search and then select the Acquisitions option from the left hand menu
2) Enable MarcOrderingAutomation
3) Paste the following into MarcFieldsToOrder
price: 975$p
quantity: 975$q
budget_code: 975$h
4) Paste the following into MarcItemFieldsToOrder
homebranch: 949$a
holdingbranch: 949$b
itype: 949$y
nonpublic_note: 949$x
public_note: 949$z
loc: 949$c
ccode: 949$8
notforloan: 949$7
uri: 949$u
copyno: 949$t
price: 949$g
replacementprice: 949$v
itemcallnumber: 949$o
quantity: 949$k
budget_code: 949$l
Now save the sysprefs
5) In the administration homepage, under Acquisition parameters there should now be a link called "MARC order accounts"
6) Click on this and click the New account button
7) Choose a vendor and a budget
8) Enter a description
9) Enter the following in download directory - "/var/lib/koha/kohadev/tmp/koha_kohadev_upload"
10) Select any record matching settings you want (not important for the test plan, this uses the same code as the marc file staging that already exists in Koha)
11) Check Yes under Check for embedded item record data
12) Click submit
13) We now need to upload a MARC file into the directory that we put in the account. The easiest way to do this is through the marc staging tool
14) Navigate to Cataloging > Stage MARC records for import
15) Upload the file attached to this bug using the Choose file button
16) Click Upload file.
17) Leave the page without completing the new form that appears, we don't want to stage this file. The file we uploaded is now in the directory ready for the cronjob to process as if it had been transferred via SFTP.
18) In the CLI run, perl misc/cronjobs/marc_ordering_process.pl -v -c
19) The logging should show that the file has been processed for that account
20) Navigate to Acquisitions
21) Search for the Vendor that you put in the MARC ordering account
22) There should be a basket created on the vendor with the name of the file you uploaded (there will also be a reference at the start e.g. ea2323966faef2c59a6056dbd1f7d658_filename.marcxml. This is because we used the marc staging tool to upload the file and that adds a reference to filenames. This will not appear when using files from an SFTP location).
23) Click in the basket
24) There should be three orderlines, complete with accounting details that have been mapped from the file.
One line should have a fund of Fund 1_2, quantity of 4 and a replacement price of 6.75, one should have Secondary fund, 2, 6.40 and one should have Main fund,2, 6.30

Signed-off-by: Andrew Fuerste Henry <andrewfh@dubcolib.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:12 +01:00
Nick Clemens cfb72ca7fa
Bug 37478: Rename strict to skip_bad_records and add POD
Option name changed, POD updated.
The reason not to do this by default is that it is an extra record conversion
to check the validity, so may slow down the import.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-08 11:23:46 +01:00
Nick Clemens 0306281e41
Bug 37478: Add strict mode to bulkmarcimport
This patch adds a conversion from MARC -> XML -> MARC to catch any parsing errors.
If errors are found, we then lint the record to catch any problems, output the
warnings, and skip the record.

To test:
1 - Download the sample records from this bug report
2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v
    fails!
3 - perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v
    fails!
4 - Apply patch
5 - Repeat 2 & 3 - no change
6 - Add -st switch to the commands:
    perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v -st
    perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v -st
7 - The records are now skipped, and the script completes
8 - Confirm the warnings generated are useful
9 - Sign off!

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-08 11:23:45 +01:00
Nick Clemens 5e15ede505
Bug 37550: Move item check after error handling
To test:
1 - Grab the sample file on this bug report
2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=spaceditems.pl -v
3 - An exception for 'Duplicate ID' is thrown and script dies
4 - Apply patch
5 - run script again
6 - The script finishes, no items are added, but record is

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-07 17:32:38 +01:00
Jonathan Druart 6c48b39b17
Bug 31224: (follow-up) Update instances of metadata->record
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-07 16:29:25 +01:00
Martin Renvoize 2c52b27d8c
Bug 31224: Update instances of metadata->record
We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-07 16:29:23 +01:00
Kyle M Hall f6052fbb7c
Bug 37495: Add ability to use metadata to filter plugins to run for plugins_nightly.pl
It would be nice to be able to filter the plugins run by plugins
nightly for development and testing, and for added flexibility
as to when to run cronjob_nightly for different plugins if need be.

Test Plan:
1) Apply this patch
2) Check the metadata for an installed plugin, copy the name or another
   metadata value.
3) Run plugins_nightly.pl with a filter that does not match
   e.g. plugins_nightly.pl -m name="No plugin has this name"
4) Note no plugins cronjob methods are run
5) Run plugins_nightly.pl with a filter that *does* match
   e.g. plugins_nightly.pl -m name="Example Kitchen-Sink Plugin"
6) Not only the matching plugin runs!
7) Run plugins_nightly with no filter
   e.g. plugins_nightly.pl
8) Note all plugins with nightly cronjobs are run!

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-30 14:34:51 +01:00
Kyle M Hall 9c75f67560
Bug 37869: Tidy code
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-29 08:46:36 +01:00
Kyle M Hall 4a7c732f1f
Bug 37869: Move plugin hook before_send_messages to SendQueuedMessages
Most messages in Koha are handled by process_message_queue.pl. This script calls before_send_messages which calls the plugin hook before_send_messages. This works for nearly all messages, but some messages like WELCOME notices are triggered immediately and do not way to for process_message_queue.pl to run. We should move the triggering of before_send_messages into SendQueuedMessages so it is always handled.

Test Plan:
1) Install version 2.5.1 of the kitchen sink plugin
2) Run process_messsage_queue.pl and also trigger a WELCOME notice
3) Note the "Plugin hook before_send_message called with the params" message in the logs for the former but not the latter
4) Apply this patch
5) Repeat step 2, note there the message also shows in the logs for the WELCOME notice!

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-29 08:46:35 +01:00
Andreas Roussos 614392c6f0
Bug 35659: (follow-up) Use new column name in cron script
In an earlier patch, the `id` column of the `oai_servers`
table was renamed to `oai_server_id`. However, the code
in misc/cronjobs/harvest_oai.pl still uses the old column
name, which results in the following error:

$ misc/cronjobs/harvest_oai.pl --list
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Unknown column 'id' in 'order clause' at /kohadevbox/koha/Koha/Objects.pm line 403

This patch fixes that.

Test plan:

1) $ misc/cronjobs/harvest_oai.pl --list
   [...you get the DBI Exception...]
2) Apply the patch.
3) $ misc/cronjobs/harvest_oai.pl --list
   [...list of OAI repositories...]

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-28 14:01:32 +01:00
Matthias Meusburger d1b3ac5033
Bug 35659: (more QA follow-ups)
Rename Oaipmh to OAI
Rename oaiservers to oai_servers
Rename OaiServer to OAIServer
Rename oai_servers.id to oai_servers.oai_server_id
Rename ServerSearch to server_search
Use a callback function for logs
Use context for string "Set"
Split translation string

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-28 14:01:29 +01:00
Matthias Meusburger c5fe62a65f
Bug 35659: (QA follow-up)
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-28 14:01:29 +01:00
Matthias Meusburger d2367657b2
Bug 35659: OAI Harvester
Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-28 14:01:27 +01:00
Jonathan Druart 738c641bcc
Bug 38085: Pick multiple_sortable options for translation
Bug 29948 added a "multiple_sortable" type for sysprefs, to have the
options sortable. But the options were not picked for translation.

This patch fixes it.

Test plan:
Run `cd misc/translator && perl translate update LANG` before and after
this patch. Notice that with this patch you see the different options of
OPACAuthorIdentifiersAndInformation in the LANG-pref.po file (eg.
"Associated group (373$a$s$t$u$v$0)")
Translate the options
Install the translated version of the templates and confirm that the
strings appear translated on the interface.

Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-25 16:41:39 +02:00
Kyle M Hall d8fc079eee
Bug 28833: Speed up holds queue builder via parallel processing
The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-25 11:09:53 +02:00
Katrin Fischer 82638e85b5
Bug 23295: (QA follow-up) Terminology: borrowers => patrons
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-24 10:45:40 +02:00
Alex Buckley 3e984446ca
Bug 23295: Automatically restrict (debar) patrons when email/sms notices fail
When the 'RestrictPatronsWithFailedNotices' syspref is enabled then
patrons with email and sms notices which failed sending (have a
message_queue.status field of 'failed') have a restriction (debarment)
applied to them.

Test plan:
1. In the Koha staff client > Tools > Overdue notice/status triggers and
create the 'First' rule for all patron categories as:

Delay: 1
Letter: Overdue Notice
SMS: ticked

Ensure you have an SMS notice for the ODUE letter.

2. In the system preferences make sure you enter dummy data into the
SMSSendUsername, SMSSendPassword and EmailSMSSendDriverFromAddress
sysprefs

2. Find two non-debarred patrons and make sure they have invalid SMS numbers set. The SMS numbers must be INCORRECT, for example "123" as an SMS number. Leaving this field empty will result in the message_transport_type defaulting to 'print' instead of 'sms'.

3. Check one item out to each patron in step 2

4. Jump into the database and run the query:

UPDATE issues SET date_due=<2 days before current date> WHERE
borrowernumber=<borrower1>;

UPDATE issues SET date_due=<2 days before current date> WHERE
borrowernumber=<borrower2>;

5. Go to  misc/cronjobs directory and enter the Koha shell:

sudo koha-shell <instancename>

6. Run: ./overdue_notices.pl

7. Exit the shell and jump back into the database and run the query:

SELECT message_transport_type, status FROM message_queue WHERE
borrowernumber=<borrower1> OR borrowernumber=<borrower2>;

8. Confirm both new notice records have the message_transport_type is 'sms' and the status of 'pending'

9. Exit the database and re-enter the Koha shell and run the command:
./process_message_queue.pl

10. Jump back into the database re-run the query from step 7 and confirm
the status is 'failed' for both

11. Also run the query:
SELECT * FROM borrower_debarments WHERE borrowernumber=<borrower1> OR
borrowernumber=<borrower2>;

Notice there is no added debarment to these two patrons

12. Apply patch, restart memcached and plack. In the
installer/data/mysql directory enter the Koha shell and run the command:
./update_database.pl

13. In the Administration > Global System Preferences interface of the
staff client notice there is a new system (set to "Don't" by default)
named 'RestrictPatronsWithFailedNotices'.

Enable it (i.e. select 'Do')

14. Create a new file in the /etc/cron.daily directory named koha-custom and
add the following line to it:

koha-foreach --chdir --enabled
/usr/share/koha/bin/cronjobs/restrict_patrons_with_failed_notices.pl

15. In the  misc/cronjobs directory enter the Koha shell and run the command:
./restrict_patrons_with_failed_notices.pl

16. The script should output text saying: There are borrowers with
failed SMS or email notices

However because you haven't given the script the argument -c it won't
apply debarments (restrictions) to any of the patrons with the failed
SMS or email notices.

16. Query the borrower_debarments table:

SELECT * FROM borrower_debarments WHERE borrowernumber=<borrower1> OR
borrowernumber=<borrower2>;

Notice they still have no restriction

17. Now in the Koha shell run the command:
./restrict_patrons_with_failed_notices.pl -c

18. Notice the script outputs the text:
There are borrowers with failed SMS or email notices
Applying restriction to patron <borrowernumber>: <borrower firstname>
<borrower surname>;

19. Repeat step 16 and notice both patrons now have 1 restriction each
with the borrower_debarments.type=SUSPENSION and comment=SMSnumber
invalid and expiration=NULL

20. Query the borrowers table:
SELECT debarred, debarredcomment FROM borrowers WHERE
borrowernumber=<borrower1> OR borrowernumber=<borrower2>;

21. Notice the values are:
debarred= 9999-12-31
debarredcomment= SMS number invalid

22. Repeat step 17 and notice the script outputs:
There are borrowers with failed SMS or email notices

Patron <borrowernumber>: <borrower firstname> <borrower surname> is currently restricted due to having an invalid SMS number. No new restriction applied"

23. Repeat step 16 and notice no new debarment has been added to those
borrowers as they have already been restricted from having a failed SMS
notice.

24. In the Koha home directory run the command:
prove t/db_dependent/Koha/Notices.t

This unit test contains the tests for the new subroutines added to
Koha/Notice/Message.pm which are restrict_patron_when_notice_fails() and
get_failed_notices()

25. All tests should pass

26. Sign off

Sponsored-by: Catalyst IT
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-24 10:45:38 +02:00
Kyle M Hall e63608ea43
Bug 36766: (QA follow-up) Pass more variable to the template
Libraries may want to use this for more varied purposes and
adjust the language of the email based on the sftp upload
that succeeded or failed.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:24 +02:00
Kyle M Hall fe3fc2cf4a
Bug 36766: (QA follow-up) Add single dash paramter options
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:24 +02:00
Nick Clemens 4018c59bd7
Bug 36766: (QA follow-up) Tidy and use die_on_error over return
Tidy and add exec to atomic update
Fix POD

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:23 +02:00
Alex Buckley e4db67fe3a
Bug 36766: Only list the file basename as the SFTP put destination
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:22 +02:00
Alex Buckley 62569feec9
Bug 36766: Make SFTP notice more customisable
SFTP_FAILURE and SFTP_SUCCESSFUL notices can be configured from Notices
and slip tool.

Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:22 +02:00
Alex Buckley 2f508087f5
Bug 36766: Add a command-line utility to SFTP a file to a remote server
To test you need to have a remote server you are authorised to SFTP files to.

Test plan:
1. Apply patch and restart services

2. Generate a file on your Koha server. Place it in /tmp/test.mrc

3. Manually connect to the SFTP remote host and accept the hostkey

4. Enter Koha shell

5. Run the sftp_file.pl command-line script
./sftp_file.pl --host <your remote hostname> --user <your user> --pass <your password> --upload_dir <your upload directory> --file /tmp/test.mrc

6. Confirm the SFTP was successful

Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:21 +02:00
Nick Clemens a900e9307b
Bug 31143: (QA follow-up) Tidy
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:21 +02:00
Nick Clemens b598154fbf
Bug 31143: (QA follow-up) move script to maintenance
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:20 +02:00
wainuiwitikapark d4bfae0977
Bug 31143: Fix/identify all cases where '0000-00-00' may still remain in the database
This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL

Test plan -

Examples of setting up some 0000-00-00 date fields:

e.g. Borrowers table
update borrowers set dateexpiry="0000-00-00" where borrowernumber=49;

e.g. Items table
update items set datelastseen="0000-00-00" where itemnumber=12;

To identify the cases:
1) Set up some 0000-00-00 date fields
2) Run the misc/maintenance/search_for_data_inconsistencies.pl script
3) Run fix_invalid_dates.pl -v

To fix the cases:
1) Run fix_invalid_dates.pl -c -v
2) Notice the value has changed from 0000-00-00 to NULL

e.g. Borrowers table
select dateexpiry from borrowers where borrowernumber=49;

e.g. Items table
select datelastseen from items where itemnumber=12;

Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 20:02:20 +02:00
Magnus Enger 476ae27df0
Bug 37787: Remove mention of elastic_index from POD
After the introduction of misc/workers/es_indexer_daemon.pl,
misc/workers/background_jobs_worker.pl should not handle the
elastic_index queue. But the latter script still mentions this
queue in the POD. This patch removes that mention.

To test:
- Run perldoc misc/workers/background_jobs_worker.pl
- Verify "elastic_index" is mentioned under --queue
- Apply this patch
- Run perldoc again and verify the mention of "elastic_index"
  is gone

Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 19:14:55 +02:00
Nick Clemens 0ff61ab9d9
Bug 38237: Add POD
This patch adds POD to the script and rpelaces the custom help script with pod2usage

TO test:
perl misc/cronjobs/erm_run_harvester.pl
Confirm you get helpful usage data

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 19:14:54 +02:00
Nick Clemens 6ce5df7f8e
Bug 38237: Add logging to erm_run_harvester cronjob
This patch simply adds logging to the cronjob

To test:
1 - Enable CronjobLog
    http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=CronjobLog
2 - Enable ERMModule
    http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=ERMModule
3 - Home -> E-resource Management -> Data providers -> New data provider
    Fill from registry or create your own, but change the URL and put in any required numbers
    We don't need a real test, just to run the ob
4 - perl misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run --debug
5 - View the logs, no log of the script
6 - Apply patches
7 - Run the job again
8 - Check logs
9 - Success!

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 19:14:53 +02:00
Matthias Meusburger 2e2fba91bf
Bug 18273: Do not insert authority if there already are duplicates in koha
Test plan:

1) Create an authority with a unique value in a searchable field, for instance a PERSO_NAME authority with 'BZ18273' in the 100$a field.

2) Create a second authority with the same value in the same field.

3) Export this second authority as iso2709 (Save -> MARC (Unicode/UTF-8))

4) Run bulkmarcimport.pl with the following parameters:
perl bulkmarcimport.pl -a --file=/kohadevbox/koha/auth-xxxx.utf8 --match='Heading,100$a' -l /kohadevbox/koha/bulk.log

5) Check in the log file and/or through search that a third authority has been added (despite using the --match option)

6) Apply the patch

7) Delete the 3 authorities and repeat step 1 to 4.

8) Check that a third authority has not been added (--match worked)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-21 15:10:32 +02:00
Jonathan Druart c89e020e7d
Bug 37845: Remove C4::Members::DeleteExpiredOpacRegistrations
This patch remove the subroutine C4::Members::DeleteExpiredOpacRegistrations.
The code is moved and adjusted to Koha::Patrons.
We now have 2 new methods:
  Koha::Patrons->filter_by_expired_opac_registrations
  Koha::Patrons->filter_by_safe_to_delete

Test plan:
Run the misc/cronjobs/cleanup_database.pl script with the
--del-exp-selfreg (as well as --verbose and --confirm) and confirm that
it behaves as expected

Note that it improve the output of the verbose mode and now tell if the something
is wrong with the config (syspref not set)

Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-21 15:10:31 +02:00
Jonathan Druart 73f890ab2d
Bug 37844: Remove C4::Members::DeleteUnverifiedOpacRegistrations
This patch removes the subroutine
C4::Members::DeleteUnverifiedOpacRegistrations and use the
Koha::Objects->filter_by_last_update method that is used for the other
flag in this script.

Test plan:
Confirm that --del-unv-selfreg works the same before and after this
patch applied

Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-21 15:10:30 +02:00
Thomas Klausner e200988cb8
Bug 35466: (follow-up) document skip_indexing parameter to bulkmarciport.pl
Add documentation for the new -si, --skip_indexing parameter

Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-21 12:25:57 +02:00
Nick Clemens f522fbff32
Bug 35466: (follow-up) Go to next if error, add to indexing call if not
Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed
we were also indexing during the commity, however, it appears we were not. This patch ensures we push
the record and ids into the arrays for indexing during commit.

Additionally I add a skip to next record on error, to match biblio behaviour.
I also correct a log referring to biblios during authority importing

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-21 12:25:56 +02:00
Nick Clemens e01fb16455
Bug 35466: Add skip_indexing parameter to bulkmarcimport.pl
This patch adds a new optoin to skip indexing to bulkmarcimport.
It also fixes a bug where authorities were being indexed multiple times
during import.

To test:
 1 - Apply patch
 2 - Download the sample files on this bug
 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES)
 4 - Note the total (435 in KTD)
 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc
 6 - Search again, note increaed by 100
 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing
 8 - Search again, no increase
 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v
10 - Search again, increase, records were added but not initially indexed
11 - Browse to authorities and search as for biblios
12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc
13 - Search again, note increase
14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing
15 - Search again, no increase
16 - perl misc/search_tools/rebuild_elasticsearch.pl -v
17 - Search again, increase, records were added but not initially indexed
18 - Sign off! Hi5!

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-21 12:25:56 +02:00
Tomas Cohen Arazi 5102f708a5
Bug 38164: Skip fuzzy translations
This patch was envisioned by Julian Maurice.

Test plan:
0. Apply the 1st patch (the revert)
1. Go to `misc/translator/po`
2. es-ES-messages-js.po
3. Mark "Edit" string as fuzzy (around line 48). It should look like:

```
msgid "Edit"
msgstr "Editar"
```

4. Edit ./intranet-main.tt and add the following lines at the bottom,
   inside the `$(document).ready` block:

```javascript
  console.log(_("Edit"));
  console.log(__("Edit"));
```

5. Install the templates
  k$ koha-translate --install es-ES --dev kohadev && restart_all
6. Enable *es-ES* by searching for `language` in the sysprefs and switch
   to it for the staff interface.
7. Go to the Koha home page, open the browser console
=> FAIL: Notice that the second log in the console is displaying the
fuzzy string (i.e. is being translated when it shouldn't)
8. Apply this patch
9. Re-install the translated templates:
  k$ koha-translate --update es-ES --dev kohadev && restart_all
10. Repeat 7
=> SUCCESS: With this patch applied both logs show the English version of the
string.
11. Remove the fuzzy flag on `es-ES-messages-js.po`
12. Repeat 9 and 10
=> SUCCESS: The results are similar to step 7, but in this case they are
expected as the string is not marked fuzzy.
13. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-18 17:44:17 +02:00
Tomas Cohen Arazi b285fd57c8
Bug 38164: Revert "Bug 37303: Replace po2json with a JS version"
This reverts commit 4eb9816354.

Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-18 17:44:16 +02:00
Emily Lamancusa 4033699062
Bug 38156: Sort issues by borrowernumber before parallel chunking
When the automatic renewal cron job is using parallel processing, it
aims to process all of the renewals for any given patron together in one
chunk to avoid data conflicts. To accomplish this, it starts a new data
chunk each time it encounters a new patron. However, if a patron's
renewing checkouts aren't all consecutive in the database, that patron's
data ends up split across multiple chunks.

We need to sort the issues by borrowernumber before attempting to chunk
them in order to make sure they are chunked and processed correctly.

To test (using KTD default test data):
Setup:
1. Edit the default circulation rule:
    - Set Automatic renewal to "Yes"
    - Set No automatic renewal before to 3
2. Open the following patron accounts in separate tabs:
    - Floyd Delgado
    - Joyce Gaines
    - Edna Acosta
    - Mary Burton
3. Perform the following patron account edits for each of the above
   patrons (and keep the tabs open):
    - Enable automatic renewal notices, and set them to digests only
    - Add a value to the email field
4. Enter the kshell (ktd --shell)
5. Edit /etc/koha/sites/kohadev/koha-conf.xml, and add the following
   lines near the end, just above the </config> and </yazgfs> closing
   tags:
<auto_renew_cronjob>
    <parallel_loops_count>2</parallel_loops_count>
</auto_renew_cronjob>
6. restart_all

Reproducing the issue:
7. Apply the test patch only
8. Run perl generate_checkouts.pl to generate test data
9. perl misc/cronjobs/automatic_renewals.pl -v -c
--> The test patch added output that will show how the renewals were
    chunked into "chunk 0" and "chunk 1" for the two parallel loops.
    Note that the issues for each borrower are not processed nicely in
    one chunk, but are separated across multiple chunks and alternated
    with other borrowers.
10. Check the checkouts for each of the four patrons from above
--> All checkouts should have renewed
11. Check the notices tab for each of the four patrons
--> Notice errors in the automatic renewal digest notices. A patron's
    renewals may be split across multiple digests, a digest may be
    missing renewals, or a patron may not have received a digest at all

Testing the patch:
12. Apply the second patch
13. Reset the due dates on all checkouts so that they will all be
    eligible for automatic renewal again:
    - koha-mysql kohadev
    - UPDATE issues SET date_due=<two days from today>;
14. perl misc/cronjobs/automatic_renewals.pl -v -c
--> Note that the renewals are now correctly chunked by patron
15. Check the checkouts and notices tab for each of the four patrons
--> All checkouts should have renewed, and all patrons should have a
    single new Auto Renewals Digest notice that correctly lists all of
    their renewed items

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: tidied the code block inline (tcohen)
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-18 14:25:49 +02:00
Tomas Cohen Arazi 67e7c7f5a4
Bug 37791: (follow-up) Last occurence
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Olivier V <olivier.vezina@inLibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-11 12:21:30 +02:00
Nick Clemens 0546217cdc
Bug 37709: Die and notify if file cannot be opened
1 - perl misc/migration_tools/bulkmarcimport.pl -b --file=notafile.mrc -v
2 - Output:
    Characteristic MARC flavour: MARC21

    0 MARC records done in 0.00804400444030762 seconds
3 - Apply patch
4 - perl misc/migration_tools/bulkmarcimport.pl -b --file=notafile.mrc -v
5 - Output:
    Characteristic MARC flavour: MARC21
    Could not open notafile.mrc: No such file or directory at misc/migration_tools/bulkmarcimport.pl line 262.

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-11 11:13:51 +02:00
Fridolin Somers 996c9d9fdc
Bug 35442: Script migration_tools/build_oai_sets.pl add ORDER BY
Script migration_tools/build_oai_sets.pl is missing ORDER BY
biblionumber.
This is a problem when using OFFSET and LIMIT, the results may be
differenly orderder between two calls of this script.

Test plan :
1) Create a OAI SET with all records
2) Run migration_tools/build_oai_sets.pl with the first half of records
3) Run migration_tools/build_oai_sets.pl with the second half of records
4) Check all records are in the OAI set

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2024-09-30 15:24:21 +01:00
Nick Clemens d785fb1c1e
Bug 37775: Spelling and tidy
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2024-09-19 12:39:14 +01:00
Nick Clemens dba9dd4053
Bug 37775: Remove delayed commits from update_totalissues.pl and add progress option
This patch removes setting AutoCommit to 0 and commiting only every X records.
Instead we commit as we go and report progress using a parameter.

Bug 36474 reduced the numebr of changes that are being committed, so this should be a reasonable change. The
use of commits without transactions was causing problems if the library was active while the script ran.

To test:
1 - perl misc/cronjobs/update_totalissues.pl -c
2 - Script runs, but with unknown parameter
3 - perl misc/cronjobs/update_totalissues.pl -p 10
4 - Script runs and reports every 10 records
5 - per; misc/cronjobs/update_totalissues.pl
6 - Script runs and reports every 100 records by default

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2024-09-19 12:39:14 +01:00
Nick Clemens d7b36d7f69
Bug 36474: Don't update records when total issues has not changed
This patch adds a new check in UpdateTotalIssues to check that we are changing the number
of total issues before calling ModBiblio

To test:
0 - Enable CataloguingLog
1 - Checkout an item
2 - Run : misc/cronjobs/update_totalissues.pl --use-stats --commit=1000 -v
3 - In report, note all biblios were updated
4 - Check action_logs - note a new entry for every biblio
5 - Apply patch
6 - Repeat
7 - Note no biblios reported updated
8 - Note no new cataloguing log entries
9 - Checkout the item again
10 - Run again
11 - Note biblionumber has updated count in verbose output
12 - Note report only rpeort 1 biblio modified, the rest only processed
13 - Only one line added to action_logs
14 - Run it again
15 - Confirm no updates

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2024-09-19 12:39:14 +01:00
David Cook 5321e0fb09
Bug 37682: Lazy-load modules for setters in koha-preferences tool
This change lazy-loads modules for setters in the
koha-preferences tool, so that getters are free to run super fast.

Between BZ 37657 and BZ 37682 we effectively eliminate the
overhead of running "get" or "dump" commands via the koha-preferences
tool.

Test plan:

1. time misc/admin/koha-preferences get SearchEngine
2. Note time is about .35 seconds
3. time misc/admin/koha-preferences dump
4. Note time is about .35 seconds
5. Create sysprefs.yml

   ---
   marcflavour: MARC21
   viewMARC: 1

6. time misc/admin/koha-preferences load -i sysprefs.yml
7. Note time is about .35 seconds
8. time misc/admin/koha-preferences set SearchEngine Elasticsearch
9. Note time is about 1 seconds

10. Apply patch
11. Repeat the koha-preferences commands above
12. Note that the "dump" and "get" commands run in about .09-.1
seconds. The "load" and "set" commands still take the same amount
of time as their behaviours haven't changed

13. misc/admin/koha-preferences set SearchEngine Elasticsearch1
14. koha-mysql kohadev
15. select * from action_logs where module = 'SYSTEMPREFERENCE'
 order by action_id desc limit 5;
16. Note that the action log showing Elasticsearch1 update says
    "interface"
 of "commandline" and "script" of "koha-preferences"

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-27 12:14:23 +02:00
David Cook 673aaaaba2
Bug 37657: Make koha-preferences CLI tool faster
By using Koha::Database->dbh() to use a minimal database handle
which doesn't preload the whole DBIx::Class schema, we're able to
run the same command 2-3 times faster.

This is beneficial when running the tool in a loop which runs the
command serially one by one.

Test plan:

1. time misc/admin/koha-preferences get SearchEngine
2. Note time is about 1 second
3. time misc/admin/koha-preferences dump
4. Note time is about 1 second
5. Create sysprefs.yml

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-27 12:14:22 +02:00
Nick Clemens 8ef9ac5445
Bug 37552: Wrap auto renewal attempt in eval to ensure script does not die
When libraries have a lot of checkouts, or an AMH, checkins can happen while the cron is running.

This patch simply adds an eval around the auto renewal attempt in case of early check in or other errors.

You can verify cron completed by enabling cronjob log in system preferences and checking the action logs

To test:
1 - Add 'sleep(10);' to automatic_renewals.pl
2 - Set circulation rules to enable automatic renewals
3 - Issue an item to a patron
4 - perl misc/cronjobs/automatic_renewals.pl -v
5 - Confirm item would not be renewed
6 - perl misc/cronjobs/automatic_renewals.pl -v -c
7 - Quickly check in the item
8 - The cronjob dies
    DBIx::Class::Row::update(): Can't update Koha::Schema::Result::Issue=HASH(0x586e1a674fb0): row not found at /kohadevbox/koha/Koha/Object.pm line 172
9 - Apply patch
10 - Checkout the item again
11 - perl misc/cronjobs/automatic_renewals.pl -v -c
12 - Quickly checkin the item
13 - You get a warning, but the cron completes

Signed-off-by: CJ Lynce <cj.lynce@westlakelibrary.org>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-16 17:33:49 +02:00
Nick Clemens cf315751cf
Bug 37543: (follow-up) Tidy
Tidy the whole thing

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-16 17:33:43 +02:00
Rudolf Byker 0e9ea3c9f5
Bug 37543: Use CSRF tokens in connexion_import_daemon.pl
Since version 24.05, due to the changes mentioned at
https://wiki.koha-community.org/wiki/Koha_/svc/_HTTP_API#Changes_coming_in_Koha_24.05 ,
the `connexion_import_daemon.pl` stopped working. The reason for this is that
it did not use CSRF tokens.

To test:
1. Get a Koha instance on 24.05, before applying the patch.
2. Create a plain text file somewhere on the server containing a raw MARC
   record (not XML). Let's call it `marc.txt`.
3. On the server, create a config file like this:
```
host: 0.0.0.0
port: 5500
koha: http://localhost:82  # Where 82 is the port of the Koha staff interface.
user: foo  # A Koha staff user.
password: Fooo1234  # The Koha staff user's password.
import_mode: stage
```
4. Run `./connexion_import_daemon.pl --config the-config-file-path`
5. In another terminal on the same server (or from anywhere that can reach the
   port opened by the `connexion_import_daemon.pl` script,
   run `nc localhost 5500 < marc.txt`
6. Observe in the stderr of the daemon script: `Response: Unsuccessful request`
7. Stop the daemon script.
8. Apply the patch and repeat steps 4 and 5.
9. Observe in the stderr of the daemon script:
   `Response: Success.  Batch number ... - biblio record number HASH(...) added to Koha`
10. Check at /cgi-bin/koha/tools/manage-marc-import.pl for a batch named
    `(webservice)`. It should contain one record now. This is how we know that
    authentication between the daemon and Koha worked, which is what this
    patch tries to address.

Thanks-to: David Cook <dcook@prosentient.com.au>
Sponsored-by: Reformational Study Centre <www.refstudycentre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-16 17:33:42 +02:00
Michael Hafen 88500ccef4
Bug 37613: (Follow-up) Change the option and documentation to match terminology guidelines
Trivial patch.

Change the --branch and --skip-branch options of the longoverdue cron script
to --library and --skip-library to meet the Terminology Guidelines.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-12 14:01:07 +02:00
Kyle M Hall 99c2064126
Bug 9596: (QA follow-up) Only find a given patron once
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-09 18:44:58 +02:00
Kyle M Hall bbd722be5b
Bug 9596: (QA follow-up) Add some clarifying comments
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-09 18:44:58 +02:00
Michael Hafen 8e4963d97c
Bug 9596: Add branch and skip-branch options to cronjobs/longoverdue.pl
This script doesn't seem to be included in cron files by default.
This change is to allow script parameters to effect only certain branches.
This allows the script to be added multiple times to a cron file with
  different settings for different branches.

Test plan:
1. apply patch
2. identify two books at different branches the same number of days overdue.
3. run the longoverdue.pl script specifying one of the branches in the
     --branch command line parameter.
     i.e. koha-shell -c 'misc/cronjobs/longoverdue.pl --branch branch_code --lost 60=2 --maxdays=61 --confirm' instance_name
4. observe that the book at the specified branch has been or would be affected
     by the script while the other book is not.

Signed-off-by: Tadeusz „tadzik” Sośnierz <tadeusz@sosnierz.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-09 18:44:57 +02:00
Kyle M Hall fa8048e63a
Bug 36770: (QA follow-up) Tidy export_records.pl
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-09 18:44:54 +02:00
Alex Buckley 1a32355eb3
Bug 36770: Export records using a report
Enable the export_records.pl script use a report output to export biblios or authorities

Test plan:
1. Apply patches and restart services
2. Create a SQL report (id=1)
    SELECT biblionumber
    FROM biblio
3. Create a SQL report (id=2) and set an item as notforloan = -1
    SELECT title, author, biblio.biblionumber
    FROM biblio
    LEFT JOIN items USING (biblionumber)
    WHERE items.notforloan = <<Not for loan|NOT_LOAN>>
4. Create a SQL report (id=3)
    SELECT title, author
    FROM biblio
5. Create a SQL report (id=4)
    SELECT authid
    FROM auth_header
6. Run export_records.pl using report id=1 and confirm a koha.mrc file is created in the misc directory:
cd misc
./export_records.pl --report_id=1

7. Delete the koha.mrc file

8. Run export_records.pl using report id=2
./export_records.pl --report_id=2

9. Notice you are prompted to supply a parameter

10. Re-run report id=2 supplying a parameter. Confirm the koha.mrc file is created and contains bib data
./export_records.pl --report_id=2 --report_param=-1

11. Run export_records.pl using report id=3
./export_records.pl --report_id=3

12. Notice you get the message: The --report_id you specified does not fetch a biblionumber

13. Delete the koha.mrc file

14. Run export_records.pl using report id=4
./export_records.pl --report_id=4

15. Notice you get a message 'The --report_id you specified does not fetch a biblionumber'

16. Re-run export_records.pl setting the record-type=auths
./export_records.pl --record-type=auths --report_id=4

17. Notice the koha.mrc file is generated and contains auth data

Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand

Signed-off-by: Alexandre Noel <alexandre.noel@inlibro.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-09 18:44:54 +02:00
David Cook 1a747ddcb7
Bug 37553: Tidy to make QA tools happy
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-02 18:59:59 +02:00
David Cook bf427a2947
Bug 37553: Consistently send Csrf-Token in request header
This change consistently sends the Csrf-Token in the request header.
Previously, one POST sent it in the request body, while the other POST
sent it in the request header. Since we're using an API, it's best
for us to always send it in the request header

Test plan:
0. Apply the patch
1. perl ./misc/migration_tools/koha-svc.pl \
	http://localhost:8081/cgi-bin/koha/svc koha koha 29 > bib-29.xml
2. perl ./misc/migration_tools/koha-svc.pl \
	http://localhost:8081/cgi-bin/koha/svc koha koha 29 bib-29.xml
3. Note that the following appears in STDOUT and there is no 403 error:
"update 29 from bib-29.xml"

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-02 18:59:59 +02:00
David Cook eabd792ee9
Bug 37553: Use CSRF token for authenicated session for POSTing
This change fixes the Koha::SVC to store the CSRF token for
the authenticated session for further POSTing.

Test plan:
0. Apply the patch
1. perl ./misc/migration_tools/koha-svc.pl \
    http://localhost:8081/cgi-bin/koha/svc koha koha 29 > bib-29.xml
2. perl ./misc/migration_tools/koha-svc.pl \
    http://localhost:8081/cgi-bin/koha/svc koha koha 29 bib-29.xml
3. Note that the following appears in STDOUT and there is no 403 error:
"update 29 from bib-29.xml"

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-02 18:59:58 +02:00
Jonathan Druart 4eb9816354
Bug 37303: Replace po2json with a JS version
When using __() (ie. Gettext.js) we are seeing the translations that are marked as fuzzy.
This is definitely not the expected behaviour.

It happens because (our version of) po2json are old and no longer maintained,
and just embed them.

It seems that the bin we have has been upgraded to a JS version
(different authors).

Test plan:
(replace LANG with your language code)
0. Do not apply this patch
Edit misc/translator/po/LANG-messages-js.po
Mark a string as fuzzy
Edit ./intranet-main.tt and add the following lines inside $(document).ready
  console.log(_("Your string"));
  console.log(__("Your string"));
Replace "Your string" with the string you are actually testing.

Update the templates: `koha-translate --update LANG --dev kohadev && restart_all`

Go to the Koha home page, open the console.

=> Notice that the second log in the console is displaying the fuzzy string.

1. Apply this patch
Install the new version of po2json using `yarn install`
Repeat the previous steps.

=> With this patch applied both logs show the English version of the
string.

Remove fuzzy, update the templates and try again.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2024-07-26 14:49:53 +01:00
Janusz Kaczmarek 771db8c399
Bug 37271: Recall status should be 'requested' in overdue_recalls.pl
In "Bug 19532: Make recalls.status an ENUM" one-letter statuses were
replaced by their verbal equivalents.  But in overdue_recalls.pl
remained "R".  As a result, recalls never overdue.

No test plan provided--obvious correction.

Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-07-12 10:21:29 +02:00
Nick Clemens 3024b560c4
Bug 29507: Tidy
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-07-05 15:48:11 +02:00
Nick Clemens 3d385d0e11
Bug 29507: Speed up auto renew cronjob via parallel processing
The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-07-05 15:48:11 +02:00
Tomas Cohen Arazi b54c854d24
Bug 36901: Add logging for uncaught exceptions in background job classes
This patch adds logging of unhandled exceptions that could occur. This
is happening on busy production sites right now. This is also useful for
plugin jobs that might not be 100% following the guidelines and would
benefit from this.

But as the [DO NOT PUSH] patch highlights, this is something we really
want to have on our current codebase, as a database connection drop
might make us reach that `catch` block we are adding logging to on this
patch.

To test:
1. Apply the [DO NOT PUSH] patch
2. Run:
   $ ktd --shell
  k$ restart_all ; tail -f /var/log/koha/kohadev/worker*.log
3. Pick a valid barcode on the staff UI
4. Use the 'Batch delete items' tool in the cataloguing section
5. Start the job for deleting the item
=> FAIL: The item got deleted, but the job marked as failed and no logs
about the reasons
6. Apply this patch and repeat 2-5
=> SUCCESS: Same scenario but there's a log with the error message
7. Sign off :-D

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-07-01 18:55:54 +02:00
David Cook 3e1d8b0628
Bug 32575: Add an empty text body to fix multipart/mixed handling
By adding an empty text body, we force Email::Stuffer/Email::MIME
to use multipart/mixed handling for the attachment instead of forcing
a single part (ie direct attachment) email, which is not consistently
handled by different email clients.

An empty text body is language-neutral (ie not imposing English),
and it allows SMTP servers to inject organisational footers into
the email (e.g. confidentiality notices).

Signed-off-by: Magnus Enger <magnus@libriotech.no>
I have tested this solution in production, and it works for me.

Bug 32575: Tidy patch

Signed-off-by: Magnus Enger <magnus@libriotech.no>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-06-28 13:49:20 +02:00
Nick Clemens 444f3d8bad
Bug 37181: Add --confirm option to pseudonymize_statistics.pl
To test:
1 - Perform some transactions
2 - Enable Pseudonymization
3 - perl misc/maintenance/pseudonymize_statistics.pl -v
4 - Confirm test run an nothing changed
5 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
6 - Confirm statistics are pseudonymized

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Amend: tidied the changes [tcohen]
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-06-27 14:04:54 +02:00