In an earlier patch, the `id` column of the `oai_servers`
table was renamed to `oai_server_id`. However, the code
in misc/cronjobs/harvest_oai.pl still uses the old column
name, which results in the following error:
$ misc/cronjobs/harvest_oai.pl --list
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Unknown column 'id' in 'order clause' at /kohadevbox/koha/Koha/Objects.pm line 403
This patch fixes that.
Test plan:
1) $ misc/cronjobs/harvest_oai.pl --list
[...you get the DBI Exception...]
2) Apply the patch.
3) $ misc/cronjobs/harvest_oai.pl --list
[...list of OAI repositories...]
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Rename Oaipmh to OAI
Rename oaiservers to oai_servers
Rename OaiServer to OAIServer
Rename oai_servers.id to oai_servers.oai_server_id
Rename ServerSearch to server_search
Use a callback function for logs
Use context for string "Set"
Split translation string
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- Run updatedatabase.pl
- Apply the DBIC update patch
- Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)
If the repository dataformat is in oai_dc, you can use the sample XSLT files:
For Unimarc:
OAIDCtoUNIMARCXML.xsl for bibliographic records
AuthOAIDCtoUNIMARCXML.xsl for authorities
For Marc21:
OAIDCtoMARC21XML.xsl for bibliographic records
AuthOAIDCtoMARC21XML.xsl for authorities
If the repository dataformat is in marc-xml, you can use
OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.
- Some repositories to test with:
Bibliographic records, oai_dc:
Endpoint: http://staroai.theses.fr/OAIHandler
Set: ddc:260
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
or
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl
Authorities, oai_dc:
Endpoint: https://www.idref.fr/OAI/oai.jsp
Set: a
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
or
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl
Authorities, marc-xml, Unimarc:
Endpoint: https://www.idref.fr/OAI/oai.jsp
Set: a
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl
- Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
Version 4.xx is needed.
See Bug 17704 for updating Koha default version to 4.xx
- If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.
- start a harvest with the following command:
misc/cronjobs/harvest_oai.pl
This script starts an OAI Harvest
This script has the following parameters :
-h --help: this message
-v --verbose (enable output to stdout)
-r --repository: id of the OAI repository
-d --days: number of days to harvest from (optional)
-l --list: list the OAI repositories
-f --force: force harvesting (ignore records datestamps) (optional)
Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f
- Check the logs in the Log Viewer (tools/viewlog.pl)
(CronjobLog syspref must be activated)
Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Bug 29948 added a "multiple_sortable" type for sysprefs, to have the
options sortable. But the options were not picked for translation.
This patch fixes it.
Test plan:
Run `cd misc/translator && perl translate update LANG` before and after
this patch. Notice that with this patch you see the different options of
OPACAuthorIdentifiersAndInformation in the LANG-pref.po file (eg.
"Associated group (373$a$s$t$u$v$0)")
Translate the options
Install the translated version of the templates and confirm that the
strings appear translated on the interface.
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes.
Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When the 'RestrictPatronsWithFailedNotices' syspref is enabled then
patrons with email and sms notices which failed sending (have a
message_queue.status field of 'failed') have a restriction (debarment)
applied to them.
Test plan:
1. In the Koha staff client > Tools > Overdue notice/status triggers and
create the 'First' rule for all patron categories as:
Delay: 1
Letter: Overdue Notice
SMS: ticked
Ensure you have an SMS notice for the ODUE letter.
2. In the system preferences make sure you enter dummy data into the
SMSSendUsername, SMSSendPassword and EmailSMSSendDriverFromAddress
sysprefs
2. Find two non-debarred patrons and make sure they have invalid SMS numbers set. The SMS numbers must be INCORRECT, for example "123" as an SMS number. Leaving this field empty will result in the message_transport_type defaulting to 'print' instead of 'sms'.
3. Check one item out to each patron in step 2
4. Jump into the database and run the query:
UPDATE issues SET date_due=<2 days before current date> WHERE
borrowernumber=<borrower1>;
UPDATE issues SET date_due=<2 days before current date> WHERE
borrowernumber=<borrower2>;
5. Go to misc/cronjobs directory and enter the Koha shell:
sudo koha-shell <instancename>
6. Run: ./overdue_notices.pl
7. Exit the shell and jump back into the database and run the query:
SELECT message_transport_type, status FROM message_queue WHERE
borrowernumber=<borrower1> OR borrowernumber=<borrower2>;
8. Confirm both new notice records have the message_transport_type is 'sms' and the status of 'pending'
9. Exit the database and re-enter the Koha shell and run the command:
./process_message_queue.pl
10. Jump back into the database re-run the query from step 7 and confirm
the status is 'failed' for both
11. Also run the query:
SELECT * FROM borrower_debarments WHERE borrowernumber=<borrower1> OR
borrowernumber=<borrower2>;
Notice there is no added debarment to these two patrons
12. Apply patch, restart memcached and plack. In the
installer/data/mysql directory enter the Koha shell and run the command:
./update_database.pl
13. In the Administration > Global System Preferences interface of the
staff client notice there is a new system (set to "Don't" by default)
named 'RestrictPatronsWithFailedNotices'.
Enable it (i.e. select 'Do')
14. Create a new file in the /etc/cron.daily directory named koha-custom and
add the following line to it:
koha-foreach --chdir --enabled
/usr/share/koha/bin/cronjobs/restrict_patrons_with_failed_notices.pl
15. In the misc/cronjobs directory enter the Koha shell and run the command:
./restrict_patrons_with_failed_notices.pl
16. The script should output text saying: There are borrowers with
failed SMS or email notices
However because you haven't given the script the argument -c it won't
apply debarments (restrictions) to any of the patrons with the failed
SMS or email notices.
16. Query the borrower_debarments table:
SELECT * FROM borrower_debarments WHERE borrowernumber=<borrower1> OR
borrowernumber=<borrower2>;
Notice they still have no restriction
17. Now in the Koha shell run the command:
./restrict_patrons_with_failed_notices.pl -c
18. Notice the script outputs the text:
There are borrowers with failed SMS or email notices
Applying restriction to patron <borrowernumber>: <borrower firstname>
<borrower surname>;
19. Repeat step 16 and notice both patrons now have 1 restriction each
with the borrower_debarments.type=SUSPENSION and comment=SMSnumber
invalid and expiration=NULL
20. Query the borrowers table:
SELECT debarred, debarredcomment FROM borrowers WHERE
borrowernumber=<borrower1> OR borrowernumber=<borrower2>;
21. Notice the values are:
debarred= 9999-12-31
debarredcomment= SMS number invalid
22. Repeat step 17 and notice the script outputs:
There are borrowers with failed SMS or email notices
Patron <borrowernumber>: <borrower firstname> <borrower surname> is currently restricted due to having an invalid SMS number. No new restriction applied"
23. Repeat step 16 and notice no new debarment has been added to those
borrowers as they have already been restricted from having a failed SMS
notice.
24. In the Koha home directory run the command:
prove t/db_dependent/Koha/Notices.t
This unit test contains the tests for the new subroutines added to
Koha/Notice/Message.pm which are restrict_patron_when_notice_fails() and
get_failed_notices()
25. All tests should pass
26. Sign off
Sponsored-by: Catalyst IT
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Libraries may want to use this for more varied purposes and
adjust the language of the email based on the sftp upload
that succeeded or failed.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tidy and add exec to atomic update
Fix POD
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
SFTP_FAILURE and SFTP_SUCCESSFUL notices can be configured from Notices
and slip tool.
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test you need to have a remote server you are authorised to SFTP files to.
Test plan:
1. Apply patch and restart services
2. Generate a file on your Koha server. Place it in /tmp/test.mrc
3. Manually connect to the SFTP remote host and accept the hostkey
4. Enter Koha shell
5. Run the sftp_file.pl command-line script
./sftp_file.pl --host <your remote hostname> --user <your user> --pass <your password> --upload_dir <your upload directory> --file /tmp/test.mrc
6. Confirm the SFTP was successful
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL
Test plan -
Examples of setting up some 0000-00-00 date fields:
e.g. Borrowers table
update borrowers set dateexpiry="0000-00-00" where borrowernumber=49;
e.g. Items table
update items set datelastseen="0000-00-00" where itemnumber=12;
To identify the cases:
1) Set up some 0000-00-00 date fields
2) Run the misc/maintenance/search_for_data_inconsistencies.pl script
3) Run fix_invalid_dates.pl -v
To fix the cases:
1) Run fix_invalid_dates.pl -c -v
2) Notice the value has changed from 0000-00-00 to NULL
e.g. Borrowers table
select dateexpiry from borrowers where borrowernumber=49;
e.g. Items table
select datelastseen from items where itemnumber=12;
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
After the introduction of misc/workers/es_indexer_daemon.pl,
misc/workers/background_jobs_worker.pl should not handle the
elastic_index queue. But the latter script still mentions this
queue in the POD. This patch removes that mention.
To test:
- Run perldoc misc/workers/background_jobs_worker.pl
- Verify "elastic_index" is mentioned under --queue
- Apply this patch
- Run perldoc again and verify the mention of "elastic_index"
is gone
Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds POD to the script and rpelaces the custom help script with pod2usage
TO test:
perl misc/cronjobs/erm_run_harvester.pl
Confirm you get helpful usage data
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch simply adds logging to the cronjob
To test:
1 - Enable CronjobLog
http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=CronjobLog
2 - Enable ERMModule
http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=ERMModule
3 - Home -> E-resource Management -> Data providers -> New data provider
Fill from registry or create your own, but change the URL and put in any required numbers
We don't need a real test, just to run the ob
4 - perl misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run --debug
5 - View the logs, no log of the script
6 - Apply patches
7 - Run the job again
8 - Check logs
9 - Success!
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1) Create an authority with a unique value in a searchable field, for instance a PERSO_NAME authority with 'BZ18273' in the 100$a field.
2) Create a second authority with the same value in the same field.
3) Export this second authority as iso2709 (Save -> MARC (Unicode/UTF-8))
4) Run bulkmarcimport.pl with the following parameters:
perl bulkmarcimport.pl -a --file=/kohadevbox/koha/auth-xxxx.utf8 --match='Heading,100$a' -l /kohadevbox/koha/bulk.log
5) Check in the log file and/or through search that a third authority has been added (despite using the --match option)
6) Apply the patch
7) Delete the 3 authorities and repeat step 1 to 4.
8) Check that a third authority has not been added (--match worked)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch remove the subroutine C4::Members::DeleteExpiredOpacRegistrations.
The code is moved and adjusted to Koha::Patrons.
We now have 2 new methods:
Koha::Patrons->filter_by_expired_opac_registrations
Koha::Patrons->filter_by_safe_to_delete
Test plan:
Run the misc/cronjobs/cleanup_database.pl script with the
--del-exp-selfreg (as well as --verbose and --confirm) and confirm that
it behaves as expected
Note that it improve the output of the verbose mode and now tell if the something
is wrong with the config (syspref not set)
Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch removes the subroutine
C4::Members::DeleteUnverifiedOpacRegistrations and use the
Koha::Objects->filter_by_last_update method that is used for the other
flag in this script.
Test plan:
Confirm that --del-unv-selfreg works the same before and after this
patch applied
Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add documentation for the new -si, --skip_indexing parameter
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed
we were also indexing during the commity, however, it appears we were not. This patch ensures we push
the record and ids into the arrays for indexing during commit.
Additionally I add a skip to next record on error, to match biblio behaviour.
I also correct a log referring to biblios during authority importing
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new optoin to skip indexing to bulkmarcimport.
It also fixes a bug where authorities were being indexed multiple times
during import.
To test:
1 - Apply patch
2 - Download the sample files on this bug
3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES)
4 - Note the total (435 in KTD)
5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc
6 - Search again, note increaed by 100
7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing
8 - Search again, no increase
9 - perl misc/search_tools/rebuild_elasticsearch.pl -v
10 - Search again, increase, records were added but not initially indexed
11 - Browse to authorities and search as for biblios
12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc
13 - Search again, note increase
14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing
15 - Search again, no increase
16 - perl misc/search_tools/rebuild_elasticsearch.pl -v
17 - Search again, increase, records were added but not initially indexed
18 - Sign off! Hi5!
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch was envisioned by Julian Maurice.
Test plan:
0. Apply the 1st patch (the revert)
1. Go to `misc/translator/po`
2. es-ES-messages-js.po
3. Mark "Edit" string as fuzzy (around line 48). It should look like:
```
msgid "Edit"
msgstr "Editar"
```
4. Edit ./intranet-main.tt and add the following lines at the bottom,
inside the `$(document).ready` block:
```javascript
console.log(_("Edit"));
console.log(__("Edit"));
```
5. Install the templates
k$ koha-translate --install es-ES --dev kohadev && restart_all
6. Enable *es-ES* by searching for `language` in the sysprefs and switch
to it for the staff interface.
7. Go to the Koha home page, open the browser console
=> FAIL: Notice that the second log in the console is displaying the
fuzzy string (i.e. is being translated when it shouldn't)
8. Apply this patch
9. Re-install the translated templates:
k$ koha-translate --update es-ES --dev kohadev && restart_all
10. Repeat 7
=> SUCCESS: With this patch applied both logs show the English version of the
string.
11. Remove the fuzzy flag on `es-ES-messages-js.po`
12. Repeat 9 and 10
=> SUCCESS: The results are similar to step 7, but in this case they are
expected as the string is not marked fuzzy.
13. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This reverts commit 4eb9816354.
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When the automatic renewal cron job is using parallel processing, it
aims to process all of the renewals for any given patron together in one
chunk to avoid data conflicts. To accomplish this, it starts a new data
chunk each time it encounters a new patron. However, if a patron's
renewing checkouts aren't all consecutive in the database, that patron's
data ends up split across multiple chunks.
We need to sort the issues by borrowernumber before attempting to chunk
them in order to make sure they are chunked and processed correctly.
To test (using KTD default test data):
Setup:
1. Edit the default circulation rule:
- Set Automatic renewal to "Yes"
- Set No automatic renewal before to 3
2. Open the following patron accounts in separate tabs:
- Floyd Delgado
- Joyce Gaines
- Edna Acosta
- Mary Burton
3. Perform the following patron account edits for each of the above
patrons (and keep the tabs open):
- Enable automatic renewal notices, and set them to digests only
- Add a value to the email field
4. Enter the kshell (ktd --shell)
5. Edit /etc/koha/sites/kohadev/koha-conf.xml, and add the following
lines near the end, just above the </config> and </yazgfs> closing
tags:
<auto_renew_cronjob>
<parallel_loops_count>2</parallel_loops_count>
</auto_renew_cronjob>
6. restart_all
Reproducing the issue:
7. Apply the test patch only
8. Run perl generate_checkouts.pl to generate test data
9. perl misc/cronjobs/automatic_renewals.pl -v -c
--> The test patch added output that will show how the renewals were
chunked into "chunk 0" and "chunk 1" for the two parallel loops.
Note that the issues for each borrower are not processed nicely in
one chunk, but are separated across multiple chunks and alternated
with other borrowers.
10. Check the checkouts for each of the four patrons from above
--> All checkouts should have renewed
11. Check the notices tab for each of the four patrons
--> Notice errors in the automatic renewal digest notices. A patron's
renewals may be split across multiple digests, a digest may be
missing renewals, or a patron may not have received a digest at all
Testing the patch:
12. Apply the second patch
13. Reset the due dates on all checkouts so that they will all be
eligible for automatic renewal again:
- koha-mysql kohadev
- UPDATE issues SET date_due=<two days from today>;
14. perl misc/cronjobs/automatic_renewals.pl -v -c
--> Note that the renewals are now correctly chunked by patron
15. Check the checkouts and notices tab for each of the four patrons
--> All checkouts should have renewed, and all patrons should have a
single new Auto Renewals Digest notice that correctly lists all of
their renewed items
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: tidied the code block inline (tcohen)
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Olivier V <olivier.vezina@inLibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
1 - perl misc/migration_tools/bulkmarcimport.pl -b --file=notafile.mrc -v
2 - Output:
Characteristic MARC flavour: MARC21
0 MARC records done in 0.00804400444030762 seconds
3 - Apply patch
4 - perl misc/migration_tools/bulkmarcimport.pl -b --file=notafile.mrc -v
5 - Output:
Characteristic MARC flavour: MARC21
Could not open notafile.mrc: No such file or directory at misc/migration_tools/bulkmarcimport.pl line 262.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Script migration_tools/build_oai_sets.pl is missing ORDER BY
biblionumber.
This is a problem when using OFFSET and LIMIT, the results may be
differenly orderder between two calls of this script.
Test plan :
1) Create a OAI SET with all records
2) Run migration_tools/build_oai_sets.pl with the first half of records
3) Run migration_tools/build_oai_sets.pl with the second half of records
4) Check all records are in the OAI set
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch removes setting AutoCommit to 0 and commiting only every X records.
Instead we commit as we go and report progress using a parameter.
Bug 36474 reduced the numebr of changes that are being committed, so this should be a reasonable change. The
use of commits without transactions was causing problems if the library was active while the script ran.
To test:
1 - perl misc/cronjobs/update_totalissues.pl -c
2 - Script runs, but with unknown parameter
3 - perl misc/cronjobs/update_totalissues.pl -p 10
4 - Script runs and reports every 10 records
5 - per; misc/cronjobs/update_totalissues.pl
6 - Script runs and reports every 100 records by default
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a new check in UpdateTotalIssues to check that we are changing the number
of total issues before calling ModBiblio
To test:
0 - Enable CataloguingLog
1 - Checkout an item
2 - Run : misc/cronjobs/update_totalissues.pl --use-stats --commit=1000 -v
3 - In report, note all biblios were updated
4 - Check action_logs - note a new entry for every biblio
5 - Apply patch
6 - Repeat
7 - Note no biblios reported updated
8 - Note no new cataloguing log entries
9 - Checkout the item again
10 - Run again
11 - Note biblionumber has updated count in verbose output
12 - Note report only rpeort 1 biblio modified, the rest only processed
13 - Only one line added to action_logs
14 - Run it again
15 - Confirm no updates
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This change lazy-loads modules for setters in the
koha-preferences tool, so that getters are free to run super fast.
Between BZ 37657 and BZ 37682 we effectively eliminate the
overhead of running "get" or "dump" commands via the koha-preferences
tool.
Test plan:
1. time misc/admin/koha-preferences get SearchEngine
2. Note time is about .35 seconds
3. time misc/admin/koha-preferences dump
4. Note time is about .35 seconds
5. Create sysprefs.yml
---
marcflavour: MARC21
viewMARC: 1
6. time misc/admin/koha-preferences load -i sysprefs.yml
7. Note time is about .35 seconds
8. time misc/admin/koha-preferences set SearchEngine Elasticsearch
9. Note time is about 1 seconds
10. Apply patch
11. Repeat the koha-preferences commands above
12. Note that the "dump" and "get" commands run in about .09-.1
seconds. The "load" and "set" commands still take the same amount
of time as their behaviours haven't changed
13. misc/admin/koha-preferences set SearchEngine Elasticsearch1
14. koha-mysql kohadev
15. select * from action_logs where module = 'SYSTEMPREFERENCE'
order by action_id desc limit 5;
16. Note that the action log showing Elasticsearch1 update says
"interface"
of "commandline" and "script" of "koha-preferences"
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
By using Koha::Database->dbh() to use a minimal database handle
which doesn't preload the whole DBIx::Class schema, we're able to
run the same command 2-3 times faster.
This is beneficial when running the tool in a loop which runs the
command serially one by one.
Test plan:
1. time misc/admin/koha-preferences get SearchEngine
2. Note time is about 1 second
3. time misc/admin/koha-preferences dump
4. Note time is about 1 second
5. Create sysprefs.yml
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When libraries have a lot of checkouts, or an AMH, checkins can happen while the cron is running.
This patch simply adds an eval around the auto renewal attempt in case of early check in or other errors.
You can verify cron completed by enabling cronjob log in system preferences and checking the action logs
To test:
1 - Add 'sleep(10);' to automatic_renewals.pl
2 - Set circulation rules to enable automatic renewals
3 - Issue an item to a patron
4 - perl misc/cronjobs/automatic_renewals.pl -v
5 - Confirm item would not be renewed
6 - perl misc/cronjobs/automatic_renewals.pl -v -c
7 - Quickly check in the item
8 - The cronjob dies
DBIx::Class::Row::update(): Can't update Koha::Schema::Result::Issue=HASH(0x586e1a674fb0): row not found at /kohadevbox/koha/Koha/Object.pm line 172
9 - Apply patch
10 - Checkout the item again
11 - perl misc/cronjobs/automatic_renewals.pl -v -c
12 - Quickly checkin the item
13 - You get a warning, but the cron completes
Signed-off-by: CJ Lynce <cj.lynce@westlakelibrary.org>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tidy the whole thing
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Since version 24.05, due to the changes mentioned at
https://wiki.koha-community.org/wiki/Koha_/svc/_HTTP_API#Changes_coming_in_Koha_24.05 ,
the `connexion_import_daemon.pl` stopped working. The reason for this is that
it did not use CSRF tokens.
To test:
1. Get a Koha instance on 24.05, before applying the patch.
2. Create a plain text file somewhere on the server containing a raw MARC
record (not XML). Let's call it `marc.txt`.
3. On the server, create a config file like this:
```
host: 0.0.0.0
port: 5500
koha: http://localhost:82 # Where 82 is the port of the Koha staff interface.
user: foo # A Koha staff user.
password: Fooo1234 # The Koha staff user's password.
import_mode: stage
```
4. Run `./connexion_import_daemon.pl --config the-config-file-path`
5. In another terminal on the same server (or from anywhere that can reach the
port opened by the `connexion_import_daemon.pl` script,
run `nc localhost 5500 < marc.txt`
6. Observe in the stderr of the daemon script: `Response: Unsuccessful request`
7. Stop the daemon script.
8. Apply the patch and repeat steps 4 and 5.
9. Observe in the stderr of the daemon script:
`Response: Success. Batch number ... - biblio record number HASH(...) added to Koha`
10. Check at /cgi-bin/koha/tools/manage-marc-import.pl for a batch named
`(webservice)`. It should contain one record now. This is how we know that
authentication between the daemon and Koha worked, which is what this
patch tries to address.
Thanks-to: David Cook <dcook@prosentient.com.au>
Sponsored-by: Reformational Study Centre <www.refstudycentre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Trivial patch.
Change the --branch and --skip-branch options of the longoverdue cron script
to --library and --skip-library to meet the Terminology Guidelines.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This script doesn't seem to be included in cron files by default.
This change is to allow script parameters to effect only certain branches.
This allows the script to be added multiple times to a cron file with
different settings for different branches.
Test plan:
1. apply patch
2. identify two books at different branches the same number of days overdue.
3. run the longoverdue.pl script specifying one of the branches in the
--branch command line parameter.
i.e. koha-shell -c 'misc/cronjobs/longoverdue.pl --branch branch_code --lost 60=2 --maxdays=61 --confirm' instance_name
4. observe that the book at the specified branch has been or would be affected
by the script while the other book is not.
Signed-off-by: Tadeusz „tadzik” Sośnierz <tadeusz@sosnierz.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Enable the export_records.pl script use a report output to export biblios or authorities
Test plan:
1. Apply patches and restart services
2. Create a SQL report (id=1)
SELECT biblionumber
FROM biblio
3. Create a SQL report (id=2) and set an item as notforloan = -1
SELECT title, author, biblio.biblionumber
FROM biblio
LEFT JOIN items USING (biblionumber)
WHERE items.notforloan = <<Not for loan|NOT_LOAN>>
4. Create a SQL report (id=3)
SELECT title, author
FROM biblio
5. Create a SQL report (id=4)
SELECT authid
FROM auth_header
6. Run export_records.pl using report id=1 and confirm a koha.mrc file is created in the misc directory:
cd misc
./export_records.pl --report_id=1
7. Delete the koha.mrc file
8. Run export_records.pl using report id=2
./export_records.pl --report_id=2
9. Notice you are prompted to supply a parameter
10. Re-run report id=2 supplying a parameter. Confirm the koha.mrc file is created and contains bib data
./export_records.pl --report_id=2 --report_param=-1
11. Run export_records.pl using report id=3
./export_records.pl --report_id=3
12. Notice you get the message: The --report_id you specified does not fetch a biblionumber
13. Delete the koha.mrc file
14. Run export_records.pl using report id=4
./export_records.pl --report_id=4
15. Notice you get a message 'The --report_id you specified does not fetch a biblionumber'
16. Re-run export_records.pl setting the record-type=auths
./export_records.pl --record-type=auths --report_id=4
17. Notice the koha.mrc file is generated and contains auth data
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: Alexandre Noel <alexandre.noel@inlibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change consistently sends the Csrf-Token in the request header.
Previously, one POST sent it in the request body, while the other POST
sent it in the request header. Since we're using an API, it's best
for us to always send it in the request header
Test plan:
0. Apply the patch
1. perl ./misc/migration_tools/koha-svc.pl \
http://localhost:8081/cgi-bin/koha/svc koha koha 29 > bib-29.xml
2. perl ./misc/migration_tools/koha-svc.pl \
http://localhost:8081/cgi-bin/koha/svc koha koha 29 bib-29.xml
3. Note that the following appears in STDOUT and there is no 403 error:
"update 29 from bib-29.xml"
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change fixes the Koha::SVC to store the CSRF token for
the authenticated session for further POSTing.
Test plan:
0. Apply the patch
1. perl ./misc/migration_tools/koha-svc.pl \
http://localhost:8081/cgi-bin/koha/svc koha koha 29 > bib-29.xml
2. perl ./misc/migration_tools/koha-svc.pl \
http://localhost:8081/cgi-bin/koha/svc koha koha 29 bib-29.xml
3. Note that the following appears in STDOUT and there is no 403 error:
"update 29 from bib-29.xml"
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When using __() (ie. Gettext.js) we are seeing the translations that are marked as fuzzy.
This is definitely not the expected behaviour.
It happens because (our version of) po2json are old and no longer maintained,
and just embed them.
It seems that the bin we have has been upgraded to a JS version
(different authors).
Test plan:
(replace LANG with your language code)
0. Do not apply this patch
Edit misc/translator/po/LANG-messages-js.po
Mark a string as fuzzy
Edit ./intranet-main.tt and add the following lines inside $(document).ready
console.log(_("Your string"));
console.log(__("Your string"));
Replace "Your string" with the string you are actually testing.
Update the templates: `koha-translate --update LANG --dev kohadev && restart_all`
Go to the Koha home page, open the console.
=> Notice that the second log in the console is displaying the fuzzy string.
1. Apply this patch
Install the new version of po2json using `yarn install`
Repeat the previous steps.
=> With this patch applied both logs show the English version of the
string.
Remove fuzzy, update the templates and try again.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Andreas Roussos
Matthias Meusburger
Jonathan Druart
Kyle M Hall
Katrin Fischer
Alex Buckley
Nick Clemens
wainuiwitikapark
Magnus Enger
Thomas Klausner
Tomas Cohen Arazi
Emily Lamancusa
Fridolin Somers
David Cook
Rudolf Byker
Michael Hafen