This patch adds a 'delete' flag to the cronjob to select whether a file should be deleted or archived once processed
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Andrew Fuerste Henry <andrewfh@dubcolib.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a cronjob to take a marc file from the Koha file directory system and import it before creating order lines
Test plan:
This test plan involves running a cronjob, not sure how easy this is in the sandboxes
1) In system preferences, click Search and then select the Acquisitions option from the left hand menu
2) Enable MarcOrderingAutomation
3) Paste the following into MarcFieldsToOrder
price: 975$p
quantity: 975$q
budget_code: 975$h
4) Paste the following into MarcItemFieldsToOrder
homebranch: 949$a
holdingbranch: 949$b
itype: 949$y
nonpublic_note: 949$x
public_note: 949$z
loc: 949$c
ccode: 949$8
notforloan: 949$7
uri: 949$u
copyno: 949$t
price: 949$g
replacementprice: 949$v
itemcallnumber: 949$o
quantity: 949$k
budget_code: 949$l
Now save the sysprefs
5) In the administration homepage, under Acquisition parameters there should now be a link called "MARC order accounts"
6) Click on this and click the New account button
7) Choose a vendor and a budget
8) Enter a description
9) Enter the following in download directory - "/var/lib/koha/kohadev/tmp/koha_kohadev_upload"
10) Select any record matching settings you want (not important for the test plan, this uses the same code as the marc file staging that already exists in Koha)
11) Check Yes under Check for embedded item record data
12) Click submit
13) We now need to upload a MARC file into the directory that we put in the account. The easiest way to do this is through the marc staging tool
14) Navigate to Cataloging > Stage MARC records for import
15) Upload the file attached to this bug using the Choose file button
16) Click Upload file.
17) Leave the page without completing the new form that appears, we don't want to stage this file. The file we uploaded is now in the directory ready for the cronjob to process as if it had been transferred via SFTP.
18) In the CLI run, perl misc/cronjobs/marc_ordering_process.pl -v -c
19) The logging should show that the file has been processed for that account
20) Navigate to Acquisitions
21) Search for the Vendor that you put in the MARC ordering account
22) There should be a basket created on the vendor with the name of the file you uploaded (there will also be a reference at the start e.g. ea2323966faef2c59a6056dbd1f7d658_filename.marcxml. This is because we used the marc staging tool to upload the file and that adds a reference to filenames. This will not appear when using files from an SFTP location).
23) Click in the basket
24) There should be three orderlines, complete with accounting details that have been mapped from the file.
One line should have a fund of Fund 1_2, quantity of 4 and a replacement price of 6.75, one should have Secondary fund, 2, 6.40 and one should have Main fund,2, 6.30
Signed-off-by: Andrew Fuerste Henry <andrewfh@dubcolib.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Option name changed, POD updated.
The reason not to do this by default is that it is an extra record conversion
to check the validity, so may slow down the import.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a conversion from MARC -> XML -> MARC to catch any parsing errors.
If errors are found, we then lint the record to catch any problems, output the
warnings, and skip the record.
To test:
1 - Download the sample records from this bug report
2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v
fails!
3 - perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v
fails!
4 - Apply patch
5 - Repeat 2 & 3 - no change
6 - Add -st switch to the commands:
perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v -st
perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v -st
7 - The records are now skipped, and the script completes
8 - Confirm the warnings generated are useful
9 - Sign off!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Grab the sample file on this bug report
2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=spaceditems.pl -v
3 - An exception for 'Duplicate ID' is thrown and script dies
4 - Apply patch
5 - run script again
6 - The script finishes, no items are added, but record is
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can now call metadata_record directly on the Koha::Biblio object.
This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate. The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
It would be nice to be able to filter the plugins run by plugins
nightly for development and testing, and for added flexibility
as to when to run cronjob_nightly for different plugins if need be.
Test Plan:
1) Apply this patch
2) Check the metadata for an installed plugin, copy the name or another
metadata value.
3) Run plugins_nightly.pl with a filter that does not match
e.g. plugins_nightly.pl -m name="No plugin has this name"
4) Note no plugins cronjob methods are run
5) Run plugins_nightly.pl with a filter that *does* match
e.g. plugins_nightly.pl -m name="Example Kitchen-Sink Plugin"
6) Not only the matching plugin runs!
7) Run plugins_nightly with no filter
e.g. plugins_nightly.pl
8) Note all plugins with nightly cronjobs are run!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Most messages in Koha are handled by process_message_queue.pl. This script calls before_send_messages which calls the plugin hook before_send_messages. This works for nearly all messages, but some messages like WELCOME notices are triggered immediately and do not way to for process_message_queue.pl to run. We should move the triggering of before_send_messages into SendQueuedMessages so it is always handled.
Test Plan:
1) Install version 2.5.1 of the kitchen sink plugin
2) Run process_messsage_queue.pl and also trigger a WELCOME notice
3) Note the "Plugin hook before_send_message called with the params" message in the logs for the former but not the latter
4) Apply this patch
5) Repeat step 2, note there the message also shows in the logs for the WELCOME notice!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
In an earlier patch, the `id` column of the `oai_servers`
table was renamed to `oai_server_id`. However, the code
in misc/cronjobs/harvest_oai.pl still uses the old column
name, which results in the following error:
$ misc/cronjobs/harvest_oai.pl --list
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Unknown column 'id' in 'order clause' at /kohadevbox/koha/Koha/Objects.pm line 403
This patch fixes that.
Test plan:
1) $ misc/cronjobs/harvest_oai.pl --list
[...you get the DBI Exception...]
2) Apply the patch.
3) $ misc/cronjobs/harvest_oai.pl --list
[...list of OAI repositories...]
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Rename Oaipmh to OAI
Rename oaiservers to oai_servers
Rename OaiServer to OAIServer
Rename oai_servers.id to oai_servers.oai_server_id
Rename ServerSearch to server_search
Use a callback function for logs
Use context for string "Set"
Split translation string
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- Run updatedatabase.pl
- Apply the DBIC update patch
- Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)
If the repository dataformat is in oai_dc, you can use the sample XSLT files:
For Unimarc:
OAIDCtoUNIMARCXML.xsl for bibliographic records
AuthOAIDCtoUNIMARCXML.xsl for authorities
For Marc21:
OAIDCtoMARC21XML.xsl for bibliographic records
AuthOAIDCtoMARC21XML.xsl for authorities
If the repository dataformat is in marc-xml, you can use
OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.
- Some repositories to test with:
Bibliographic records, oai_dc:
Endpoint: http://staroai.theses.fr/OAIHandler
Set: ddc:260
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
or
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl
Authorities, oai_dc:
Endpoint: https://www.idref.fr/OAI/oai.jsp
Set: a
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
or
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl
Authorities, marc-xml, Unimarc:
Endpoint: https://www.idref.fr/OAI/oai.jsp
Set: a
XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl
- Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
Version 4.xx is needed.
See Bug 17704 for updating Koha default version to 4.xx
- If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.
- start a harvest with the following command:
misc/cronjobs/harvest_oai.pl
This script starts an OAI Harvest
This script has the following parameters :
-h --help: this message
-v --verbose (enable output to stdout)
-r --repository: id of the OAI repository
-d --days: number of days to harvest from (optional)
-l --list: list the OAI repositories
-f --force: force harvesting (ignore records datestamps) (optional)
Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f
- Check the logs in the Log Viewer (tools/viewlog.pl)
(CronjobLog syspref must be activated)
Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Bug 29948 added a "multiple_sortable" type for sysprefs, to have the
options sortable. But the options were not picked for translation.
This patch fixes it.
Test plan:
Run `cd misc/translator && perl translate update LANG` before and after
this patch. Notice that with this patch you see the different options of
OPACAuthorIdentifiersAndInformation in the LANG-pref.po file (eg.
"Associated group (373$a$s$t$u$v$0)")
Translate the options
Install the translated version of the templates and confirm that the
strings appear translated on the interface.
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes.
Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When the 'RestrictPatronsWithFailedNotices' syspref is enabled then
patrons with email and sms notices which failed sending (have a
message_queue.status field of 'failed') have a restriction (debarment)
applied to them.
Test plan:
1. In the Koha staff client > Tools > Overdue notice/status triggers and
create the 'First' rule for all patron categories as:
Delay: 1
Letter: Overdue Notice
SMS: ticked
Ensure you have an SMS notice for the ODUE letter.
2. In the system preferences make sure you enter dummy data into the
SMSSendUsername, SMSSendPassword and EmailSMSSendDriverFromAddress
sysprefs
2. Find two non-debarred patrons and make sure they have invalid SMS numbers set. The SMS numbers must be INCORRECT, for example "123" as an SMS number. Leaving this field empty will result in the message_transport_type defaulting to 'print' instead of 'sms'.
3. Check one item out to each patron in step 2
4. Jump into the database and run the query:
UPDATE issues SET date_due=<2 days before current date> WHERE
borrowernumber=<borrower1>;
UPDATE issues SET date_due=<2 days before current date> WHERE
borrowernumber=<borrower2>;
5. Go to misc/cronjobs directory and enter the Koha shell:
sudo koha-shell <instancename>
6. Run: ./overdue_notices.pl
7. Exit the shell and jump back into the database and run the query:
SELECT message_transport_type, status FROM message_queue WHERE
borrowernumber=<borrower1> OR borrowernumber=<borrower2>;
8. Confirm both new notice records have the message_transport_type is 'sms' and the status of 'pending'
9. Exit the database and re-enter the Koha shell and run the command:
./process_message_queue.pl
10. Jump back into the database re-run the query from step 7 and confirm
the status is 'failed' for both
11. Also run the query:
SELECT * FROM borrower_debarments WHERE borrowernumber=<borrower1> OR
borrowernumber=<borrower2>;
Notice there is no added debarment to these two patrons
12. Apply patch, restart memcached and plack. In the
installer/data/mysql directory enter the Koha shell and run the command:
./update_database.pl
13. In the Administration > Global System Preferences interface of the
staff client notice there is a new system (set to "Don't" by default)
named 'RestrictPatronsWithFailedNotices'.
Enable it (i.e. select 'Do')
14. Create a new file in the /etc/cron.daily directory named koha-custom and
add the following line to it:
koha-foreach --chdir --enabled
/usr/share/koha/bin/cronjobs/restrict_patrons_with_failed_notices.pl
15. In the misc/cronjobs directory enter the Koha shell and run the command:
./restrict_patrons_with_failed_notices.pl
16. The script should output text saying: There are borrowers with
failed SMS or email notices
However because you haven't given the script the argument -c it won't
apply debarments (restrictions) to any of the patrons with the failed
SMS or email notices.
16. Query the borrower_debarments table:
SELECT * FROM borrower_debarments WHERE borrowernumber=<borrower1> OR
borrowernumber=<borrower2>;
Notice they still have no restriction
17. Now in the Koha shell run the command:
./restrict_patrons_with_failed_notices.pl -c
18. Notice the script outputs the text:
There are borrowers with failed SMS or email notices
Applying restriction to patron <borrowernumber>: <borrower firstname>
<borrower surname>;
19. Repeat step 16 and notice both patrons now have 1 restriction each
with the borrower_debarments.type=SUSPENSION and comment=SMSnumber
invalid and expiration=NULL
20. Query the borrowers table:
SELECT debarred, debarredcomment FROM borrowers WHERE
borrowernumber=<borrower1> OR borrowernumber=<borrower2>;
21. Notice the values are:
debarred= 9999-12-31
debarredcomment= SMS number invalid
22. Repeat step 17 and notice the script outputs:
There are borrowers with failed SMS or email notices
Patron <borrowernumber>: <borrower firstname> <borrower surname> is currently restricted due to having an invalid SMS number. No new restriction applied"
23. Repeat step 16 and notice no new debarment has been added to those
borrowers as they have already been restricted from having a failed SMS
notice.
24. In the Koha home directory run the command:
prove t/db_dependent/Koha/Notices.t
This unit test contains the tests for the new subroutines added to
Koha/Notice/Message.pm which are restrict_patron_when_notice_fails() and
get_failed_notices()
25. All tests should pass
26. Sign off
Sponsored-by: Catalyst IT
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Libraries may want to use this for more varied purposes and
adjust the language of the email based on the sftp upload
that succeeded or failed.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tidy and add exec to atomic update
Fix POD
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
SFTP_FAILURE and SFTP_SUCCESSFUL notices can be configured from Notices
and slip tool.
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test you need to have a remote server you are authorised to SFTP files to.
Test plan:
1. Apply patch and restart services
2. Generate a file on your Koha server. Place it in /tmp/test.mrc
3. Manually connect to the SFTP remote host and accept the hostkey
4. Enter Koha shell
5. Run the sftp_file.pl command-line script
./sftp_file.pl --host <your remote hostname> --user <your user> --pass <your password> --upload_dir <your upload directory> --file /tmp/test.mrc
6. Confirm the SFTP was successful
Sponsored-by: Horowhenua Libraries, Toi Ohomai Institute of Technology, Plant and Food Research Limited, Waitaki District Council, South Taranaki District Council New Zealand
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL
Test plan -
Examples of setting up some 0000-00-00 date fields:
e.g. Borrowers table
update borrowers set dateexpiry="0000-00-00" where borrowernumber=49;
e.g. Items table
update items set datelastseen="0000-00-00" where itemnumber=12;
To identify the cases:
1) Set up some 0000-00-00 date fields
2) Run the misc/maintenance/search_for_data_inconsistencies.pl script
3) Run fix_invalid_dates.pl -v
To fix the cases:
1) Run fix_invalid_dates.pl -c -v
2) Notice the value has changed from 0000-00-00 to NULL
e.g. Borrowers table
select dateexpiry from borrowers where borrowernumber=49;
e.g. Items table
select datelastseen from items where itemnumber=12;
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
After the introduction of misc/workers/es_indexer_daemon.pl,
misc/workers/background_jobs_worker.pl should not handle the
elastic_index queue. But the latter script still mentions this
queue in the POD. This patch removes that mention.
To test:
- Run perldoc misc/workers/background_jobs_worker.pl
- Verify "elastic_index" is mentioned under --queue
- Apply this patch
- Run perldoc again and verify the mention of "elastic_index"
is gone
Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds POD to the script and rpelaces the custom help script with pod2usage
TO test:
perl misc/cronjobs/erm_run_harvester.pl
Confirm you get helpful usage data
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch simply adds logging to the cronjob
To test:
1 - Enable CronjobLog
http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=CronjobLog
2 - Enable ERMModule
http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=ERMModule
3 - Home -> E-resource Management -> Data providers -> New data provider
Fill from registry or create your own, but change the URL and put in any required numbers
We don't need a real test, just to run the ob
4 - perl misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run --debug
5 - View the logs, no log of the script
6 - Apply patches
7 - Run the job again
8 - Check logs
9 - Success!
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1) Create an authority with a unique value in a searchable field, for instance a PERSO_NAME authority with 'BZ18273' in the 100$a field.
2) Create a second authority with the same value in the same field.
3) Export this second authority as iso2709 (Save -> MARC (Unicode/UTF-8))
4) Run bulkmarcimport.pl with the following parameters:
perl bulkmarcimport.pl -a --file=/kohadevbox/koha/auth-xxxx.utf8 --match='Heading,100$a' -l /kohadevbox/koha/bulk.log
5) Check in the log file and/or through search that a third authority has been added (despite using the --match option)
6) Apply the patch
7) Delete the 3 authorities and repeat step 1 to 4.
8) Check that a third authority has not been added (--match worked)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch remove the subroutine C4::Members::DeleteExpiredOpacRegistrations.
The code is moved and adjusted to Koha::Patrons.
We now have 2 new methods:
Koha::Patrons->filter_by_expired_opac_registrations
Koha::Patrons->filter_by_safe_to_delete
Test plan:
Run the misc/cronjobs/cleanup_database.pl script with the
--del-exp-selfreg (as well as --verbose and --confirm) and confirm that
it behaves as expected
Note that it improve the output of the verbose mode and now tell if the something
is wrong with the config (syspref not set)
Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch removes the subroutine
C4::Members::DeleteUnverifiedOpacRegistrations and use the
Koha::Objects->filter_by_last_update method that is used for the other
flag in this script.
Test plan:
Confirm that --del-unv-selfreg works the same before and after this
patch applied
Signed-off-by: Sukhmandeep Benipal <sukhmandeep.benipal@inLibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add documentation for the new -si, --skip_indexing parameter
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed
we were also indexing during the commity, however, it appears we were not. This patch ensures we push
the record and ids into the arrays for indexing during commit.
Additionally I add a skip to next record on error, to match biblio behaviour.
I also correct a log referring to biblios during authority importing
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new optoin to skip indexing to bulkmarcimport.
It also fixes a bug where authorities were being indexed multiple times
during import.
To test:
1 - Apply patch
2 - Download the sample files on this bug
3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES)
4 - Note the total (435 in KTD)
5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc
6 - Search again, note increaed by 100
7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing
8 - Search again, no increase
9 - perl misc/search_tools/rebuild_elasticsearch.pl -v
10 - Search again, increase, records were added but not initially indexed
11 - Browse to authorities and search as for biblios
12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc
13 - Search again, note increase
14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing
15 - Search again, no increase
16 - perl misc/search_tools/rebuild_elasticsearch.pl -v
17 - Search again, increase, records were added but not initially indexed
18 - Sign off! Hi5!
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch was envisioned by Julian Maurice.
Test plan:
0. Apply the 1st patch (the revert)
1. Go to `misc/translator/po`
2. es-ES-messages-js.po
3. Mark "Edit" string as fuzzy (around line 48). It should look like:
```
msgid "Edit"
msgstr "Editar"
```
4. Edit ./intranet-main.tt and add the following lines at the bottom,
inside the `$(document).ready` block:
```javascript
console.log(_("Edit"));
console.log(__("Edit"));
```
5. Install the templates
k$ koha-translate --install es-ES --dev kohadev && restart_all
6. Enable *es-ES* by searching for `language` in the sysprefs and switch
to it for the staff interface.
7. Go to the Koha home page, open the browser console
=> FAIL: Notice that the second log in the console is displaying the
fuzzy string (i.e. is being translated when it shouldn't)
8. Apply this patch
9. Re-install the translated templates:
k$ koha-translate --update es-ES --dev kohadev && restart_all
10. Repeat 7
=> SUCCESS: With this patch applied both logs show the English version of the
string.
11. Remove the fuzzy flag on `es-ES-messages-js.po`
12. Repeat 9 and 10
=> SUCCESS: The results are similar to step 7, but in this case they are
expected as the string is not marked fuzzy.
13. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This reverts commit 4eb9816354.
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When the automatic renewal cron job is using parallel processing, it
aims to process all of the renewals for any given patron together in one
chunk to avoid data conflicts. To accomplish this, it starts a new data
chunk each time it encounters a new patron. However, if a patron's
renewing checkouts aren't all consecutive in the database, that patron's
data ends up split across multiple chunks.
We need to sort the issues by borrowernumber before attempting to chunk
them in order to make sure they are chunked and processed correctly.
To test (using KTD default test data):
Setup:
1. Edit the default circulation rule:
- Set Automatic renewal to "Yes"
- Set No automatic renewal before to 3
2. Open the following patron accounts in separate tabs:
- Floyd Delgado
- Joyce Gaines
- Edna Acosta
- Mary Burton
3. Perform the following patron account edits for each of the above
patrons (and keep the tabs open):
- Enable automatic renewal notices, and set them to digests only
- Add a value to the email field
4. Enter the kshell (ktd --shell)
5. Edit /etc/koha/sites/kohadev/koha-conf.xml, and add the following
lines near the end, just above the </config> and </yazgfs> closing
tags:
<auto_renew_cronjob>
<parallel_loops_count>2</parallel_loops_count>
</auto_renew_cronjob>
6. restart_all
Reproducing the issue:
7. Apply the test patch only
8. Run perl generate_checkouts.pl to generate test data
9. perl misc/cronjobs/automatic_renewals.pl -v -c
--> The test patch added output that will show how the renewals were
chunked into "chunk 0" and "chunk 1" for the two parallel loops.
Note that the issues for each borrower are not processed nicely in
one chunk, but are separated across multiple chunks and alternated
with other borrowers.
10. Check the checkouts for each of the four patrons from above
--> All checkouts should have renewed
11. Check the notices tab for each of the four patrons
--> Notice errors in the automatic renewal digest notices. A patron's
renewals may be split across multiple digests, a digest may be
missing renewals, or a patron may not have received a digest at all
Testing the patch:
12. Apply the second patch
13. Reset the due dates on all checkouts so that they will all be
eligible for automatic renewal again:
- koha-mysql kohadev
- UPDATE issues SET date_due=<two days from today>;
14. perl misc/cronjobs/automatic_renewals.pl -v -c
--> Note that the renewals are now correctly chunked by patron
15. Check the checkouts and notices tab for each of the four patrons
--> All checkouts should have renewed, and all patrons should have a
single new Auto Renewals Digest notice that correctly lists all of
their renewed items
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit: tidied the code block inline (tcohen)
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Olivier V <olivier.vezina@inLibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
1 - perl misc/migration_tools/bulkmarcimport.pl -b --file=notafile.mrc -v
2 - Output:
Characteristic MARC flavour: MARC21
0 MARC records done in 0.00804400444030762 seconds
3 - Apply patch
4 - perl misc/migration_tools/bulkmarcimport.pl -b --file=notafile.mrc -v
5 - Output:
Characteristic MARC flavour: MARC21
Could not open notafile.mrc: No such file or directory at misc/migration_tools/bulkmarcimport.pl line 262.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Script migration_tools/build_oai_sets.pl is missing ORDER BY
biblionumber.
This is a problem when using OFFSET and LIMIT, the results may be
differenly orderder between two calls of this script.
Test plan :
1) Create a OAI SET with all records
2) Run migration_tools/build_oai_sets.pl with the first half of records
3) Run migration_tools/build_oai_sets.pl with the second half of records
4) Check all records are in the OAI set
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch removes setting AutoCommit to 0 and commiting only every X records.
Instead we commit as we go and report progress using a parameter.
Bug 36474 reduced the numebr of changes that are being committed, so this should be a reasonable change. The
use of commits without transactions was causing problems if the library was active while the script ran.
To test:
1 - perl misc/cronjobs/update_totalissues.pl -c
2 - Script runs, but with unknown parameter
3 - perl misc/cronjobs/update_totalissues.pl -p 10
4 - Script runs and reports every 10 records
5 - per; misc/cronjobs/update_totalissues.pl
6 - Script runs and reports every 100 records by default
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a new check in UpdateTotalIssues to check that we are changing the number
of total issues before calling ModBiblio
To test:
0 - Enable CataloguingLog
1 - Checkout an item
2 - Run : misc/cronjobs/update_totalissues.pl --use-stats --commit=1000 -v
3 - In report, note all biblios were updated
4 - Check action_logs - note a new entry for every biblio
5 - Apply patch
6 - Repeat
7 - Note no biblios reported updated
8 - Note no new cataloguing log entries
9 - Checkout the item again
10 - Run again
11 - Note biblionumber has updated count in verbose output
12 - Note report only rpeort 1 biblio modified, the rest only processed
13 - Only one line added to action_logs
14 - Run it again
15 - Confirm no updates
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This change lazy-loads modules for setters in the
koha-preferences tool, so that getters are free to run super fast.
Between BZ 37657 and BZ 37682 we effectively eliminate the
overhead of running "get" or "dump" commands via the koha-preferences
tool.
Test plan:
1. time misc/admin/koha-preferences get SearchEngine
2. Note time is about .35 seconds
3. time misc/admin/koha-preferences dump
4. Note time is about .35 seconds
5. Create sysprefs.yml
---
marcflavour: MARC21
viewMARC: 1
6. time misc/admin/koha-preferences load -i sysprefs.yml
7. Note time is about .35 seconds
8. time misc/admin/koha-preferences set SearchEngine Elasticsearch
9. Note time is about 1 seconds
10. Apply patch
11. Repeat the koha-preferences commands above
12. Note that the "dump" and "get" commands run in about .09-.1
seconds. The "load" and "set" commands still take the same amount
of time as their behaviours haven't changed
13. misc/admin/koha-preferences set SearchEngine Elasticsearch1
14. koha-mysql kohadev
15. select * from action_logs where module = 'SYSTEMPREFERENCE'
order by action_id desc limit 5;
16. Note that the action log showing Elasticsearch1 update says
"interface"
of "commandline" and "script" of "koha-preferences"
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Katrin Fischer
Nick Clemens
Matt Blenkinsop
Jonathan Druart
Martin Renvoize
Kyle M Hall
Andreas Roussos
Matthias Meusburger
Alex Buckley
wainuiwitikapark
Magnus Enger
Thomas Klausner
Tomas Cohen Arazi
Emily Lamancusa
Fridolin Somers
David Cook