StageMARCForImport:
- Rollback in catch
- Setting progress, size or status after BatchStageMarcRecords
in both try and catch block
ImportCommitBatch:
- Move setting size back to enqueue moment
- Rollback in catch
- Setting progress, size or status after BatchStageMarcRecords
in both try and catch block
ImportRevertBatch:
- Move setting size back to enqueue moment
- Adding transaction/rollback to module since import routine
does not support it. Could be moved later.
- Setting progress, size or status after BatchStageMarcRecords
in both try and catch block
Test plan:
Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t
Test staging file
Bonus: Put a die statement in BatchStageMarcRecords.
Test importing batch
Bonus: Include some records with an invalid library code; this will
trigger an FK exception. (Reduce the progress from 50 to 1. If your
first record would be fine, check if it is NOT imported when the job
fails.)
Test reverting batch.
Bonus: Put a die in BatchRevertRecords.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When you submit a background jobs, and it fails, you do not expect
partial results in the database.
Note that when the Background feature would support a partially
completed status, things might change again.
Note that the >0 test was superfluous if you check for ^\d+$.
Test plan:
Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t
Note: This serves to verify that it still runs as expected.
The test plan of the following patch covers the new param.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
Look at job results when reverting imported batch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
Run an stage and import.
Check import commit job.
Click to managed batch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Making the disabling utf8 flag explicit instead of depending on
the default of the CPAN module.
Incorporating the change in background_jobs_worker too.
Test plan:
See next patches when we look at unit tests.
Restart koha-worker.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Our database connections have been set up so that they will
automatically convert perl encoded strings to utf8 encoded strings
when sending data to database tables and decode the utf8 encoded
strings from the tables back to internal perl strings. As we can see
from a call in Koha/BackgroundJob.pm we are encoding the perl internal
string to utf8 and then decoding it back to perl internal string:
my $data_dump = decode_json encode_utf8 $self->data;
We can skip this unnecessary encode<->decode step (as the database
object has done the decoding for us) by simply calling the
JSON->new->decode() method which doesn't perform any string decoding.
Furthermore, the original code was buggy and didn't always remember to
encode the unencoded strings, in Koha::BackgroundJob::process we can
see
my $context = decode_json($self->context);
is missing the encode step. Now after this change encoding before
decoding is not necessary as we are using the methods from the JSON
module that do not perform any transcoding.
Note to those concerned whether the old data in the database is
compatible with this new code:
Luckily our database connection object seems to be smart and
didn't utf8 encode the utf8 returned data from the old encode_json()
calls (probably checks the utf8 flag for the string (Encode::is_utf8($str))).
To test whether this fixes the original bug reported of not being able
to schedule background jobs with koha user having non-ASCII letters in
their surname:
1) Change your staff users surname/lastname to "ääää"
2) Log out and back in.
3) Go to a biblio record detail page and click "Select all" in the
items table
4) Click Delete selected items and proceed with the deletion
5) Notice the batch item deletion job has failed status
6) Apply patch and repeat but this time the deletion job should finish.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Andrew <andrewfh@dubcolib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
JD Amended patch:
-# FIXME Special case here
- print "Biblio not found\n,";
+ print "Biblio not found\n";
- my $biblio = Koha::Biblio->find($hostbiblionumber);
+ my $biblio = Koha::Biblios->find($hostbiblionumber);
Rebased-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
job's size is the number of records to index, but when finished the progress is always set to 1.
Wich leads to: finished 1/42
Test plan:
Don't apply this patch
Use the batch item modification tool to update more than 1 item
Notice that the new indexing job has a size of the number of items you
modified
When finished, the job detail shows "Finished. 1/X"
Apply this patch, repeat
Notice that now the progress shows 0/1 before the job is done, then 1/1
when finished
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch makes the places in which Koha enqueues holds queue for
real time updates verify the feature is enabled.
To test:
1. Apply this patches up to the unit tests
2. Run:
$ updatedatabase
$ kshell
k$ git diff origin/master --name-only | grep -e '\.t$' | xargs prove
=> FAIL: tests fail, the code doesn't care about the syspref
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass!
5. Be happy!
6. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch makes the background job only enqueue holds queue updates
once per biblio, when appropriate.
To test:
1. Apply the regression tests patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/BackgroundJobs/BatchDeleteBiblio.t
=> FAIL: It enqueues 3 times for a bib!
3. Apply this patch
4. Repeat 2
=> SUCCESS: It only enqueues once!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch restores indexing for item only bulk deletions.
JD amended patch:
* remove trailing whitespaces
* Move if inside first condition
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
We are sending the request when all the items have been deleted.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
When using the batch record modification tool to modify several
bibliographic records, we don't want to send one index request per
biblio, we want to index them all on the fly after the records have been
modified.
Otherwise we will end up with one task per record, and records will be
indexed in background.
Test plan:
Use the batch mod tool to modify bibliographic records and confirm the
behaviour is correct.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Same as bug 30465 for authorities
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
No need to send 1 indexation request per item + 1 per biblio.
Test plan:
Confirm that the batch biblio deletion tool still works correctly
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Same as bug 30460 for authorities
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
We set biblio_ids in enqueue then referred to biblio_id in process ;)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch adds a background job that takes care of updating the holds
queue rows for a given list of biblios.
It is designed to be used for real-time batch updating the holds queue
when required. Tasks would be added by the relevant methods under the
right use cases.
Sponsored-by: Montgomery County Public Libraries
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch makes the ES indexing task record progress properly.
To test:
1. Have ES setup
2. Edit a record making any change that would trigger indexing
3. Go to Administration > Background jobs
=> FAIL: The task status is 'Finished' but progress shows '0/1'.
4. Apply this patch
5. Restart the default worker:
$ koha-worker --restart kohadev
6. Repeat 2 and 3
=> SUCCESS: Things work the same, but progress is '1/1'
7. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a background job submodule, UpdateElasticIndex, to deal
with async ES index update (not the deletion).
Using NYTProf (on a checkin):
Without
618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals.
With
521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals.
However there are some problems with this patch:
1. We don't want *all* the index update to be in the background_jobs
tabtle (we could add a filter on the list view)
2. We don't track the "progress" of the job as we are sending all the
records to Elastic. It is okish in my opinion but it must be noted.
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Use of inherited AUTOLOAD for non-method Koha::BackgroundJob::BatchUpdateAuthority::ModAuthority() is
no longer allowed
Test plan:
Use the batch authority tools to edit authority record and confirm that
the task processes correctly
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch renames the queues so the default is the **real-time** one, and
the other (*long_tasks*) is kept for **long running** tasks.
All current *batch* tasks are explicitly assigned to the **long_tasks**
queue as well.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Use of inherited AUTOLOAD for non-method Koha::BackgroundJob::BatchUpdateAuthority::ModAuthority() is
no longer allowed
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Now $self is actually an instance of the job class, there's no need to
have the job_id parameter passed, or the have the ->process method
re-fetch the object from the database.
This patch cleans things up.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
and some more...
There are lot of inconsistencies in our ->search calls. We could
simplify some of them, but not in this patch. Here we want to prevent
regressions as much as possible and so don't add unecessary changes.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch reuse the (awesome) Koha::Result::Boolean module to retrieve
the return of Koha::Item->safe_to_delete.
Test plan:
Try to delete an item that has previously been checked out and confirm
that you are still blocked.
Try using the cronjobs, the item and biblio detail pages, as well as the
batch delete item tool.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
If an exception is raised by one the methods/subroutines called when batch biblio update, the job will fail in the middle and some records won't be processed.
Test plan:
0. Don't apply the patch
In KTD you can run the a batch mod biblio on the 100 first records and
the worker will fail with
"""
countered object 'C4::Biblio::_koha_modify_biblioitem_nonmarc(): DBI Exception: DBD::mysql::st execute failed: Data too long for column 'lccn' at row 1 at /kohadevbox/koha/C4/Biblio.pm line 423
', but neither allow_blessed, convert_blessed nor allow_tags settings are enabled (or TO_JSON/FREEZE method missing) at /kohadevbox/koha/Koha/BackgroundJob/BatchUpdateBiblio.pm line 120.
"""
The UI is not saying anything about this problem.
Generate 100 biblionumbers: perl -e 'for (1..100){print "$_\n"}'
1. Apply the patch and confirm than now you see the detail of the batch
update and that biblionumber 72 is marked as not been modified with the
correct error (raw DBI error)
JK: add executable bit to BatchUpdateBiblio.t
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch renames the (passed through) 'context' param for
'overlay_context'. I propose doing so, because in Koha-land 'context'
has a special meaning, related to C4::Context and it reads ambigous.
The patch itself is pretty trivial.
Tests should pass:
1. Run:
$ kshell
k$ prove t/db_dependent/Biblio/MarcOverlayRules.t
=> SUCCESS: Tests pass
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests still pass!
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 14957: (follow-up) Clarify 'context' param
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a rule based system for merging MARC records to for example
prevent field data from being overwritten.
To test:
1. Apply this patch.
2. Log in to staff client.
3. Enable new syspref MARCMergeRules.
4. Click the new link "MARC merge rules" in the "Catalog"
section of the Koha administration page.
5. Create a new rule:
Module: source, Filter: *, Tag: 245, Preset: Protect.
6. Clicking "Edit" should allow you to edit corresponding rule.
7. Clicking "Delete" should remove corresponding rule after confirmation.
8. Selecting one or more rules followed by clicking "Delete
selected" should remove all selected rules after confirmation.
9. Try creating a rule with tag set to "**", the other options does
not matter. Verify that saving this rule produces an error
message complaining about invalid tag regular expression.
10. Try creating a rule with tag set to "008" (or other control
field) and set Appended: Append and Removed: Skip, the other
options does not matter. Verify that saving this rule produces
an error message complaining about invalid combination of actions
for control field.
11. With the 245 rule in step 5 in place, edit a bibliographic record,
change 245a for example (which should be Title for MARC21) and save.
12. Verify that the changes has not been saved.
13. Create a new rule:
Module: source, Filter: intranet, Tag: 245, Preset: Overwrite.
14. Repeat step 12, and verify that the changes has now been saved.
15. Run tests in t/db_dependent/Biblio/MarcMergeRules.t and very
that all tests pass.
Sponsored-by: Halland County Library
Sponsored-by: Catalyst IT
Sponsored-by: Gothenburg University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Christian Stelzenmüller <christian.stelzenmueller@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Tomas Cohen Arazi
Marcel de Rooy
Jonathan Druart
Joonas Kylmälä
Nick Clemens
Martin Renvoize
Fridolin Somers
Andrew Nugged
David Gustafsson
David Gustafsson