Large files were failing to enqueue/process because the whole CSV was
embedded in the job's JSON args, hitting DB/broker size limits. This
now uploads the file via Koha::Uploader first (same pattern as MARC
staging) and only passes a small uploaded_file_id through the job,
and fixes the enqueue-failure message to not claim nothing happened
when a job was in fact created.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Introduces a new background job class for patron imports. enqueue() reads
the uploaded file content into the job_args and uses line count as job_size
for progress tracking. process() reconstitutes the file handle from the
stored content, delegates to Koha::Patrons::Import with step_callback wired
to $self->step, handles patron list creation, stores the import results, and
clears file_content from the stored data to free space.
Also registers patron_import in Koha::BackgroundJob::core_types_to_classes
so _derived_class can resolve the job correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
This patch displays the directory tested in the background jobs results
It updates the language when test results are not stored for the transport and
links to the job when results are stored
To test:
1 - Setup an ftp file transport with good credentials
2 - Enter non-existent directories in incoming/outgoing
3 - Save
4 - Verify on save 'Never used' shows for status
5 - Refresh the page, status now:
Connection ok
Change directory ok
Change directory failed
Change directory ok
Change directory failed
6 - Apply patch, restart all
7 - Edit/save the transport again, change nothing
8 - Confirm the page says 'No test results stored'
9 - Reload the page
10 - Confirm results are now:
Connection ok
Change directory ok
Change directory failed
Change directory ok
Change directory failed
Test job details
11 - Click on the job details link
12 - Confirm actual directory and errors are shown
13 - Edit the transport, use existing directories
14 - Save, reload
15 - Confirm the success results and link t background job
Signed-off-by: John Vinke <jvinke@austinseminary.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Three searches pass large ID lists:
* Koha::BackgroundJob::BatchUpdateItem passes every itemnumber of the
batch, and batch item modifications can cover tens of thousands of
items
* Koha::Items::batch_update does the same with the modified
itemnumbers when collecting biblionumbers for reindexing
* Koha::Patrons::filter_by_safe_to_delete does it with the
borrowernumbers found safe to delete ( unlikely to approach
the same numbers as items/bibs, but still good to fix )
This patch switches all three to IN. The lists are still passed as
values, but the query stays a flat IN list the optimizer handles well
instead of a chain of ORs that can't be optimized.
Test Plan:
1) Apply this patch
2) prove t/db_dependent/Koha/Items/BatchUpdate.t
3) prove t/db_dependent/Koha/Patrons.t
4) Note all tests pass!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Always include the plural version, not the singular one (the plural
imports the singular)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
This patch replaces all calls to C4::Stats::UpdateStats with the wrapped
function from the Koha namespace. As this is the last function in C4::Stats, the
module is also deleted.
Several tests depended on transitive includes to Koha::Statistics through
C4::Stats. I've identified uses of that module by searching, and added the
necessary use in each case. I've run `prove -r t xt` with this patch applied and
didn't hit any relevant errors.
Test plan:
1. Inspect the patch, and verify each changed call maintains the same parameters.
2. Apply patch.
3. Prove the following files:
- t/db_dependent/Accounts.t
- t/db_dependent/Circulation.t
- t/db_dependent/Koha/Account.t
- t/db_dependent/Koha/ILL/Request.t
- t/db_dependent/Koha/Item.t
- t/db_dependent/Koha/Pseudonymization.t
- t/db_dependent/Koha/Recalls.t
Sponsored-by: Open Fifth
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
When RealTimeHoldsQueueUnallocated is enabled, the background job
passes unallocated => 1 and delete => 0 to update_queue_for_biblio.
This skips deleting existing queue entries and only processes holds
not already allocated, matching the --unallocated behavior of
build_holds_queue.pl.
Test plan:
1. Apply patches
2. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblioHoldsQueue.t
=> SUCCESS: Tests pass!
3. Tests verify:
- delete => 1 and no unallocated when pref is off
- delete => 0 and unallocated => 1 when pref is on
4. Sign off :-D
Signed-off-by: Alex Carver [Acerock7] <alex@rcls.org>
Sponsored-by: The Main Library Alliance <https://www.mainlib.org/>
Signed-off-by: Alex Carver [Acerock7] <alex@rcls.org>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Currently, a job is marked failed if there are no records added or updated.
We should take ignored records into account.
Additionally, I mark the job failed when the number of records processed
doesn't equal the number of records in the batch.
To test:
1) Export some records from your Koha, include some with items
2) Stage and import those records into Koha, matching on biblionumber,
ignoring matcheds, updating items
3) View the import job, it is marked failed
4) View the batch, all records imported
5) Apply patch, restart all
6) Stage/import as before
7) Job is now marked finished
8) Records imported as before
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To test:
Enable pseudonymization, may need to add bcrypt settings (see bug 28911)
Perform some circulations
View the jobs (Admin->Manage jobs)
Filter for Pseudonymize statistic
Note the progress is 0/1 even when job finished
Apply patch, restart all
Do more circulations
Confirm jobs now say 1/1 complete
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We now use background jobs triggered by Transport->store to update the
connection status.
Status is now a json blob with more details, so we update the api
definition to reflect that.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch moves logic out of the background job file and into the Koha::ERM::EHoldings::Title class where it should belong. Methods are also renamed to differentiate between object methods and helper methods
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch addresses CSV formula injection vulnerabilities by ensuring
all Text::CSV instances use formula protection.
Changes made:
* Add formula => 'empty' to all Text::CSV constructors in:
- Koha::Patrons::Import
- Koha::BackgroundJob::ImportKBARTFile
- Koha::ERM::EUsage::CounterFile
The 'empty' formula mode causes potentially dangerous formulas (starting
with =, +, -, @) to be treated as empty strings rather than executed,
preventing CSV injection attacks while maintaining data integrity.
Security impact:
* Prevents execution of malicious formulas in CSV imports
* Protects against data exfiltration via CSV formula injection
* Maintains backward compatibility for legitimate CSV data
To test:
1. Apply the previous patch (author test)
2. Run:
$ ktd --shell
k$ prove xt/author/Text_CSV_Various.t
=> FAIL: Should show 3 files without formula protection
3. Apply this patch
4. Repeat 2
=> SUCCESS: All tests should now pass
5. Try importing CSV with formula content (=cmd|"/c calc",test,data)
=> SUCCESS: Formula content is rejected/emptied, not executed
6. Import normal CSV data
=> SUCCESS: Normal data imports correctly
7. Sign off :-D
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
- Add a new subscription to a serial record (with the option 'create an item record' selected)
- Receive several issues and create the linked items
- Return to the bibliographic record
- Select several items
- Click to 'Delete selected items'
- Tick the box 'delete serial issues linked to items' and confirm
- The items should be deleted correctly
- Back to subscription : the issues linked to items should no longer exist
Signed-off-by: Boubacar OUATTARA <boubacar.ouattara@biblibre.com>
Signed-off-by: Nathalie <nathalie.savarit@univ-amu.fr>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This new method name better reflects the fact that it now creates - and stores - the PseudonymizedTransaction object
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
Run the tests before and after applying this patch:
perl t/db_dependent/Koha/Pseudonymization.t
They should fail before this patch and pass after applying patch.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The unit tests highlighted that without passing an frameworkcode to the
MarcSubfieldStructures->find call we throw errors as we can't identify
the correct subfield structure rules to apply.
This patch adds the default frameworkcode, empty string, to the lookup.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Follow the same test plan as above, but attempt to add an item without having at least homebranch, holdingbranch and type subfields.
Notice you now get an error with an appropriate message and the item is not added.
Signed-off by: Chris Rowlands <chris.rowlands6@nhs.net>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan, clean k-t-d:
Before test, pick a record and confirm it has no items, for this test we're using biblio 79:
<staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79
1) Add a new MARC modification template, visit:
<staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl
2) Click 'New template', enter a name and click 'Submit'
3) Click 'New action'
4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value
5) Click 'Add action'
6) Do a batch record modification, visit:
<staff_url>/cgi-bin/koha/tools/batch_record_modification.pl
7) Select the tab 'Enter a list of record numbers' and enter '79'
8) On the template, select the template previously created. Click 'continue'
9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors
10) Check the biblio, visit:
<staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79
11) Notice it now has an item with homebranch 'CPL'
test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made
test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made
test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown
test with no 952 mapping -> confirm things work as intended
test with 952 mapping plus other tags -> confirm things work as intended
prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t
prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t
Signed-off by: Chris Rowlands <chris.rowlands6@nhs.net>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The ud_provider->{report_type} is only defined on ->harvest_sushi, meaning it'll never be defined here.
We don't need this anyway. The filename should be the filename of the file being manually uploaded. We only need this concat logic for COUNTER files being generated by Koha following a SUSHI harvest, and that works.
This also gets rid of a warning when running the tests
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
In fact the system adds a previous value to the progress. As if the system keeps a previous value and adds the correct count of records in addition.
This patch removes the previously stored value.
To test:
1. Go to Cataloging > Stage records for import
2. Choose a file with bibliographic records and click Upload file
3. In "Look for existing records in catalog?" choose a record matching rule (e.g. ISBN)
4. Click Stage for import
5. Click View detail of the enqueued job
--> Progress show 2/1
6. Apply the patch
7. Repeat steps 1 to 4
8. Click View detail of the enqueued job
--> Progress show 1/1
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
qa script:
Commit title does not start with 'Bug XXXXX: ' - d1bebb34d7
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Fixed patch subject line
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch removes the logged warning if a file parse is successful.
Test plan:
1) prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
2) The output will have a logged message - 'EOF - End of data in parsing input stream'
3) Apply patch
4) Repeat step 1
5) The warning will be gone
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch fixes a few things:
1) The blue dialog box now clears when navigating away from the page
2) The background job now uses skip_record_index to avoid queuing indexing jobs for every new biblio and instead queues one job at the end
3) Large files that get chunked now successfully create linked biblios if requested
4) Title matching rules have been expanded to check the package ID so that we can have duplicate titles in different packages
5) A link to the package is now included on the job report page
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch allows a file with additional columns to be imported. When the file is submitted, the system will enqueue the background job and send back information to the UI that certain columns have been ignored. The data will stil l import as normal but only the standard KBART columns will be parsed and imported
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch rebases in the changes from bug 36618 to make biblio creation optional
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit allows CSV files to be imported alongside TSV files.
It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes
Test plan:
1) Enable the ERM module
2) Navigate to E-resource management > eHoldings > Local > Packages
3) Create at least one package
4) Navigate to E-resource management > eHoldings > Local > Titles
5) There should be a button for "Import from KBART file"
6) Click this button
7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button. I have attached some example files to the bug but feel free to use your own if you have them.
8) Click Submit
9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file
10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected
11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs)
12) It should display a progress bar followed by a report and any error messages
13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles.
14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a background job that will import a KBART file
Sponsored-by: UKHSA
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Foundation for removing same check elsewhere (later on).
Test plan:
Run t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblioHoldsQueue.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
PA amended: Fix bug # in commit message
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
If the user unchecks the skip checkbox (on by default), we must
make sure that order cancellation is done in background job.
Test plan:
Pick two biblios. One has linked open orders, the other not.
Go to batch delete records. Select 'Enter list of numbers'.
Enter both biblio numbers and check that only one is used on the
follow-up form.
Run the deletion without the skip open orders and verify
that linked order line is cancelled.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
Run background job tests. Especially the changed one.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
All test pass when running this:
prove t/db_dependent/Koha/BackgroundJobs*
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new background job for pseduonymization
To test:
1 - Download the benchmarking script
2 - Ensure your koha has bcrypt_settings in koha-conf.xml
See bug 28911
3 - perl perf_check_pseudo.pl
4 - Note the slowdown after pseudonymization enabled
5 - Apply patches, restart all
6 - perl perf_check_pseudo.pl
7 - Note improvement
8 - Enable pseudonymization in sytem preferences
9 - Perform some checkouts and returns
10 - Verify the background jobs complete successfully
11 - Verify the pseudonymized_transactions table is updated correctly
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This is doing a few things, checking for sushi errors:
1) Abort background job and provide error message if SUSHI returned
{Severity} in response, this means SUSHI error, docs here:
https://app.swaggerhub.com/apis/COUNTER/counter-sushi_5_0_api/5.0.2#/SUSHI_error_model
2) Abort background job and provide error message if SUSHI provided
proper response, but contains Exceptions in Report_Header
3) Abort background job and provide error message if SUSHI returned 0
report items (e.g. everything is fine but there are no items for the
harvest period provided
4) Abort background job and provide error message if COUNTER file
created is larger than allowed by the database.
5) Provide error message if manual upload file size exceeds the max
allowed by the database
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Remove obsolete total_records code, this was a leftover from when this code was in UsageDataProvider.pm
Add a new job_callback 'added_usage_objects' to handle total objects added (titles, platforms, databases, items)
Update job report include file
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
ErmSushiHarvester background job will now be initiated by either
enqueue_counter_file_processing_job or by enqueue_sushi_harvest_jobs
from Koha/ERM/UsageDataProvider.pm, the former if triggered by a manual
file upload, the latter if by the 'run now' button or by the cron script.
This commit also includes some rewording/refactoring, namely:
- COUNTER file validation now happens in the API, before enqueuing the job.
- Removal of no longer used POST /erm/counter_files endpoint
- Koha/ERM/UsageDataProvider.pm:
-- run method is now enqueue_sushi_harvest_jobs
-- new enqueue_counter_file_processing_job method
-- harvest method is now harvest_sushi
-- new set_background_job_callbacks method to set the background job callbacks
- REST/V1/ERM/UsageDataProviders
-- run method is now process_SUSHI_response
-- new process_COUNTER_file endpoint
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This commit is a squash of the following:
SUSHI harvesting process in the data providers class:
* Builds the URL query and requests the SUSHI service endpoint
* Parses the JSON response and builds the csv COUNTER file and adds it to counter_files table
Usage statistics data processing:
* When a counter_files entry is stored, CounterFile.pm will:
* Parse the csv COUNTER file and
* Add a usage_titles entry for each unique title in the COUNTER file
* Add the title's respective erm_usage_mus (monthly usage) entries, repeating for each metric_type
* Add the title's respective erm_usage_yus (yearly usage) entries, repeating for each metric_type
Harvesting cronjob;
'Run now':
* API endpoint to start the harvesting process of a data provider
* Button in the data providers list to run the harvesting process for each data provider upon clicked
ERM SUSHI: Background job
Job progress is updated to total amount of usage titles after retrieving
the response from SUSHI;
Job warning and success messages are added accordingly
Redundant duplicate titles will not be added
Redundant duplicate monthly and yearly usage statistics will not be added
Data provider harvest background job harvests once per report_type
Enqueue one background job for each report_type in the usage data provider
Update the way we measure progress in the background job.
It now uses the COUNTER report body rows instead of SUSHI response results.
We're now incrementing and showing the number of skipped mus, skipped
yus, added mus and added yus
There's a bug in the way we calculate yus
Updates to background job progress bar - Depends on 34468
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds a missed parameter (profile_id) in stage-marc-import.pl and also adds a missing import in the background job for staging marc files. This means that the profile used when importing can now be shown in the import batches table and also in batch details
Test plan:
1) Stage a record for import and make sure to save the import profile you use
2) Import a record using that profile
3) Navigate to Cataloging > Manage staged MARC records
4) In the Profile column, no value will be shown for the staged record
5) Click on the batch, the profile will be missing in this screen as well
6) Apply patch
7) Restart_all
8) Repeat steps 2-5, the profile name should be visible in both places
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Emmanuel Bétemps <e.betemps@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Pedro Amorim
Nick Clemens
Kyle M Hall
Jonathan Druart
Olivia Reynolds
Tomás Cohen Arazi
Martin Renvoize
Matt Blenkinsop
Lucas Gass
Pedro Amorim
Koha Development Team
Hinemoea Viault
Marcel de Rooy