Commit Graph

96 Commits

Author SHA1 Message Date
Jonathan Druart 9fbff51f0a
Bug 39367: Automatic changes
Added head2 right before sub with missing POD

Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-23 09:35:14 +02:00
Koha Development Team d659526b5a
Bug 38664: Tidy the whole codebase
This commit is generated using:
  % perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-11 14:58:24 +01:00
Pedro Amorim 15803e4f29
Bug 36627: (QA follow-up): Rename borrowernumber -> patron_id
To test:
$ prove t/db_dependent/api/v1/erm_counter_logs.t

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-30 15:12:44 +01:00
Katrin Fischer 3b86455e02
Bug 36627: Fix typo "embeded"
A search turned up more of these in other files. Fixed as well.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-23 13:36:16 +01:00
Matt Blenkinsop 7b75586ead
Bug 36627: Rename embed method to patron
This patch renames the embed method from borrowernumber to patron to fit with the use of 'patron' in the API

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-23 13:36:14 +01:00
Matt Blenkinsop da475321a5
Bug 36627: Embed patron record for display
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-23 13:36:13 +01:00
Matt Blenkinsop 062e04c9c3
Bug 36627: Record who is importing sushi data
This patch adds functionality for storing the patron (or cronjob) that triggers a harvest. Currently every counter log is stored as being imported by the cronjob

Test plan:
1) In the usage statistics module in ERM, create a data provider (the details in the provider are not important)
2) Click into the provider and click into manual upload
3) Upload the file attached to this bug
4) Once the background job is completed click into the counter logs tab in the provider. The imported by columns should show a user with a clickable link

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-23 13:36:13 +01:00
Magnus Enger b62edd4177
Bug 33339: (Follow up) Fix more cases of formula=empty
This patch adds formula=empty to:
- Koha::ERM::EUsage::SushiCounter::_build_COUNTER_report_file()

Updated 2024-09-26: Earlier version also removed unnecessary
use of formula=empty from two places.

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-14 10:46:31 +01:00
Jake Deery 6ece137123
Bug 37577: Add additional fields for ERM packages
This bug adds support for additional fields in the ERM module's
packages, similar to how Bug 35287 added support for additional fields
in the ERM module's licences, and Bug 37576 added support for additional
fields in the ERM module's agreements.

To test (k.t.d):
a)  APPLY PATCH
b)  Run cypress --spec t/cypress/integration/AdditionalFields_spec.ts
    1)  Note how the tests pass
c)  Add new additional fields for 'erm_packages'
    1)  Go to /cgi-bin/koha/admin/additional-fields.pl?tablename=erm_packages
    2)  Create field one called 'text non-repeatable'
    3)  Create field two called 'text repeatable', check 'repeatable'
    4)  Create field three called 'av non-repeatable', pick LANG authorised
        value
    5)  Create field four called 'av repeatable', pick LOC authorised value,
        check repeatable
d)  Create new local package
    1)  Go to /cgi-bin/koha/erm/eholdings/local/packages/add, notice the new
        additional fields section
    2)  Complete all mandatory fields for a new package
    3)  Experiment with the four different additional field types – it
        may be useful to watch the Cypress-generated test video for
        guidance
    4)  Select some AV options from the AV fields, deselect them, notice
        the repeatable one allows for multiple selection, the non-repeatable
        one does not
e)  Save the package, then view it
    1)  To view the package, click its name in the table list on
        /cgi-bin/koha/erm/eholdings/local/packages
    2)  notice the additional fields on display
    3)  also notice that the authorised values are in their human-readable
        form, and not their coded form
f)  Edit the package, then view it again
    1)  Go to /cgi-bin/koha/erm/eholdings/local/packages/edit/1
    2)  Play around again with the fields. Save.
    3)  Repeat step e, have the edits shown up?
g)  Test searchability
    1)  Go to /cgi-bin/koha/admin/additional-fields.pl?tablename=erm_packages
    2)  Edit one of the fields and make sure searchable is checked (ensure
        you choose a field that has some data in it)
    3)  Go to /cgi-bin/koha/erm/eholdings/local/packages
    4)  Notice that the searchable field(s) are now columns in the table
    5)  Notice that searchable AV fields contain a drop-down
    6)  Notice that searchable text fields have an input field
    7)  Perform a search on the additional field
    8)  Notice that the searching is working as expected
h) SIGN OFF

Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-08 16:09:19 +01:00
Jake Deery f348afbae8
Bug 37576: Add additional fields for ERM agreements
This bug adds support for additional fields in the ERM module's
agreements, similar to how Bug 35287 added support for additional fields
in the ERM module's licences.

To test (k.t.d):
a)  APPLY PATCH
b)  Run cypress --spec t/cypress/integration/AdditionalFields_spec.ts
    1)  Note how the tests pass
c)  Add new additional fields for 'erm_agreements'
    1)  Go to /cgi-bin/koha/admin/additional-fields.pl?tablename=erm_agreements
    2)  Create field one called 'text non-repeatable'
    3)  Create field two called 'text repeatable', check 'repeatable'
    4)  Create field three called 'av non-repeatable', pick LANG authorised
        value
    5)  Create field four called 'av repeatable', pick LOC authorised value,
        check repeatable
d)  Create new agreement
    1)  Go to /cgi-bin/koha/erm/agreements/add, notice the new additional
        fields section
    2)  Complete all mandatory fields for a new licence
    3)  Experiment with the four different additional field types – it
        may be useful to watch the Cypress-generated test video for
        guidance
    4)  Select some AV options from the AV fields, deselect them, notice
        the repeatable one allows for multiple selection, the non-repeatable
        one does not
e)  Save the agreement, then view it
    1)  To view the licence, click its name in the table list on
        /cgi-bin/koha/erm/agreements
    2)  notice the additional fields on display
    3)  also notice that the authorised values are in their human-readable
        form, and not their coded form
f)  Edit the agreement, then view it again
    1)  Go to /cgi-bin/koha/erm/agreements/edit/1
    2)  Play around again with the fields. Save.
    3)  Repeat step e, have the edits shown up?
g)  Test searchability
    1)  Go to /cgi-bin/koha/admin/additional-fields.pl?tablename=erm_agreements
    2)  Edit one of the fields and make sure searchable is checked (ensure
        you choose a field that has some data in it)
    3)  Go to /cgi-bin/koha/erm/agreements
    4)  Notice that the searchable field(s) are now columns in the table
    5)  Notice that searchable AV fields contain a drop-down
    6)  Notice that searchable text fields have an input field
    7)  Perform a search on the additional field
    8)  Notice that the searching is working as expected
h) SIGN OFF

Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-08 16:09:17 +01:00
Pedro Amorim f1f33fce68
Bug 35287: (Follow-up 37389): ExtendedAttributes
Apply ExtendedAttributes Mixin to AdditionalFields

To test the ExtendedAttributes mixin search is working:
1) Create 2 or more searchable licenses additional fields
2) Create a new license and enter values in these searchable fields
3) Go to the licenses list, use the column filters to search on more
than one column, verify that the search work as intended (AND operator)

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-24 13:38:21 +02:00
Pedro Amorim 6037bd78ae
Bug 35287: Preparation: AdditionalFields License Mixin
Koha::Object::Mixin::AdditionalFields needs to come before Koha::Object
in the inheritance so that the strings_map method that is called is
the one from Mixin and not from Koha::Object.
This is only required for Koha classes implementing the AdditionalFields Mixin and expose data through the REST API, because strings_map is implement in both the aforementioned classes

Signed-off-by: Edith Speller <edith.speller@ukhsa.gov.uk>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-24 13:38:18 +02:00
Jonathan Druart 61b3aa2aa2
Bug 38177: Do not use stash to retrieve parameters
The pagination for EBSCO's titles, packages and resources is broken. We
shouldn't rely on $c->stash but use params from the request instead.

Test plan:
1) Setup all the system preferences in ERM:
<staff_url>/cgi-bin/koha/admin/preferences.pl?tab=erm
2) Enable EBSCO in ERMProviders. Set the ERMProviderEbscoApiKey and ERMProviderEbscoCustomerID prefs.
3) Do a search that returns more than 20 records. Click on a page other than 1. Notice the records do not change (Page is the same).
4) Apply patch. Repeat. Notice pagination now works.

Search titles, packages and resources
Use the pagination and modify the number of results per page

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-23 19:14:51 +02:00
Pedro Amorim fa6cd2f9b7
Bug 37810: Consider ServiceActive on status response
Some SUSHI providers return ServiceActive instead of the documented Service_Active:
https://countermetrics.stoplight.io/docs/counter-sushi-api/f0dd30f814944-server-status

This ensures the test connection does not fail regardless of what is used

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2024-10-02 10:17:18 +01:00
Matt Blenkinsop a1f0fc04d1
Bug 37526: Handle redirects in SUSHI requests
Test plan:
1) Apply test patch only. Run:
prove t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t
2) Verify tests fail.
3) Apply this patch. Run tests again. Verify they pass.

PA amended: Added test plan

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:42 +02:00
Nick Clemens b7600e4801
Bug 37856: (QA follow-up) Tidy
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:40 +02:00
Pedro Amorim 72b38a1dbf
Bug 37856: Consider absence of 'Report_Items'
The SUSHI provider in question for this patchset also does not return any
Report_Items entry. This patch accounts for that.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:38 +02:00
Pedro Amorim 7c175c2160
Bug 37856: Consider 'Exceptions' not being part of 'Report_Header'
When fixing the issue around the missing platform parameter, we
noticed this particular provider also returns 'Exceptions' as
its own thing, not part of 'Report_Header', despite documentation:
https://countermetrics.stoplight.io/docs/counter-sushi-api/7cp0h7n1hm2en-tr-j1-journal-requests-controlled

From the link above, search for 'Exceptions'. Notice it should be under 'Report_Header'.
Either way, this patch accounts for this possibility.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:38 +02:00
Pedro Amorim dc9a5684f4
Bug 37856: Append platform parameter to SUSHI url if defined
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:36 +02:00
Nick Clemens 5b51ef78d4
Bug 37963: (QA follow-up) Tidy
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:35 +02:00
Pedro Amorim 23ab84c9b6
Bug 37963: Case 2 - Consider providers returning error messages lower cased
Test plan:
1) Apply only test patch for case 1. Run tests:
prove t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t
2) Verify tests fail. Apply the other patch for case 1.
3) Run tests again. Verify they now pass.
4) Repeat the test plan but using patches for case 2

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:35 +02:00
Pedro Amorim 5d53bb4bff
Bug 37963: Case 1 - Return if sushi_errors even if response code >= 400
This ensures that SUSHI errors, if present, are shown on the UI even if
response->code >= 400.
This means that if, for example, SUSHI provider returns 401 with SUSHI
error like so:
{"Code": 2020, "Severity": "Error", "Message": "API Key Invalid"}

Then, the 'API Key Invalid' will be shown to the user, without having to
check logs.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-09-27 18:00:34 +02:00
Kyle M Hall 5a06bd0d73
Bug 37308: (QA follow-up) Tidy code
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-16 17:33:44 +02:00
Pedro Amorim 8175cccca2
Bug 37308: Add user-agent for SUSHI outgoing requests
Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-08-16 17:33:43 +02:00
Matt Blenkinsop d2820f57d6
Bug 34788: (QA follow-up)
This patch fixes a few things:
1) The blue dialog box now clears when navigating away from the page
2) The background job now uses skip_record_index to avoid queuing indexing jobs for every new biblio and instead queues one job at the end
3) Large files that get chunked now successfully create linked biblios if requested
4) Title matching rules have been expanded to check the package ID so that we can have duplicate titles in different packages
5) A link to the package is now included on the job report page

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:09 +02:00
Matt Blenkinsop 08992ab711
Bug 36618: Make biblio creation optional for ERM local titles
This patch makes biblio creation on local title creation optional.

Test plan:
1) Navigate to the local title creation form
2) Create a title and leave the "Create record" box unchecked
3) Submit the form
4) Search the catalogue for the title you have just created - no biblio result will be available
5) Repeat steps 2 - 3 but this time check the box to create a record
6) You should now be able to find a record in the catalogue for the title you just imported

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-08 20:04:07 +02:00
Pedro Amorim 9988925fb5
Bug 35418: Fix _get_usage_months
No longer falls in an endless loop if the last month in the report period is december.

Test plan:
1) Apply just the tests patch, run EUsage tests
prove t/db_dependent/Koha/ERM/EUsage/
2) Wait for a few seconds. Notice SushiCounter.t it hangs. CTRL+C (cancel it)
3) Apply the fix patch.
4) Run the tests again. Notice all EUsage tests pass with no issues.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Auld <andrew.auld@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-11-29 17:36:11 -03:00
Pedro Amorim 55f024c0e3
Bug 35115: (QA follow-up):
Spelling.
Tidyness.
Removal of leftover warn

Nick's patch fixes the issue as described, and I agree that it is the ideal solution here.
prove t/db_dependent/api/v1/erm_eholdings* passes
prove t/db_dependent/Koha/BackgroundJob/CreateEHoldingsFromBiblios.t passes

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-11-29 17:36:10 -03:00
Nick Clemens f984969690
Bug 35115: Move store routine from Resource to Title and update code
These patches move the store routine from Koha::ERM::EHoldings::Resource to Koha::ERM::EHoldings::Title as the code deals exclusively with title fields.

It updates the code to ensure that records are created when a title is not attched to a biblio, and that only the biblio title field is updated when updating an eholdings title.

To test:
 1 - Enable ERMModule sys pref
 2 - Create a new public list, visit:
     /cgi-bin/koha/virtualshelves/shelves.pl
 3 - Click "New list" enter name, set public -> public
 4 - Click "Add items", enter 112\n113\n114 (new line for each), in "Biblio numbers"
 5 - Notice that all added biblios have quite a few MARC fields
 6 - Go to packages, visit:
     /cgi-bin/koha/erm/eholdings/local/packages
 7 - Create a new package, add a name and hit 'Submit'
 8 - Go to titles, visit:
     /cgi-bin/koha/erm/eholdings/local/titles
 9 - Click "import from list"
10 - Pick the package created in 7)
11 - On the row of the list created in 2), click "Import"
12 - Go back to the list, visit:
     /cgi-bin/koha/virtualshelves/shelves.pl?op=view&shelfnumber=1
13 - Notice all the biblios have been stripped of their MARC data, and only title remains.
14 - Go to details page for one of the bibs
15 - Edit -> Replace record via Z39.50
16 - You can search for anything, just make sure to import a mostly full record
17 - Go back to ERM - eHoldings - Local - Titles
18 - Edit the title for the record that was replaced
19 - Save
20 - Go to record details - note the record is gone again
21 - Apply patches
22 - Search the catalog
23 - Check some titles
24 - Add to a new list
25 - repeat 8 - 11 with the new list
26 - View and confirm records are intact
27 - Edit the eholdings title for one of the records, changing the title field
28 - Save and view record details
29 - Confirm record is intact and title updated

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-11-29 17:36:09 -03:00
Nick Clemens 17cf580f19
Bug 34587: (QA follow-up) Tidy
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:32 -03:00
Pedro Amorim a35403cb84
Bug 34587: Add IR_A1 fields to COUNTER report
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:31 -03:00
Pedro Amorim c81112e798
Bug 34587: (QA follow-up): squash
Tidy
$c->validation in API controllers
File permissions
.inc file filters

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:30 -03:00
Pedro Amorim f7e2bc5d9b
Bug 34587: Move YOP to mus/yus instead of erm_usage_titles
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:29 -03:00
Pedro Amorim ee4db6d763
Bug 34587: Improve title uniqueness rules
Improve robustness of unique matching here, to make sure we always match for same title if the case,

Some report rows may correspond to the same title as the previous row but have an empty match field, or alternatively come with a filled match field (e.g. DOI or Print_ISSN in TR_J4. Because of this we only verify a uniqueness match field if both current row and previous have it non-empty, otherwise we keep checking the remaining uniqueness match fields.

Example of this use-case, COUNTER report:
title    | publisher | platform  | Proprietary_ID | Print_ISSN | DOI     | YOP  | usages
examplet | examplep  | examplepl | 1              | 123        |        | 2020 | usages
examplet | examplep  | examplepl | 1              | 123        |someDOI | 2021 | usages

The above 2 rows is the same title, same publisher, same proprietary_id,
same Print_ISSN, etc. It just so happens that one was returned by SUSHI
with DOI and the other wasnt.
These 2 rows correspond to different usage statistics by YOP, not 2
different usage_titles

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:27 -03:00
Pedro Amorim 184602ce38
Bug 34587: Add proprietary_id and platform to erm_usage_titles
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:26 -03:00
Pedro Amorim d55ea648ac
Bug 34587: Improve SUSHI COUNTER error handling
This is doing a few things, checking for sushi errors:
1) Abort background job and provide error message if SUSHI returned
   {Severity} in response, this means SUSHI error, docs here:
https://app.swaggerhub.com/apis/COUNTER/counter-sushi_5_0_api/5.0.2#/SUSHI_error_model

2) Abort background job and provide error message if SUSHI provided
   proper response, but contains Exceptions in Report_Header

3) Abort background job and provide error message if SUSHI returned 0
   report items (e.g. everything is fine but there are no items for the
   harvest period provided

4) Abort background job and provide error message if COUNTER file
   created is larger than allowed by the database.

5) Provide error message if manual upload file size exceeds the max
   allowed by the database

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:25 -03:00
Pedro Amorim e629e37bd7
Bug 34587: Fix total added objects in job report
Remove obsolete total_records code, this was a leftover from when this code was in UsageDataProvider.pm
Add a new job_callback 'added_usage_objects' to handle total objects added (titles, platforms, databases, items)
Update job report include file

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:24 -03:00
Matt Blenkinsop 7ac6b1b72c
Bug 34587: Add report type and provider name to failed background job report
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:23 -03:00
Matt Blenkinsop d4afc38c7c
Bug 34587: Add try catch blocks to object creation
Currently a harvest fails completely if a single title fails to harvest. Rather than failing the entire job, this patch introduces try catch blocks to stop this and allow the job to continue. Failed imports are recorded and displayed in the background job UI on job completion

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:22 -03:00
Matt Blenkinsop 979ff589d9
Bug 34587: Remove counter files from API embedding to improve performance
Rather than fetching the counter files and embedding the counter logs, we now add a foreign key to the data provider in the counter logs table and fetch them directly.

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:21 -03:00
Matt Blenkinsop ebc98070d6
Bug 34587: Fix url validation
We need to check whether the url ends in a trailing "/" and also whether it ends in "/reports" for a SUSHI harvest. Two new functions are added to manage this and prevent failed harvests.

report_type is also now lowercase in the url and the dates are truncated to YYYY-MM format

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:19 -03:00
Pedro Amorim 3ab4036636
Bug 34587: New sushi counter class
Moves a lot of the logic for creating a counter file and parsing a SUSHI response to a new class to simplify logic

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:19 -03:00
Pedro Amorim ca8ea02fda
Bug 34587: Move everything into a ERM/EUsage subfolder
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:18 -03:00
Matt Blenkinsop 89b6d51f07
Bug 34587: Improve error handling for connection testing
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:16 -03:00
Pedro Amorim cf87082682
Bug 34587: File upload - Background job
ErmSushiHarvester background job will now be initiated by either
enqueue_counter_file_processing_job or by enqueue_sushi_harvest_jobs
from Koha/ERM/UsageDataProvider.pm, the former if triggered by a manual
file upload, the latter if by the 'run now' button or by the cron script.

This commit also includes some rewording/refactoring, namely:
- COUNTER file validation now happens in the API, before enqueuing the job.
- Removal of no longer used POST /erm/counter_files endpoint
- Koha/ERM/UsageDataProvider.pm:
-- run method is now enqueue_sushi_harvest_jobs
-- new enqueue_counter_file_processing_job method
-- harvest method is now harvest_sushi
-- new set_background_job_callbacks method to set the background job callbacks
- REST/V1/ERM/UsageDataProviders
-- run method is now process_SUSHI_response
-- new process_COUNTER_file endpoint

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:15 -03:00
Matt Blenkinsop e5ec3df36e
Bug 34587: Code tidying
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:14 -03:00
Pedro Amorim 16592d267a
Bug 34587: Accept date inputs upon confirming 'run now' button
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:13 -03:00
Pedro Amorim 3318adf48d
Bug 34587: Add a report row for each existing metric_type in a sushi result's performance instance, instead of for reach existing metric_type in the sushi response header, because that might not exist.
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:12 -03:00
Matt Blenkinsop 8987ff1591
Bug 34587: Abstract reports backend to allow new data types
Now that harvesting is possible for platforms, databases and items we need to be able to generate reports for all of these data types. Currently the reporting backend structure is very geared towards titles. Rather than copying this for each different data type, this patch abstracts the code to accept the data type as a url parameter and use that to generate a report based on a given data type

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:07 -03:00
Pedro Amorim 13356ae74e
Bug 34587: COUNTER file upload
Adds the ability to manually upload a counter file and harvest data to the provider

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:06 -03:00