Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Consider 'ill' as a category and 'illreq_created' and 'illreq_comp' as allowed types
Signed-off-by: Jeremy Evans <Jeremy.Evans@ukhsa.gov.uk>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Cleanup_database can delete all statistics entries more than X days
old. If one is using pseudonymization to create pseudonymized_transactions
data, then one may wish to use that cleanup_database function to delete
statistics entries that have been duplicated in pseudonymized_transactions.
However, not all types of transactions in statistics are duplicated in
pseudonymized transactions.
Pseudonymized_transactions currently only includes checkouts, returns,
and renewals.
This patch adds two additional parameters to cleanup_database.pl:
1. --statistics-type
Defines the types of statistics to purge. Will purge all types if
parameter is omitted. Repeatable.
2. --statistics-type-pseudo
Grabs values from @Koha::Statistic::pseudonymization_types. At the
time of writing this patch, they are:
renew, issue, return and onsite_checkout
To test:
1. prove t/db_dependent/Koha/Statistic.t
2. Before applying this patch:
3. Create some statistics entry by checking out, renewing and checking in
items.
4. perl misc/cronjobs/cleanup_database.pl --statistics 1 --verbose
Observe:
Purging statistics older than 1 days
n statistics would have been removed
Where n is the amount of statistics rows matching your test environment
5. Apply patch
6. Repeat step 4, observe same result
7. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --verbose
Observe:
Purging statistics older than 1 days with types "onsite_checkout,renew,return,issue".
n statistics would have been removed
Where n is the amount of statistics rows matching your test environment
(order of listed types does not matter)
8. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type test1 --statistics-type test2 --verbose
Observe:
Purging statistics older than 1 days with types "test1,test2".
n statistics would have been removed
Where n is the amount of statistics rows matching your test environment
(order of listed types does not matter)
9. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --statistics-type test1 --verbose
Observe:
Purging statistics older than 1 days with types "test1,onsite_checkout,renew,return,issue".
n statistics would have been removed
Where n is the amount of statistics rows matching your test environment
(order of listed types does not matter)
10. Try previous cleanup_database.pl commands with --confirm flag and make sure correct rows are purged
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Enable Pseudonymization in system preferences
NOTE: See bug 28911 for bcrypt setup
2 - Issue an item to a patron
3 - View the patrons checkouts
4 - Check the box under 'Renew'
5 - Renew selected items
6 - Internal server error
7 - Apply patch
8 - Restart all
9 - Repeat 4&5
10 - Success!
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch does what the title says. This behavior is tested on the
previous patch. Tests are also added for the background job (which
relies on Koha::Statistic->new and was the evidence of this
overwrite behavior).
To test:
1. Apply the regression tests patch
2. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/Statistic.t \
t/db_dependent/Koha/BackgroundJob/PseudonymizeStatistic.t
=> FAIL: Tests don't pass :-/
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass!
5. Sign off :-D
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new routine to pseudonymize a statistic and adjusts
C4::Stats to use this new routine.
Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for
the existence of objects before using them (in the case of older stats where things may have been deleted)
Tests are added and can be run using:
1. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \
t/db_dependent/Koha/Pseudonymization.t
=> SUCCESS: New tests pass, old tests keep passing
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new background job for pseduonymization
To test:
1 - Download the benchmarking script
2 - Ensure your koha has bcrypt_settings in koha-conf.xml
See bug 28911
3 - perl perf_check_pseudo.pl
4 - Note the slowdown after pseudonymization enabled
5 - Apply patches, restart all
6 - perl perf_check_pseudo.pl
7 - Note improvement
8 - Enable pseudonymization in sytem preferences
9 - Perform some checkouts and returns
10 - Verify the background jobs complete successfully
11 - Verify the pseudonymized_transactions table is updated correctly
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Might come back on the subject when resuming with bug 33636.
Test plan:
Run t/db_dependent/Koha/Statistics.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Tidy Koha/Item.pm bit further. Ordered hash keys too.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We should default to C4::Context->interface.
Test plan:
Login via intranet (see Bugzilla report).
Edit an item. Set to lost.
Check entry in statistics table on interface value.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Adding exceptions, removing croaks.
No exception in new for unknown hash keys, store will catch that.
Prepare switching amount parameter to value (db column name).
Test plan:
Run t/db_dependent/Stats.t
Run t/db_dependent/Koha/Items.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
[EDIT] Fixed the mandatory check in Stats.t. Removed key_or_default.
Additional tidy.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Moving code to Koha::Statistic->new and ->insert.
Polishing code a bit further in next patch.
Test plan:
Run t/db_dependent/Stats.t
Run t/db_dependent/Koha/Statistics.t
Run t/db_dependent/Koha/Pseudonymization.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
[EDIT] POD headers, use plural module and tidy Koha::Statistic
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Bug 9978 should have fixed them all, but some were missing.
We want all the license statements part of Koha to be identical, and
using the GPLv3 statement.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Tomas Cohen Arazi
Pedro Amorim
Lari Taskula
Koha Development Team
Nick Clemens
Marcel de Rooy
Jonathan Druart