Issue description:
- call to ModZebra was unconditional inside 'store' method for Koha::Item,
so it was after each item added, or deleted.
- ModZebra called with param biblionumber, so it is the same parameter
across calls for each items with same biblionumber, especially when we
adding/removing in a batch.
- with ElasticSearch enabled this makes even more significant load
and it is also progressively grows when more items already in DB
Solution:
- to add extra parameter 'skip_modzebra_update' and propagate it down to
'store' method call to prevent call of ModZebra,
- but to call ModZebra once after the whole batch loop in the upper layer
Test plan / how to replicate:
- make sure that you have in the admin settings "SearchEngine" set to
"Elasticsearch" and your ES is configured and working
( /cgi-bin/koha/admin/preferences.pl?op=search&searchfield=SearchEngine )
- select one of biblioitems without items
( /cgi-bin/koha/cataloguing/additem.pl?biblionumber=XXX )
- press button "add multiple copies of this item",
- enter 200 items, start measuring time and submit the page/form...
On my test machine when adding 200 items 3 times in a row (so 600 in
total, but to show that time grows with every next batch gradually):
WHEN ElasticSearch DISABLED (only Zebra queue):
- 9s, 12s, 13s
WHEN ElasticSearch ENABLED:
- 1.3m, 3.2m, 4.8m
WITH PATCH WHEN ElasticSearch ENABLED:
- 10s, 13s, 15s
Same slowness (because also same call to ModZebra) happens when you try
to delete all items ("op=delallitems"). And same fix.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch corrects a few additional cases where DateTime->now is called
directly instead of via Koha::DateUtils.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
We should use Koha::DateUtils instead of Date::Time directly
This patch simplay replaces calls to now() with a call to dt_from_string()
which does effectively the same thing.
Probably reading the code and verifying changes is sufficient but...
To test:
1 - confirm the files all compile
2 - confirm all tests pass
3 - confirm Koha still works
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Use of uninitialized value $mail{"Cc"} in substitution (s///) at /usr/share/perl5/Mail/Sendmail.pm
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes the ViewPolicy filter use the 'params' accessor instead
of relying of ->{options} which has no accessor. This will allow
interacting with the filter object be similar through all the filters in
the chain.
To test, we just need to verify no behaviour change takes place:
1. Run:
$ kshell
k$ prove t/db_dependent/Filter_MARC_ViewPolicy.t
=> SUCCESS: Tests pass
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests still pass!
4. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
status varchar(6) with readable statuses
borrowernumber not null default 0
hide form if message successfully sent
fixing hide viewed and hide closed filters
adding recipient column
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
I think this was a simple case of Aliesha missing a file when commiting.
So I could proceed with testing I just quickly re-implimented the patron
relationship accessor.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
The conversion of the metadata into Base64 for passing in the API call
to the plugin API routes was failing if the metadata contained a wide
character, so we now encode it into UTF-8 first
Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds the required infrastructure to enable ILL availability
plugins to intercept the request creation process and, using the
supplied metadata, search for and display possible relevant items from
whichever availability plugins are installed.
Currently three availability plugins exist:
z39.50 - Searches any number of the Koha instance's configured Z targets
https://github.com/PTFS-Europe/koha-plugin-ill-avail-z3950
EDS - Searches the EBSCO Discovery Service
https://github.com/PTFS-Europe/koha-plugin-ill-avail-eds
Unpaywall - Searches the Unpaywall API for possible open access versions
of the requested item
https://github.com/PTFS-Europe/koha-plugin-ill-avail-unpaywall
The Unpaywall plugin is intended to serve as a "reference" plugin as the
API it deals with is extremely simple
Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes as_marc_field skip subfields that don't have a value or
contains an empty string (replicating C4::Items:1021 logic).
To test:
1. Apply the regression tests patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/Item.t
=> FAIL: Tests fail because the generated MARC::Field contains undef
subfields.
3. Apply this patch
4. Repeat 2.
=> SUCCESS: Tests pass! Undefined and empty subfields are skipped!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds the same updated logic as found in
Koha::Account::Line->apply to handle LOST_FOUND credits being applied
against their associated LOST debit.
Currently a 'LOST_FOUND' credit is never created via the pay method and
so the additional test will always pass, but the addition here accounts
for possible future uses.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Sooooo....
That was tricky, and the solution looks trivial.
However it's not.
We have unsafe set for "historical reason".
Having it on when RaiseError is on have the effect of overwritting the
DBIC error handler.
The problem is:
t/db_dependent/Circulation/MarkIssueReturned.t (and other tests) is failing with:
# expecting: Koha::Exceptions::Object::BadValue
# found: DBIx::Class::Exception ({UNKNOWN}: Can't locate object method "rethrow" via package "DBD::mysql::st execute failed: Incorrect datetime value: 'bad_date' for column 'returndate' at row 1 [for Statement "UPDATE `issues` SET `returndate` = ? WHERE ( `issue_id` = ? )" with ParamValues: 0='bad_date', 1=238] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line
In Koha::Object->store, the exception is not a DBIx::Class::Exception object (as we except), but
a string (on which we cannot call rethrow).
Swithing unsafe off restores the expected behavior.
To make sure the UI will not be affected, it is only turned off when
RaiseError is set.
The situation is still wrong (for UI), from the POD https://metacpan.org/pod/DBIx::Class::Storage::DBI (/unsafe)
"""
Note that your custom settings can cause Storage to malfunction, especially if you set a HandleError handler that suppresses exceptions and/or disable RaiseError.
"""
And also https://metacpan.org/release/DBIx-Class/source/lib/DBIx/Class/Storage/DBI.pm#L1531
Many thanks Tomas for the digging exploration!
We need to turn RaiseError and remove the unsafe flag, for UI as well,
but that should be done at the beginning of a dev cycle.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This is the only situation I found where:
* t/db_dependent/Koha/Item.t is passing
* t/db_dependent/Koha/Object.t is passing
* MySQL 8 is happy (and not fail with "Column 'timestamp' cannot be null"), which is certainly what I missed in the previous follow-up
About the change to Object.t: the next store was called without the
generated timestamp, so it is needed to call discard_changes.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In some tests we want to know if we are in a testing environment.
When run the usual way, our trick works, the perl interpreter matches 'prove':
$ENV{_} eq 'prove'
In other situations, we have the KOHA_NO_TABLE_LOCKS environment variables, for the SendCirculationAlert race conditions (see bug 15854 and bug 18364).
For unknown reasons, Jenkins runs the tests with /usr/bin/perl.
This patch suggests to rename KOHA_NO_TABLE_LOCKS and use KOHA_TESTING
instead, when prove is not used (or not correctly detected as it it the
case for Jenkins)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Have some patrons in your system
2 - Export some of their info via reports
SELECT cardnumber, userid, surname, firstname, password, branchcode, categorycode
3 - Edit the file from above, changing all the password lines
4 - Import the file with overwrite
5 - Confirm passwords have not changed (run the report again and confirm the hashes are the same)
6 - Apply patch
7 - Restart all the things
8 - Check the new box on import screen to overwrite passwrods
9 - Import file again
10 - Confirm passwords have changed
11 - Signin using new password to verify the hash is the password as supplied
12 - Repeat via commandline import supplying --overwrite_passwords option
13 - Verify works as expected
14 - Prove -v t/db_dependent/Koha/Patrons/Import.t
Sponsored-by: ByWater Solutions
Signed-off-by: Ron Marion <ron.marion@goddard.edu>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patchset removes some columns from the aqbooksellers table that are
not used:
* booksellerfax
* booksellerurl
* bookselleremail
* othersupplier
* currency
The first 3 ones are certainly leftover from bug 10402.
Test plan:
For each for these fields you will make sure that they cannot be edited
from the interface.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan :
Launching script several times and see that marc field is alway ordered increasing
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch changes the spec loading so that it is fully loaded and merged with any plugin routes before validation. The individual parts are separately validated only if the resulting complete spec is invalid. Note: the OpenAPI plugin validates the spec given to it, so normally no manual validation is necessary.
Test plan:
1. prove -v t/db_dependent/Koha/REST/*
2. prove -v t/db_dependent/api/v1/*
Sponsored-by: The National Library of Finland
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To match PERL15 we change the behavior of Koha::Object methods to make
them use the DBIx::Class relationships.
Test plan:
- Create a vendor, basket and contact
- Edit the vendor => the contact must appears
- Go to the view of a vendor => you cannot delete the vendor, it has
a basket attached
- Create another vendor, without basket
- Go to the view of the vendor => You can delete it
Additionally:
% prove t/db_dependent/Koha/Acquisition/Booksellers.t
must return green
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Owen Leonard 2018-03-16 10:47:47 UTC :
<<
I don't think the system preference adds any security. There are already multiple permissions required for working with plugins:
- Configure plugins
- Manage plugins ( install / uninstall )
- Use report plugins
- Use tool plugins
And even with those permissions your server must be configured to allow the use of plugins.
>>
Test plan :
1) Install kitchen sink plugin https://github.com/bywatersolutions/koha-plugin-kitchen-sink
2) Run misc/devel/install_plugins.pl
3) Set config enable_plugins=1
4) Check all parts of the plugin are working
5) Set config enable_plugins=0
6) Check all parts of the plugin are disabled
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In order to remove UseKohaPlugins syspref, adds to Koha Template Toolkit
plugin a method ArePluginsEnabled().
Test plan:
1) Set syspref UseKohaPlugins enabled
2) Set config enable_plugins=1
3) Check you see "Tool plugins" in "Tools" home page and menu
4) Check you see "Manage plugins" in "Koha administration" home page
5) Check you see "Report plugins" in "Reports" home page and menu
6) Set config enable_plugins=0
7) Check you don't see in 3) 4) 5)
8) Run prove t/Koha_Template_Plugin_Koha.t
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds the a js-date-format.inc file on each opac and staff interface (date-format.inc whas not available as name)
When you include that file in your TT, you'll get the following functions available:
1. $date(raw_date_string, options)
This function parses a date string, as produced by an api call, and returns the corresponding date formatted according to 'dateformat' and 'TimeFormat' parameters.
For example:
// dateformat: us
// timeformat: 12hr
// Timezone: UTC
$date('2020-03-23T15:00:00+01:00') // You will get '03/23/2020'
$date('2020-03-23T15:00:00+01:00', {withtime: true}) // You will get '03/23/2020 14:00'
$date('2020-03-23T15:00:00+01:00', {withtime: true, tz: 'Europe/Paris'}) // You will get '03/23/2020 15:00+01:00'
Options:
* dateformat: override date format as configured in staff interface (accepts also 'rfc3339')
* timeformat: override time format as configured in staff client (available options are '12hr' and '24hr')
* withtime: also print the time part (default false)
* tz: set the timezone
2. $datetime(raw_date_string, option)
The same as $date but sets withtime to true
3. $time(raw_date_string, option)
The same as $datetime but shows only the time part
To test you must implement and test bug 20936, where it will be used
Signed-off-by: Andrew Isherwood <andrew.isherwood@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Subroutine should "return;" instead of "return 0;"
We also prevent ProhibitExplicitReturnUndef by not returning "undef"
explicitly.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch changes the way news is returned from GetNewsToDisplay in
KohaNews.pm so that the template variable will evaluate as false instead
of simply being an empty data structure.
To verify the bug, remove your opacheader news entry and view source on
any page in the OPAC. You should find some empty markup right after
'</div> <!-- /navbar -->'
<div class="container-fluid">
<div class="row-fluid">
</div>
</div>
To test the fix, apply the patch and reload the OPAC page. The empty
divs should be gone. Add content to the opacheader news item and confirm
that it displays correctly.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>