Commit Graph

2752 Commits

Author SHA1 Message Date
Lucas Gass 6add1365f6 Bug 26853: DBIC changes
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-16 19:13:14 +00:00
Jonathan Druart 493e62aad8 Bug 26639: Add auto_savepoint to DBIC connection
See comment 0.

https://metacpan.org/pod/distribution/DBIx-Class/lib/DBIx/Class/Manual/Cookbook.pod#Nested-transactions-and-auto-savepoints

Otherwise:
DBIx::Class::Storage::txn_rollback(): A txn_rollback in nested transaction is ineffective! (depth 1) at t/db_dependent/Koha/Objects.t line 274

Possible side-effects? Slowness?

We need to push it to master ASAP and see how our test suite behave.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit bd63fb2bb9)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-16 18:51:05 +00:00
Nick Clemens 17571f824d Bug 17515: Order Z3950 server by rank and preserve ordering
Previously we put all the servers into an object with keys of the server id

This patch converts it to an array of objects to preserve order, and adjusts code to use the array index
where necessary and store the server id within the array

To test:
1 - Add some new Z3950 servers, they don't need to be valid
    FIRST
    SECOND
    THIRD
    FOURTH
2 - Adjust the ranking so FOURTH:1 THIRD:2 SECOND:3 FIRST:4
3 - Enable and launch the advanced editor
4 - Click 'Advanced' under search on the left
5 - Note the list displays in the order you entered the servers
6 - Apply patch
7 - Reload
8 - Order is correct
9 - With valid servers, confirm that searching still works and servers can be checked or unchecked to include/remove from results

https://bugs.koha-community.org/show_bug.cgi?id=17515

Signed-off-by: B Johnson <barbara.johnson@bedfordtx.gov>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 7bbf4f7b5f)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-16 18:31:26 +00:00
Nick Clemens 5b5c58e08b Bug 18051: Set UTF8 Flag for record searches/imports from rancor
The advanced cataloging editor uses Koha::MetaSearcher to find and import records
from Z3950.

The records are correctly converted to UTF8, however, we need to set the flag in the
record to ensure they are parsed correctly later

To test:
 1 - Add OHIOLINK as a z39 source as described in earlier comments
 2 - Browse to Cataloging->Advanced editor
 3 - Click Search->Advanced
 4 - Select OHIOLINK
 5 - Search for author 'capek karel'
 6 - Note records look bad
 7 - Apply patch
 8 - Restart all the things
 9 - Repeat
10 - Success!

Signed-off-by: Barbara Johnson <barbara.johnson@bedfordtx.gov>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 4d92e3d284)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-16 18:07:38 +00:00
Katrin Fischer 9999d88169 Bug 18050: Add relation alias to schema
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 2800b53bdd)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 21:15:52 +00:00
Lucas Gass 610fab83e0 Bug 18050: DBIC
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 21:15:16 +00:00
Jonathan Druart 7485f479fa Bug 26673: Fix perlcritic complains
This patch fixes the following perlcritic complain:
Package declaration must match filename at line 19, column 1.  Correct the filename or package statement.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 59588c3a3e)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 21:05:25 +00:00
Jonathan Druart 3890be5ba9 Bug 26673: Remove shebangs from Perl modules
It's not needed.

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 9f47b4ffd7)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 21:04:25 +00:00
Nick Clemens 06150351b4 Bug 26781: Check for subfield defined rather than truth
To recreate:
 1 - Have a record with 100$0 defined
 2 - Define a Marc Modification template:
    Delete field 100$0
 3 - Perform Batch record modification, enter the record above in the list of records
 4 - Preview the MARC
 5 - Note the entire 100 field has been deleted, don't modify the record
 6 - Apply patch
 7 - Restart all the things
 8 - Test the modification again
 9 - This time the preview confirms only the subfield 0 deleted
10 - Modify the record
11 - Confirm the record is correctly updated

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

JD amended patch: Fix typo ony vs only

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit be4f0c7639)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 20:50:22 +00:00
Martin Renvoize 67197c359e Bug 23432: Replace 'die' with 'warn'
This patch simply replaced the die statement with a warn such that the
cron script may continue to rotate other items even if this item cannot
be properly rotated due to corrupt data.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 6adb1891d2)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 18:29:19 +00:00
Björn Nylén aeb33f91a5 Bug 26750: Deleted items not indexed
After deleting an item it is still searchable in the index. Similar to bug 26507 where record is reindexed before the database is changed.

To test:
1. Delete an item
2. Search the barcode in the main search box. You'll find the original record but wiht hte deleted item missing.
3. Apply patch.
4. Repeat steps 1-2 (with another item). No record should be found.

Note to test you have to have the indexer running, not the cron job, and it
needs to be running fast so that the index is updated before the delete
takes place

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 395bb76039)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 18:20:50 +00:00
Martin Renvoize acd96409ea Bug 25549: Add error reporting to plugins-home
This patch adds handling for the newly introduced 'errors' option in
GetPlugins and reports erroroneos plugins to the user.

To test:
1 - Enable plugins in the koha-conf
2 - Install the kitchen sink plugin
3 - Your staff client should be orange now :-)
4 - The plugin should display as installed in the table at
    tools/plugins.
5 - edit the plugin module
    /var/lib/koha/kohadev/plugins/Koha/Plugin/Com/ByWaterSolutions/KitchenSink.pm
6 - Add a line to break compilation, like:
    this won't compile
7 - Restart all
8 - Your Koha is not broken, but is no longer orange (The plugin is not
    loading).
9 - The plugin should display as unable to load in the table at
    tools/plugins.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 759ff45199)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 16:21:43 +00:00
Nick Clemens bd2347fbf8 Bug 25549: Remove plugin methods for broken plugins
To test:
 1 - Enable plugins in the koha-conf
 2 - Install the kitchen sink plugin
 3 - Your staff client should be orange now :-)
 4 - edit the plugin module
     /var/lib/koha/kohadev/plugins/Koha/Plugin/Com/ByWaterSolutions/KitchenSink.pm
 5 - Add a line to break compilation, like:
     this won't compile
 6 - Restart all
 7 - Your koha is now broken
 8 - kshell
 9 - perl misc/devel/install_plugins.pl
10 - Restart all
11 - Koha remains broken
12 - Apply patch
13 - kshell
14 - perl misc/devel/install_plugins.pl
15 - Koha now works!
16 - Koha is not orange because the plugin methods are removed

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 0eae966d25)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 16:20:49 +00:00
Nick Clemens c3f61d3bd9 Bug 23828: Elasticsearch - Preserve record order when combining subfield in mappings
Currently if you combine subfields in the marc mappings the subfields are indexed in the order
listed in the mapping.

i.e. 650(avxyz) in mapping

and in record:
650 $aHeading $zGeosubdiv $vFormsubdiv

is indexed as:
Heading Formsubdiv Geosubdiv

We should preserve the order and index as:
Heading Geosubdiv Formsubdiv

We can use built in function in Marc::Field to achieve this

To test:
 1 - It is easy to find examples of this using authorities
 2 - Find or create a record with subfields order azv
     e.g. 150$aActresses$zUnited states$vBiography
 3 - Add or have a second authority
     e.g. 150$aActresses$vPortraits
 4 - Set an authorities mapping for 'Heading' to 150(abgvxyz)
     find at:
     Administration->Search engine configuration (Elasticsearch)->Authorities tab
 5 - Index the records in Elasticsearch
     perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692
 6 - View the first record in the ES index
     curl es:9200/koha_kohadev_authorities/data/1692?pretty
 7 - Note 'Heading' field is ordered as in the mapping
 8 - Search authorities for 'contains' "act"
 9 - Note the records sort incorrectly
10 - Apply patches
11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692
12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty
13 - Note the order is now preserved
14 - Search authorities for 'contains' "act"
15 - Note the records sort correctly

Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

JD amended patch: Fix
   FAIL   spelling
     combind ==> combined

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 890fd6ff06)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-13 16:12:15 +00:00
Matthias Meusburger 8a5898aab8 Bug 26506: Fix Koha::Account::pay if called when no user is logged-in
Currently, Koha::Account::pay will fail if no user is logged-in when it is
called.

In this case, this patch will use undef instead of 0 for $manager_id in order
to comply with the foreign key constraint on manager_id.

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Biblibre and Libriotech have created a OPAC payment plugin that was
having problems because of this bug. Applying the fix in this patch
to a live server solved the problem.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 63c93d3c11)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-09 13:15:11 -07:00
Marcel de Rooy cfc14edbcc Bug 26600: Missing module in Elasticsearch/Indexer.pm
Just formally needed. It is already loaded somewhere.
That is: Koha::SearchEngine::Elasticsearch.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 6deb177576)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-11-09 18:49:34 +00:00
Jonathan Druart d9599a6d33 Bug 25913: Prevent get_coins to crash if record does not have title
If a bibliographic record does not have a title, get_coins will crash
with
Can't call method "as_string" on an undefined value at /kohadevbox/koha/Koha/Biblio.pm line 645
Koha::Biblio::get_coins('Koha::Biblio=HASH(0x5558f91bb740)') called at /kohadevbox/koha/catalogue/ISBDdetail.pl line 144

We can handle that situation easily by checking the existence of the
title field.

Test plan:
1. Create a record without 245
2. Enable COinSinOpac
4. Go to the ISBD detail view
=> It must not fail with this patch applied

Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 64c939ca80)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-20 19:59:52 +00:00
Nick Clemens 3d21e7b578 Bug 25957: Don't push blank field values to ES document
To test:
 1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
 2 - For example 'author' / 100a
     100 _ _ ‡a
 3 - Index that record into Elasticsearch 5.X:
     perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 4 - Note error 'value must have length > 0'
 5 - Edit mappings to set author 100a not suggestible
 6 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 7 - Success
 8 - Set field to suggestible again
 9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
11 - Success!

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit eb3d0982ae)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-20 19:53:02 +00:00
Lucas Gass c2dd8fe671 Bug 18958: DBIC shema changes
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-16 16:57:00 +00:00
Fridolin Somers aaef045006 Bug 26434: Fix plugin dirs addition to @INC
Plugin dirs defined in koha-conf.xml are added to @INC in order to compile perl code.
Looks like with plack those dirs are added several times.
This may lead to an error "INCLUDE_PATH exceeds 64 directories".

This bug was identified with Carrousel plugin : https://inlibro.com/instructions-carrousel/

Test plan :
1) Enable plack and plugins
2) Look at page about.pl : @INC contains one plugin dir 'var/lib/plugins'
3) Install plugin KitchenSink : https://github.com/bywatersolutions/koha-plugin-kitchen-sink
4) Dont apply patch
5) Use configure on KitchenSink
6) Look at page about.pl : @INC contains server plugin dir 'var/lib/plugins'
7) Apply patch and restart plack
8) Use configure on KitchenSink
9) Look at page about.pl : @INC contains one plugin dir 'var/lib/plugins'

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit dcdc72aee7)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-16 16:23:43 +00:00
Colin Campbell dc94e57b96 Bug 26438: Rename temporary Koha::Item variables to avoid conflict
Returns from Koha::Item calls were stored in temporary variables
called $item. Unfortunately they occur in routines already with
an $item variable holding a lineitem and hide that variable although
only required for two lines of code. An error occurs if girfield
is called on the wrong variable. Renamed all three occurences
of $item to $kitem to avoid the error

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 647e05d826)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-16 16:15:27 +00:00
David Gustafsson cbd56aa5fc Bug 24807: [20.05.x] Add "year" type to improve sorting behaviour
Add a "year" search field type. Fields with this type will only
retain values that looks like years, so invalid values such as
whitespace or word characters will not be indexed.
This for instance improves the behaviour when sorting by
"date-of-publication". If all values are indexed, records with
junk data instead of valid years will appear first among the search
results, drowning out more relevant hits. If assigning this field
the "year" type these records will instead always appear last,
regarless of sort order.

To test:

1) Have at least two biblios, one with a valid year in 008 (pos 7-10)
and another with an invalid one ("uuuu" for example)
2) Perform a wildcard search (*) and sort results by publication date.
3) The record with invalid year of pulication in 008 should appear first
4) Apply patch and run database updates
5) Reindex ElasticSearch
6) Perform the same search as in 2)
7) The record with the invalid year should now appear last

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: Add database update script

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: Update tests

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: Add suppport for uncertain fields and ranges

To test:
1 - Have some records with uncertain dates in the 008
    19uu, 195u, etc.
2 - Index them in Elasticsearch
3 - Do a search that will return them
4 - Sort results by publication/copyright date
5 - Note odd results
6 - Apply patch
7 - Reindex
8 - Sorting should be improved

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: Refactor using tokenize_callbacks

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: Simplify with new and imporved value_callbacks

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: (follow-up) Fix spelling

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: (follow-up) Add support for spaces as unknown characters

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 24807: (QA follow-up) Remove uneccessary tests

These tests fail now, the code expects a real response from ES in Indexer.pm
but these tests mock 'bulk' and so don't have the necessary fields.

We are testing the same code above and can just add the _id == biblionumber test

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-16 14:27:07 +00:00
Nick Clemens 7054818646 Bug 25265: [20.05.x] Prevent double reindex of the same item in batchmod
When batch editing, 2 reindex calls are sent to ES/Zebra.
We can easily avoid that reusing the skip_modzebra_update (renamed skip_record_index)

Additionally we should only send one request for biblio, and we should
only do it if we succeed

As the whole batch mod is in a transaction it is possible to fail in which case
Zebra queue is reset, but ES indexes have already been set

In addition to the skip param this patchset moves Zebra and Elasticsearch calls to
Indexer modules and introduces a generic Koha::SearchEngine::Indexer so that we don't
need to check the engine when calling for index

The new index_records routine takes an array so that we can reduce the calls to
the ES server.

The index_records routine for Zebra loops over ModZebra to avoid affecting current behaviour

Test plan:

General tests, under both search engines:
  1 - Add a biblio and confirm it is searchable
  2 - Edit the biblio and confirm changes are searchable
  3 - Add an item, confirm it is searchable
  4 - Delete an item, confirm it is not searchable
  5 - Delete a biblio, confirm it is not searchable
  6 - Add an authority and confirm it is searchable
  7 - Delete an authority and confirm it is not searchable

Batch mod tests, under both search engines
  1 - Have a bib with several items, none marked 'not for loan'
  2 - Do a staff search that returns this biblio
  3 - Items show as available
  4 - Click on title to go to details page
  5 - Edit->Item in a batch
  6 - Set the not for loan status for all items
  7 - Repeat your search
  8 - Items show as not for loan
  9 - Test batch deleting items
    a - Test with a list of items, not deleting bibs
    b - Test with a list of items, deleting bibs if no items remain where all items are only item on a biblio:
     SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1)
    c - Test with a list of items, deleting bibs if no items remain where some items are the only item on a biblio:
     SELECT MAX(barcode) FROM items GROUP BY biblionumber HAVING COUNT(barcode) IN (1,2)
 10 - Confirm records are update/deleted as appropriate

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 25265: Rename skip_modzebra_update to skip_record_index

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 25265: Fix copy paste error for parameter

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 25265: (follow-up) Don't index malformed records

This is analogous to 26522, we shoudl skip record that cannot be retrieved for indexing

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 25265: (QA follow-up) Add shebang to Indexer.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 25265: (QA follow-up) Rename biblionumber in ModZebra, index_records

ModZebra:
The name is very misleading: we can index authid's too here.
And yes, it should not be in C4/Biblio too ;) A first step..

Adding the same change here in Koha/SearchEngine/Zebra/Indexer.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 25265: (QA follow-up) Check server type in Elasticsearch::index_records

Doing the same change as previously (renaming biblionumber), but fixing
at the same the record fetch. If (theoretically) an authority is passed
without a record, it would have fetched a biblio record.

Test plan:
You need Elasticsearch here.
Replaced this line in AddAuthority:
    $indexer->index_records( $authid, "specialUpdate", "authorityserver", $record );
by
    $indexer->index_records( $authid, "specialUpdate", "authorityserver", undef );
And updated an authority record. Check if you can search for the change.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-10-16 14:15:22 +00:00
Nick Clemens e8e87ca9b6 Bug 26507: [20.05.x] Elasticsearch - Index records after storing new item
Currently Item->store indexes the record before the DB update - that is wrong

To test:
 1 - Find/create a bib with no items
 2 - add an item with barcode "abc123"
 3 - do a general keyword search for "abc123," see your bib is not in the results
 4 - perform a search that includes your bib in the results, confirm it shows as having no items
 5 - click through to bib details, confirm it shows your item here
 6 - edit and save your item
 7 - confirm barcode is now searchable
 8 - apply patches
 9 - Add a new item "cde456"
10 - Confirm it returns in searches
11 - Edit 'cde456' and change barcode to 'fgh789'
12 - Confirm the new abrcode is searchable

Signed-off-by: Lisette Scheer <lisettes@latahlibrary.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-28 17:44:13 +00:00
Nick Clemens 401133b5b1 Bug 25273: (follow-up) Don't die on unknown authtype
We are guessing authtype code and inserting the heading built accorindg to C4::Heading

If we can't identify the auth type, we can format the heading. There is a record in the koha test
data that is missing the heading field so type cannot be idenfitied

This prevents us from dying n a record where we cannot identify the type.

Note: This code will also be triggered for custom authority types, higlighting
that they won't link because of hardcoded mappings in C4::Heading. We must tackle this
on a new bug

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 5a87514b2e)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-28 15:25:04 +00:00
Nick Clemens 5597b666e4 Bug 25273: Make match-heading rely on authority type configuration
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.

In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality

To test:
 1 - Start form a sample database with ElasticSearch working
 2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
    Waterworks
    Waterworks - Costs
 3 - Place a heading for the broader term in a record. e.g. Waterworks
       In 650$a, without the cataloguing authority plugin. We don't want
       the link created now.
       You need syspref BiblioAddsAuthorities => allow
 4 - Make sure linker is set to default
 5 - Attempt to link the records
       misc/link_bibs_to_authorities.pl
 6 - Linking fails
 7 - Apply patch
 8 - refresh index settings (if using a custom file, remove 'match-heading')
       You can reset mappings in the UI or run this:
       misc/search_tools/rebuild_elasticsearch.pl -v -d -r
 9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
     prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Bug 25273: (follow-up)

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit ce161fda9b)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-28 15:24:35 +00:00
Tomas Cohen Arazi 5a5d1b8089 Bug 26322: Permissions not checked correctly for plugins
This patch fixes the logic in a condition to address the fact that
permissions are not checked for plugins. This was due to bad parenthesis
pairing and the lack of good tests for this.

To test:
1. Apply the regression tests patch
2. Run:
   $ kshell
  k$ prove t/db_dependent/Koha/REST/Plugin/PluginRoutes.t
=> FAIL: Tests fail because of bad logic
3. Apply this patch
4. Repeat (2)
=> SUCCESS: Tests pass!
5. Verify the tests cover the use cases that are needed:
   - Anonymous access
   - Real user with wrong permissions (parameters => 1)
   - Real user with right permissions (borrowers => 1)
=> SUCCESS: Needed use cases so we catch any regression are found
6. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 66402643d7cb95bf5ebc4832bff61becfa3d5a6a)
2020-09-22 23:12:43 +00:00
Nick Clemens 5fa47568d5 Bug 26309: Make cxn_pool configurable
In get_elasticsearch_params we set the conf to static if undefined,
but we never defined it

To test:
1 - Apply unit test patch
2 - prove -v t/Koha/SearchEngine/Elasticsearch.t
3 - It fails
4 - Apply this patch
5 - It succeeds

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit b87dc492fe)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-14 21:08:24 +00:00
Tomas Cohen Arazi 3fac75b079 Bug 25662: Make the route for holds restpect maxreserves
This patch fixes the behaviour for the POST /holds route. It assumed
maxreserves was checked in CanItemBeReserved which is not the case.

Tests are added to check for this behaviour.

To test:
1. Apply the regression tests
2. Run:
   $ kshell
  k$ prove t/db_dependent/api/v1/holds.t
=> FAIL: Tests fail!
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass!
5. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 69bdc94cd4)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-04 20:10:31 +00:00
Colin Campbell 8a4a166713 Bug 26082: Call store on new items to update itemnumber
Need to add call to store to Koha::Item->new so that itemnumber
is returned - otherwise is not defined and causes an error
Bug is a side-effect of Bug 23463 as this was implicitly
done in AddItem

Also set biblioitemnumber explicitly for item adds this also
used to be done explicitly but can now cause an FK error

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit b5a36c20a2)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-04 16:34:36 +00:00
Nick Clemens 48507259b4 Bug 25872: Fix scoping of $limits
The code in build_query_compat contained a 'my' in the assigning of limits for the search in a conditional

This meant the limits were being set correctly during the conditional, but we blanked when passed to the rest
of the code. The effect was that the searches worked, however, the template params to repeat the search were
incomplete.

Removing the my ensures the same limits are applied during search and on re-sorting

To test:
1 - Enable Elasticsearch
2 - On OPAC perform advanced search, selecting only an itype, ccode, or LOC limit
3 - Attempt to sort results
4 - You are returned to Advanced search
5 - Apply patch
6 - Repeat
7 - It sorts!

Signed-off-by: Lisette Scheer <lisetteslatah@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 43dc505a52)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-03 15:33:26 +00:00
Jonathan Druart 12a02a2349 Bug 26127: local_referer should not compare with OPACBaseURL case insensitive
In our test data we have OPACBaseURL that is filled with http://kohadev.myDNSname.org:8080,
but the CGI referer in local_referer is lowercase
We should compare it without case sensitivity

Test plan:
- Use koha-testing-docker
- Notice the value of OPACBaseURL
=> http://kohadev.myDNSname.org:8080
- Use the "reports a problem" feature at the OPAC
- Notice that the "Problem found on page" is correctly filled with the
referer

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 7a53c5f04c)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-09-03 15:23:13 +00:00
Tomas Cohen Arazi 86d74d4769 Bug 23634: (QA follow-up) Our PUT is really a PATCH
This patch makes the controller not expect that there will always be all
the email fields. So it now checks if an email field was passed, and
changed, and renders the error if that stands.

To test:
1. Run:
   $ kshell
  k$ prove t/db_dependent/api/v1/patrons.t
=> FAIL: Tests written by Nick highlight a problem
2. Apply this patch
3. Repeat 1
=> SUCCESS: Problems solved
4. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit f58ae3f0ead21e77da038f37fa098593e50615d5)
2020-08-31 20:01:53 +00:00
Martin Renvoize 98a287c482 Bug 23634: (QA follow-up) Catch all email cases in API
The API was only catching the primary email change case, but we need to
catch email, emailpro and B_email.

We were also not accounting for any of the emails (on PUT or from the
DB) being undefined.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 31e362484bba5a42f218027cee0b15a714eccb8c)
2020-08-31 20:01:22 +00:00
Tomas Cohen Arazi 7071bb8149 Bug 23634: Secure the email on the API
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 624aa55ff33e6fe228ca806f8eb9c6592ab8cdbc)
2020-08-31 20:00:52 +00:00
Martin Renvoize c0f44c93fd Bug 23634: Add is_superlibrarian method to Koha::Patron
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 4825fd5d82241631745016e1dfbfe75e77138e7a)
2020-08-31 20:00:06 +00:00
Nick Clemens 455acf686d Bug 25882: Remove phr indicator from limits when processing
We assume all limits from advanced search to be a phrase and quote them
when doing this we should remove the phrase marker to avoid doulbe quoting

To test:
 1 - Have koha using ES
 2 - Go to advanced search
 3 - Limit by a single itemtype that exists
 4 - Get some results
 5 - Limit by a different itemtype that exists
 6 - Get some results
 7 - Limit by both itemtypes
 8 - Get only the results for the second itemtype
 9 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
10 - Repeat search
11 - View page source and find 'search_query'
12 - See limit looks like itype:("("BK")" OR "("CR")")
13 - Apply patches
14 - Restart all the things
15 - Repeat search for both itemtypes
16 - Note results now include both types

Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit a2b2426de8)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-08-17 14:25:26 +00:00
Nick Clemens 18215fbc57 Bug 26009: Add facet size to home/holdingbranch
To test:
0 - Set DisplayLibraryFacets to 'both' and FacetMaxCount to 20
1 - Have more than 10 branches
2 - Have items in each of those branches
3 - Enable ES, set system preference SearchEngine to Elasticsearch
4 - Search for '*'
5 - Expand homebranch/holdingbranch facets
6 - Note you only get 10
7 - Apply patch
8 - Repeat search
9 - Now you get all your facets

Signed-off-by: Michael Springer <mspringer@mylakelibrary.org>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 4dd1672c1b)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-08-14 22:16:58 +00:00
Nick Clemens 6bea771b80 Bug 25683: (follow-up) Use COALESCE instead of IFNULL
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit e1509855d3)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-08-14 21:13:18 +00:00
Nick Clemens 41f73194c4 Bug 25683: Fix grouping of results
DBIX doesn't seem to work as I expected - if trying to group by we can only
get the columns we grouped by, otherwise the queries are much more complicated
and come out wrong

This patch removes the 'amountoutstanding' from the query. Note that we do return patron
object, however, to access all the columns we must discard_changes (or refetch)

Also fixes a typo 'verbosse'

To test:
1 - charge a $1 fine to a patron
2 - pay off the fine
3 - perl misc/cronjobs/update_patrons_category.pl -f J -t J -fu=5 -v
4 - Note the patron is returned twice
5 - export DBIC_TRACE=1
6 - repeat 3 - view the SQL query and see how odd it is
7 - Apply this patch
8 - repeat 3 - simpler query
9 - patron returned only once

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit c7ae33f60c)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-08-14 21:13:01 +00:00
Nick Clemens 6ae42c8c3c Bug 25683: Patron with no accountlines should have 0 outstanding (not NULL)
Test plan:
- Have a patron with nothing in accountlines
- run update_patron_categories to find patrons with fines under $5 (-fu=5)
  - Your patron is not found
  - Give your patron a manual charge of $1
  - rerun the cron, your patron is found
  - pay off your patron's fine, putting their balance at $0
  - rerun the cron, your patron is found

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 09146f4a99)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-08-14 21:12:43 +00:00
Tomas Cohen Arazi 72bcf69f42 Bug 26143: Make the API handle per_page=-1
This patch introduces handling for per_page=-1 and actually adds a
missing feature to the API: being able to request all resources on a
route.

To test this:
1. Visit the libraries admin page
2. On the table, choose to display 'All' rows
=> FAIL: it doesn't refresh, the browser console displays a 500 error
code and so the logs
3. Apply the tests patches
4. Run:
   $ kshell
  k$ prove t/Koha/REST/Plugin/Pagination.t \
           t/db_dependent/Koha/REST/Plugin/Objects.t
=> FAIL: Tests fail loudly
5. Apply this patch
6. Restart plack
7. Repeat 2
=> SUCCESS: choosing to display all, works
8. Repeat 4
=> SUCCESS: Tests pass now!
9. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit e4cae99f7e)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-08-13 18:12:12 +00:00
Lucas Gass fcdb0a4426 Bug 23797: DBIC Schema Changes
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-31 20:03:37 +00:00
Martin Renvoize a450d3bb84 Bug 25765: (QA follow-up) Save a DB hit
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 83d63d2302)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-30 17:30:41 +00:00
Fridolin Somers 323ebe649d Bug 25765: Add GetLoggedInBranchname method
Add GetLoggedInBranchname method to Branches templates plugin.

Also changes GetName method to use Koha::Library instead of direct SQL.

Test plan :
Run prove t/db_dependent/Template/Plugin/Branches.t

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 0fcd0fd270)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-30 17:28:22 +00:00
Tomas Cohen Arazi 2270d92708 Bug 5087: (QA follow-up) Rename the attribute to fit later API usage
This patch renames the introduced attribute for later usage on the API,
and changes the logic to use it the other way around.

It also adds a KEY for the flag, as it will be used in WHERE statements.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 4780930720)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-30 17:01:53 +00:00
Aleisha Amohia 712143d26b Bug 5087: Ability to define whether CSV profile shows in OPAC
To test:
1) Update database
2) Update schema
3) In staff side, go to Tools -> CSV profiles
4) Make a CSV profile with profile type = MARC and usage = Export
records
5) Leave 'Show option in OPAC' unchecked and save
6) Go to OPAC and add an item to your cart
7) Click Cart, click Download, confirm that your newly created CSV
profile does not show as an option
8) Go back and edit CSV profile on staff side
9) Check the 'Show option in OPAC' checkbox and save
10) Go back to download cart in OPAC
11) Confirm the CSV profile now shows in the dropdown

Sponsored-by: Catalyst IT
Signed-off-by: Maryse Simard <maryse.simard@inlibro.com>

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit ae3644a594)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-30 17:01:08 +00:00
Martin Renvoize 3e3eaab724 Bug 25998: (QA follow-up) Add DBIC level relation
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 997094cc38)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-28 14:55:05 +00:00
Martin Renvoize f870e59a74 Bug 25998: Add 'library' relation to Koha::Account::Line
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 16413beea6)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-28 14:54:47 +00:00
Nick Clemens 9b51c5eb98 Bug 25723: (QA follow-up) Handle holiday and exception on same day
When a holiday is entered, then exceptions generated on a range, there exists both a holiday and exception in
the special holidays table. We should cache the exception over the holiday instead of both

Also, !1 in perl returns '' rather than 0, so we should explicitly set the value

Add blank line to clear pod error from qa tools

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Re-introduce the blank line mentioned in the commit message, it was accidentally removed by automatic formatting
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 223b603f08)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2020-07-27 17:40:25 +00:00