Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit c473480d4e)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 5b6b095f1f)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit a4f09abd5c)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 3a2ac10bcf)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 471facdf26)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 544b89cf90)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit f48c5bbb8b)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 97b58633d0)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
To test:
1 - Enable ES and index
2 - Set expandedSearchOption to 'show'
3 - Advanced search on staff and opac using Year field on staff and Publication date range on OPAC for:
2005
2005-
-2005
<2005
>2005
2005-2010
4 - Confirm ranges don't work on staff side, but do on opac
5 - Apply patch
6 - Repeat searches - they work on both staff and opac
7 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit a11633379f)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit a72ecc3b27)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit f8f9231562)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit cef190f71c)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit e926de5e8f)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 22c9ea9737)
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
When a query with "OR" is combined with a limit in Elasticsearch, the precedence is not preserved and the results are not correct.
To test:
1) Set SearchEngine to Elasticsearch
2) Index records in Elasticsearch
3) Do an advanced search
4) Select More options
5) Enter a value for the first Keyword (e.g. Novels)
6) Change "and" before the second Keyword to "or"
7) Enter another value for the second Keyword (e.g. Prose)
8) Limit the search (e.g. Item type Books)
9) Do the search
10) Observe that records with the first keyword are not in the results
11) Apply the patch
12) Repeat the search
13) Observe that results with both keywords are in the results
14) Sign off
Sponsored-by: Lund University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 3ed649a0e0)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit d6a3a10fc7)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
This patch adds quoting when handling branchcodes in searching in order
to prevent errors when branchcodes are reserved words in ES
To test:
0 - Be using Elasticsearch with Koha
1 - Add a new branch code:OR name:Orly
2 - Add an item to this branch
3 - Use advanced search to limit search to only Orly
4 - Oh really bad, the search fails!
5 - Apply patch
6 - Repeat search
7 - Oh really good, the search succeeds
8 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit a4c5bd5836)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 584c49961c)
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
We changed from 'null' being meainingful to using a tinyint and
explicitly requiring '0'. This patch simply updates the unit test to
reflect that change.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Starting in Elasticsearch 6.0 _key should be used instead of _term to order
buckets by their term
To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) If you observe an error about types, apply patch for bug 25669
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Observe if you get a deprecation warning about order key
5) Apply patch
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no deprecation warning about order key
8) Sign off
Sponsored-by: Lund University Library
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Fluid is not a valid value for cxn_pool - valid values are:
Sniff,Static,NoPing
AddAuthority call calls index - the indexing call was trying to access
the ES server with mocked values and dying when they were invalid
This patch uses a correct value for cxn_pool and inserts directly into DB
to avoid indexing
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
1. Create an authority with at least a 1XX$a and a 4XX$a, for instance:
100 $a Foo
400 $a Bar
2. Create a biblio and add a link to this authority using the
cataloguing plugin
3. Disable syspref IncludeSeeFromInSearches
4. Reindex the biblio. You should be able to find it when searching
'Foo' but not when searching 'Bar'
5. Enable syspref IncludeSeeFromInSearches
6. Reindex the biblio. You should be able to find it when searching
'Foo' and also when searching 'Bar'
without the patch, 'Bar' doesn't yeld results. With it, it does.
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Note: tests are very similar between ES and Zebra, however, ES requires the uppercase
OR and doesn't use '=' in the same way. I feel having test coverage in each module is fair and more future
proof in case of changes to search engine
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds tests and handling for calling move_to_biblio on a
Koha::Items set that contains items from more than one source biblio.
Test plan
1/ Inspect the changes to t/db_dependent/Koha/SearchEngine/Indexer.t
2/ Run t/db_dependent/Koha/SearchEngine/Indexer.t and confirm it passes
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch moves the Koha::Biblio->adopt_items_from_biblio method to the
Koha::Items set class and updates all calls from
Biblio2->adopt_items_from_biblio(Biblio1) to Biblio->items->move_to_biblio(Biblio2)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We need to update the search index record for the old biblio where the
item was moved from to keep the item info in search index up-to-date.
To test:
1) $ prove t/db_dependent/Koha/SearchEngine/Indexer.t
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In our test setup we mock the index_records() to produce warnings like
this:
Koha::Item at t/db_dependent/Koha/SearchEngine/Indexer.t line
93.
By wrapping all our item creations to warnings_are{} we can silence them.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
- Tests for adopt_items_from_biblio
- Tests for the relationship between items and acquisition orders
- Tests for indexer calls in adopt_items_from_biblio
Signed-off-by: Michal Denar <black23@gmail.com>
Rebased-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch allows merging of records with many items without the web server timing out.
Test plan:
Without the patch:
- Create 2 records (one with e.g. 1000 items).
- Do a cataloguing search that displays both records, select them and click "Merge selected".
- Choose the record with many items as the one to be eliminated.
- Start the merging.
- After a while the web server should give you a timeout error (the merging process may still continue)
With the patch:
- Do the same as above
- This time verify that the records are merged without timeout
- Create a new biblio with an item
- Add with the item:
* acquisition order
* hold (reserve)
- Merge the biblio to another one
- Verify that the item and its related data was moved
- Verify that tests pass:
prove -v t/db_dependent/Koha/Biblio.t
prove -v t/db_dependent/Koha/Item.t
prove -v t/db_dependent/Koha/SearchEngine/Indexer.t
Signed-off-by: Michal Denar <black23@gmail.com>
Rebased-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Run t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a unit test to check that skip_record_index is adhered to when
passed to LostItem with the force_mark_returned option enabled.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
With bug 27281 we removed a call to reset an items holdingbranch back to
the 'frombranch' of the transfer. This is correct, as with bug 23129
(and friends) we ensure that the holdingbranch was not changed at
transfer request time and instead would only be updated at the reciept
of transfer. As such, it's safe to remove the update here as it would be
just setting the same branch as is already present.
However, when removing this ModItem call, I neglected to realise that
also remoes the requirement for passing around the defered indexing bit
as we no longer need to trigger a re-index as there's not any change to
the item.
This patch therefore cleans up the POD for LostItem, removes the no
longer references 'params' parameter and drops the corresponding index
test.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The tests were failing because we weren't actually changing the item, so store was doing nothing.
We need to transfer 'from' a new branch so that the holdingbranch is changed
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The addition of cancellationdate meant that our test objects in the
Indexer tests now longer matched expectations and as such the tests
failed.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In zebra we sort by callnumber using 8007 cn-sort
We should do the same in elasticsearch
To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - View the mappings page, confirm table is correctly structured
2 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t t/db_dependent/Koha/SearchEngine/Elasticsearch/Reset.t t/db_dependent/Koha/SearchField.t
3 - Confirm update statement works and table is ordered correctly
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
For debugging purposes we may wish to see the requests and responses made to
Elasticsearch
To test:
1 - prove -v t/Koha/SearchEngine/Elasticsearch.t
2 - Set <trace_to>Stderr</trace_to> in koha-conf
3 - Restart all
4 - perl misc/search_tools/rebuild_elasticsearch.pl
5 - Note requests are shown
6 - Set
<trace_to>File</trace_to>
<trace_to>/var/log/koha/kohadev/plack-error.log</trace_to>
in koha-conf
7 - Restart all
8 - perl misc/search_tools/rebuild_elasticsearch.pl
9 - Check the plack log and see the ES requests
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Tests failing, seems to be because the indexes haven't been created yet.
When deleting authority we perform a 'merge' where we search for biblios
containing the authority.
Since these are simply tets we can skip this step, avoid the search, and avoid
a crash
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
These tests cover the changes to ensure the AddReturn calls index_records once
per call and that other calls pass the skip_record_index parameter correctly
Signed-off-by: Séverine Queune <severine.queune@bulac.fr>
Signed-off-by: Christopher Brannon <cbrannon@cdalibrary.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Bug 23828: (follow-up) fix unit test merge
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 23828: (QA follow-up) Fix number of tests
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
These cover Koha::SearchEngine::Indexer and ensure that all calls in the code
are routed correctly to the expected search engine
Bug 25265: (follow-up) Skip tests if elastic not configured
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
These tests fail now, the code expects a real response from ES in Indexer.pm
but these tests mock 'bulk' and so don't have the necessary fields.
We are testing the same code above and can just add the _id == biblionumber test
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a "year" search field type. Fields with this type will only
retain values that looks like years, so invalid values such as
whitespace or word characters will not be indexed.
This for instance improves the behaviour when sorting by
"date-of-publication". If all values are indexed, records with
junk data instead of valid years will appear first among the search
results, drowning out more relevant hits. If assigning this field
the "year" type these records will instead always appear last,
regarless of sort order.
To test:
1) Have at least two biblios, one with a valid year in 008 (pos 7-10)
and another with an invalid one ("uuuu" for example)
2) Perform a wildcard search (*) and sort results by publication date.
3) The record with invalid year of pulication in 008 should appear first
4) Apply patch and run database updates
5) Reindex ElasticSearch
6) Perform the same search as in 2)
7) The record with the invalid year should now appear last
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
.pm must not have -x
.t must have -x
.pl must have -x
Test plan:
Apply only the first patch, run the tests and confirm that the failures
make sense
Apply this patch and confirm that the test now returns green
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Nick Clemens
Julian Maurice
Tomas Cohen Arazi
Kevin Carnes
Martin Renvoize
Jonathan Druart
Joonas Kylmälä
Ere Maijala
Fridolin Somers
Marcel de Rooy
David Gustafsson
David Gustafsson