"return" statement with explicit "undef" at line 245, column 5. See
page 199 of PBP. (Severity: 5)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This option in combination with append seems to add null entries in the indexes
for fields that don't exist. These fields then affect sorting. Removing it solves
the issue
To test:
1 - Have an ES setup
2 - Catalog and index some records
3 - View a record in ES:
curl 'localhost:9200/koha_kohadev_biblios/data/4?pretty=yes'
4 - Note there are 'null' entries in the indexes, save this result for
comparison
5 - Apply patch
6 - Reindex, amking sure to delete existing indexes
perl misc/search_tools/rebuild_elastic_search.pl -d
7 - View the record again
8 - Note the null entries are removed
9 - Note indices are otherwise intact
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds a new statuc function to Koha::SearchEngine::ElasticSearch
which is instended to replace most of get_elasticsearch_params. This function
reads the configuration from C4::Context->config('elasticsearch') and raises
relevant exceptions when mandatory entries are missing.
Its behaviour is covered by tests.
To test:
- Run:
$ kshell
k$ prove t/Koha/SearchEngine/Elasticsearch.t
=> SUCCESS: Tests pass!
- Sign off :-D
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This code seems a bit odd, $self->sort_fields()->{$name} is checking if
there is any mapping in ES for a field, so if one marc field in an index
is marked as sortable this will exists. We need to check the individual
marc field to see if it should be added to the sort index here.
If you apply the first patch, reindex, and view a record in es:
curl -XGET 'localhost:9200/koha_kohadev_biblios/data/19/?pretty' | grep
-A 10 author__sort
You will see that 245$c is included in the record without that
additional check
Apply this, reindex again, and that field should not be included
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Fix incorrect condition for if clause for generating <field>__sort mappings
for Elasticsearch. Also remove redundant check for same condition when
generating fixer rules.
Test plan:
1. Inspect current mappings for example by viewing: http://<elasticsearch_host>:9200/koha_<koha_instance_name>_biblios/_mapping.
2. If using the default configuraion only "author" has a sort field (author__sort).
4. Appy patch.
5. Reindex using rebuild_elastic_search.pl.
6. All fields except those with sort sort set to "0" should now have sort fields, which in the default configuration is all but "author".
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html
Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Index some stuff with multiple fields defined for sorting
i.e. Authorites - make heading sortable - default is 110a and 111a for
heading - a record with 111a empty will make the sort field empty
2 - view the record:
curl http://localhost:9200/koha_kohadev_authorities/data/30?pretty=true
3 - Note the blank field
4 - Apply patch
5 - Reindex
6 - Fields should be correctly populated
Unit tests to follow (once I have the originals working for all)
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Make sure you have latest koha deps, catmandu versions should be:
libcatmandu-marc-perl 1.09-1~kohadev1
libcatmandu-perl 1.0304-2~kohadev1
2 - Reindex elastic
3 - Try searching and likely notice odd results
4 - Try:
curl -XGET
'http://localhost:9200/koha_kohadev_biblios/data/792?pretty=true'
with a known biblionumber and notice some null fields
5 - Apply patch
6 - Reindex
7 - Note fields are populated and search works as expected
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Have ES enabled and biblios and authorities indexed
2 - Perform a catalog search
3 - Try to sort results by author, it will fail
4 - Perform an authority search with sorting enabled, it will fail
5 - Apply patch
6 - Reindex, deleting current indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d -v
7 - Repeat 3&4, they should succeed
Signed-off-by: Srdjan <srdjan@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
The loading of file admin/searchengine/elasticsearch/mappings.yaml
specifies 'type' as empty, which fails with Mysql 5.7+ which is more
stringent.
Also, forcing an empty value into a boolean also fails.
Both issues arise when updatedatabase.pl hit 3.23.00.050.
NOTE: both issues could also be resolved by actually setting values in
the load file. This doesn't make this solution incorrect, though.
To Test/reproduce:
-1) Happens with Mysql 5.7.4. Maybe earlier, but certainly at
that point. Use a Xenial kohadevbox to more easily test.
0) Find a database on 3.22 or earlier, save it.
a) place
sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
into the [mysqld] section of /etc/mysql/mysql.conf.d/mysqld.cnf
b) restart you mysql server
c) drop the db and recreate it
d) checkout the 3.22.x branch
e) do a web install
f) remove the added sql_mode
g) restart the mysql server
1) Set your code base to master
2) run updatedatabase.pl
3) See the errors on 3.23.00.050
4) Apply the patch
5) Reload the 3.22 db.
a) repeat steps 0(a)-0(g)
b) don't forget the caching issues
6) succeed with updatedatabase.pl
7) drop the db and recreate it
8) run the web installer
9) notice no issues either.
10) run koha qa test tools
NOTE: This bug only solved the upgrade portion.
I added the kohastructure.sql change as well.
I confirmed that all the code changes were
triggered with this test plan.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mehdi Hamidi <mehdi.hamidi@inlibro.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Builds on top of commit:
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip
-Fix data type 'string' to 'keyword' and 'text'
-index: not_analyzed deprecated, replaced with type: keyword which is equivalent
-store: yes was deprecated, use store: true
TODO: Installer bindings to both the debian package install and the raw developer install.
A taster in Buug 17851
ZE TEST PLAN
0. Remove existing ES and reinstall ES 5.1
apt-get purge elasticsearch
Follow instruction here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings.
1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();'
1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping];
and retry 1a.
1c. Destroy elasticsearch index
curl -XDELETE localhost:9200/koha_biblios
so it can be recreated
2. Recreate the index:
perl misc/search_tools/rebuild_elastic_search.pl
2a. Add something to index if your koha.biblio-table is empty
3. Fetch all indexed records and the facet for subject__facet
curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
"aggregations": {
"my_agg": {
"terms": {
"field": "subject__facet"
}
}
}
}'
4. Run the included tests:
perl t/db_dependent/Koha_Elasticsearch_Indexer.t
perl t/db_dependent/Koha_Elasticsearch.t
perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
Have fun with your new ES 5.1 cluster!
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
-Changed deprecated facets to aggregations
-Fixed boolean datatypes not allowing analyzers to be specified
-Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber
ZE TEST PLAN
1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings.
2. perl misc/search_tools/rebuild_elastic_search.pl
3. Fetch all indexed records and the facet for subject__facet
curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
"aggregations": {
"my_agg": {
"terms": {
"field": "subject__facet"
}
}
}
}'
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
What we currently have:
Koha/ElasticSearch.pm
Koha/ElasticSearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm
What we want:
Koha/SearchEngine/Elasticsearch.pm
Koha/SearchEngine/Elasticsearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm
Test plan:
% git grep -i Koha::ElasticSearch
% git grep ElasticSearch|grep -v Catmandu::Store::ElasticSearch
should not return any result
Do a full reindex and search for records
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Jonathan Druart
Ere Maijala
Nick Clemens
Tomas Cohen Arazi
David Gustafsson
Blou
Olli-Antti Kivilahti