gitea
/
Koha-clms
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bug 35345: Add --where option to rebuild_elasticsearch.pl 

Sometimes we need to only re-index a subset of our bibliographic data or authorities. Currently this is only possible by enumerating all id (-bn or -ai), which does not work well when indexing eg 100.000 items of a 2.000.000 DB. Re-indexing everything is also overkill.

This patch adds an `--where` flag to misc/search_tools/rebuild_elasticsearch.pl which can take arbitrary SQL (that of course has to match the respective tables) and adds it as an additional param to the resultset to index

To test, start koha-testing-docker with ElasticSearch enabled, for example via `ktd --es7 up

Before applying the patch, rebuild_elasticsearch will index all data:

Biblios:
$ misc/search_tools/rebuild_elasticsearch.pl -b -v
[12387] Checking state of biblios index
[12387] Indexing biblios
[12387] Committing final records...
[12387] Total 435 records indexed
(there might be a waring regarding a broken biblio, which can be ignored)

Auth:
$ misc/search_tools/rebuild_elasticsearch.pl -a -v
[12546] Checking state of authorities index
[12546] Indexing authorities
[12546] 1000 records processed
[12546] Committing final records...
[12546] Total 1706 records indexed

Now apply the patch

Biblio, limit by range of biblioid:
$ misc/search_tools/rebuild_elasticsearch.pl -b -v --where "biblionumber between 100 and 150"
[12765] Checking state of biblios index
[12765] Indexing biblios
[12765] Committing final records...
[12765] Total 50 records indexed

Note that only 50 records where indexed (instead of the whole set of 435 records)

Auth, limit by authtypecode:
$ misc/search_tools/rebuild_elasticsearch.pl -a -v --where "authtypecode = 'GEOGR_NAME'"
[12848] Checking state of authorities index
[12848] Indexing authorities
[12848] Committing final records...
[12848] Total 142 records indexed

Again, only 142 have been indexed.

Sponsored-by: Steiermärkische Landesbibliothek
Sponsored-by: HKS3 / koha-support.eu

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 61e7aa374e)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This commit is contained in:
Thomas Klausner 2024-03-29 09:20:37 +00:00 committed by Fridolin Somers
parent 8c9501396e
commit 61c8fc97fc
3 changed files with 21 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
Koha/BiblioUtils.pm
View File

@ -140,6 +140,11 @@ sub get_all_biblios_iterator {
my $rs = Koha::Biblios->search(
$search_terms,
$search_options );
if ( my $sql = $options{where} ) {
$rs = $rs->search( \[$sql] );
}
my $next_func = sub {
# Warn and skip bad records, otherwise we break the loop
while (1) {

5
Koha/MetadataRecord/Authority.pm
View File

@ -201,6 +201,11 @@ sub get_all_authorities_iterator {
$schema->resultset('AuthHeader')->search(
$search_terms,
$search_options);
if ( my $sql = $options{where} ) {
$rs = $rs->search( \[$sql] );
}
my $next_func = sub {
# Warn and skip bad records, otherwise we break the loop
while (1) {

12
misc/search_tools/rebuild_elasticsearch.pl
View File

@ -34,6 +34,7 @@ B<rebuild_elasticsearch.pl>
[B<--desc>]
[B<-bn|--bnumber>]
[B<-ai|--authid>]
[B<-w|--where SQL>]
[B<-p|--processes>]
[B<-v|--verbose>]
[B<-h|--help>]
@ -87,6 +88,10 @@ repeated.
Only index the supplied authority id, mostly for testing purposes. May be
repeated.
=item B<-w|--where>
Pass some additional SQL to limit the records to be indexed.
=item B<-p|--processes>
Number of processes to use for indexing. This can be used to do more indexing
@ -128,7 +133,7 @@ my $verbose = 0;
my $commit = 5000;
my ($delete, $reset, $help, $man, $processes);
my ($index_biblios, $index_authorities);
my (@biblionumbers,@authids);
my (@biblionumbers,@authids,$where);
my $desc;
$|=1; # flushes output
@ -142,6 +147,7 @@ GetOptions(
'desc' => \$desc,
'bn|bnumber=i' => \@biblionumbers,
'ai|authid=i' => \@authids,
'w|where=s' => \$where,
'p|processes=i' => \$processes,
'v|verbose+' => \$verbose,
'h|help' => \$help,
@ -197,6 +203,10 @@ if( $desc ){
$iterator_options{desc} = 1;
}
if ($where) {
$iterator_options{where} = $where;
}
my $next;
if ($index_biblios) {
_log(1, "Indexing biblios\n");