23ff8a7b1d
This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.
The only change is that it will now pass a LIMIT <batch_size> to the
query.
To test:
0. Have KTD started with ES (skip if you have an instance running):
$ ktd --es7 up -d
1. Stop the indexer and rabbit:
$ ktd --shell
k$ sudo koha-es-indexer --stop kohadev
k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D
[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit
|
||
|---|---|---|
| C4 | ||
| Koha | ||
| acqui | ||
| admin | ||
| api | ||
| authorities | ||
| basket | ||
| bin | ||
| bookings | ||
| catalogue | ||
| cataloguing | ||
| circ | ||
| clubs | ||
| course_reserves | ||
| debian | ||
| docs | ||
| erm | ||
| errors | ||
| etc | ||
| ill | ||
| installer | ||
| koha-tmpl | ||
| labels | ||
| lib | ||
| members | ||
| misc | ||
| offline_circ | ||
| opac | ||
| patron_lists | ||
| patroncards | ||
| plugins | ||
| pos | ||
| preservation | ||
| recalls | ||
| reports | ||
| reserve | ||
| reviews | ||
| rotating_collections | ||
| serials | ||
| services | ||
| skel | ||
| suggestion | ||
| svc | ||
| t | ||
| tags | ||
| tools | ||
| virtualshelves | ||
| xt | ||
| .editorconfig | ||
| .eslintrc.json | ||
| .gitignore | ||
| .htaccess | ||
| .mailmap | ||
| .perlcriticrc | ||
| .perltidyrc | ||
| .prettierrc.js | ||
| .proverc.dist | ||
| .stylelintrc.json | ||
| INSTALL | ||
| Koha.pm | ||
| LICENSE | ||
| MANIFEST.SKIP | ||
| Makefile.PL | ||
| README | ||
| README.md | ||
| README.robots | ||
| about.pl | ||
| app.psgi | ||
| build-resources.PL | ||
| changelanguage.pl | ||
| cpanfile | ||
| cypress.config.ts | ||
| fix-perl-path.PL | ||
| gulpfile.js | ||
| help.pl | ||
| kohaversion.pl | ||
| mainpage.pl | ||
| package.json | ||
| rewrite-config.PL | ||
| rspack.config.js | ||
| tsconfig.json | ||
| webpack.config.js | ||
| yarn.lock | ||
README.md
Koha is a free software integrated library system (ILS).
Koha is distributed under the GNU GPL version 3 or later.
Note: Koha does not accept pull requests from git hosting sites.
Note: This project has its own bug tracker, to report a bug or submit a patch visit http://bugs.koha-community.org.
For guidelines on submitting patches for Koha please visit https://wiki.koha-community.org/wiki/SubmitingAPatch
The developers handbook can be found at https://wiki.koha-community.org/wiki/Developer_handbook
