Koha-clms/C4
Julian Maurice 4605304f62 Bug 28306: Allow to query database with minimal memory footprint
The goal is to be able to build a database handler (dbh) and to execute
queries without loading unnecessary stuff. This will be useful to reduce
memory usage of daemons that need to check the database
periodically

The patch provides a new method Koha::Database::dbh which returns a
database handler without loading the DBIx::Class schema. This method is
also used by DBIx::Class, so whether you use DBI or DBIx::Class, the
same method is used to initialize the connection.

The patch also moves some code in order to avoid loading C4::Context:
- C4::Context::timezone moves to Koha::Config
- C4::Context::db_scheme2dbi moves to Koha::Database

To measure memory usage I used the following commands:

* before the patch:
perl -MKoha::Database \
    -E 'Koha::Database->schema->storage->dbh->do("select 1");' \
    -E '$|=1; say $$; sleep 2' \
    | while read pid; do ps -p $pid -o rss=; done

* after the patch:
perl -MKoha::Database \
    -E 'Koha::Database->dbh->do("select 1");' \
    -E '$|=1; say $$; sleep 2' \
    | while read pid; do ps -p $pid -o rss=; done

It will give you the RSS (Resident Set Size) of the perl process in kB

What I get:
* before the patch: between 96.9MB and 97.2MB
* after the patch: between 17.8MB and 18.2MB

Note that if a timezone is configured (either from $KOHA_CONF or
TZ environment variable), Koha will load DateTime::Timezone to check if
it's valid, and it increases RSS to 36MB

Another interesting metric is the number of modules loaded:
* before the patch:
perl -MKoha::Database \
    -E 'Koha::Database->schema->storage->dbh;' \
    -E 'say scalar keys %INC'

Result: 567

* after the patch:
perl -MKoha::Database \
    -E 'Koha::Database->dbh;' \
    -E 'say scalar keys %INC'

Result: 51

Test plan:
1. Apply the patch & restart starman
2. Make sure Koha is still ok (ie. can access the database, does not
have encoding issues, ...)
3. Run the tests in t/Context.t, t/Koha/Config.t,
t/db_dependent/Koha/Database.t, t/timezones.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2021-08-31 10:27:52 +02:00
..
AuthoritiesMarc Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Barcodes Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ClassSortRoutine Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ClassSplitRoutine Bug 28572: Remove C4::Debug 2021-06-22 12:04:32 +02:00
Creators Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
External Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Form Bug 28572: Remove C4::Debug 2021-06-22 12:04:32 +02:00
Heading Bug 26852: subfield $e missing in X11 definition of MARC21 headings 2021-08-11 13:27:52 +02:00
ILSDI Bug 28630: Add track_login_daily to ILSDI::AuthenticatePatron 2021-08-04 14:06:43 +02:00
Installer Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Labels Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Linker Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Members Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
OAI Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Output Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Patroncards Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Reports Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
SIP Bug 25464: Add ability to specify client IP and SIP account used in SIP2 logging 2021-07-26 12:46:32 +02:00
Search Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Serials Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Utils Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Accounts.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Acquisition.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Auth.pm Bug 12561: Remove non-XSLT views 2021-08-16 09:57:04 +02:00
Auth_cas_servers.yaml.sample
…
Auth_with_cas.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Auth_with_ldap.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Auth_with_shibboleth.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
AuthoritiesMarc.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
BackgroundJob.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Barcodes.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Biblio.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Breeding.pm Bug 17600: Add missing imports in authorities/ 2021-08-03 14:55:26 +02:00
Budgets.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Calendar.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Charset.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Circulation.pm Bug 28774: Don't store blank values for rental discount 2021-08-31 10:27:52 +02:00
ClassSortRoutine.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ClassSource.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ClassSplitRoutine.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Context.pm Bug 28306: Allow to query database with minimal memory footprint 2021-08-31 10:27:52 +02:00
Contract.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
CourseReserves.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Creators.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
HTML5Media.pm Bug 27422: YouTube videos are HTTPS only 2021-02-08 17:09:34 +01:00
Heading.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
HoldsQueue.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ImportBatch.pm Bug 28758: (bug 17600 follow-up) Import subroutines from C4/AuthoritiesMarc 2021-07-26 16:29:22 +02:00
ImportExportFramework.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
InstallAuth.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Installer.pm Bug 25078: Keep atomic updates in "atomicupdate" dir 2021-08-16 11:55:56 +02:00
ItemCirculationAlertPreference.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Items.pm Bug 22690: Remove uneeded return and add no_triggers 2021-08-27 15:28:23 +02:00
Koha.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Labels.pm Bug 21395: Make perlcritic happy 2020-06-29 12:37:02 +02:00
Languages.pm Bug 15067: Follow up to fix sorting 2021-08-04 14:06:43 +02:00
Letters.pm Bug 24387: Rename "News" with "Additional contents" 2021-08-16 11:55:55 +02:00
Linker.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Log.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
MarcModificationTemplates.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Matcher.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Members.pm Bug 24387: Rename "News" with "Additional contents" 2021-08-16 11:55:55 +02:00
Message.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Output.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Overdues.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Patroncards.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Record.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Reports.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Reserves.pm Bug 25619: Adjust POD and move date check before logging 2021-08-27 15:28:23 +02:00
Ris.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
RotatingCollections.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
SMS.pm Bug 27673: Replace YAML with YAML::XS 2021-02-16 14:54:50 +01:00
Scheduler.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Scrubber.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Search.pm Bug 28373: Add PassItemMarcToXSLT system preference 2021-08-31 08:49:04 +02:00
Serials.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Service.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
ShelfBrowser.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
SocialData.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Stats.pm Bug 17600: Fix tests 2021-07-16 09:47:01 +02:00
Suggestions.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
TTParser.pm Bug 26673: Remove #!/usr/bin/env perl from .pm modules too 2020-10-26 00:14:42 +01:00
Tags.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
Templates.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
TmplToken.pm
…
TmplTokenType.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
UsageStats.pm Bug 12561: Remove HighlightOwnItemsOnOPAC and HighlightOwnItemsOnOPACWhich 2021-08-16 09:57:04 +02:00
XISBN.pm Bug 17600: Standardize our EXPORT_OK 2021-07-16 08:58:47 +02:00
XSLT.pm Bug 26302: (QA follow-up) Prefix prefs with OPAC 2021-08-30 17:04:31 +02:00