Koha-clms/t/db_dependent
Jonathan Druart bffa27725a Bug 18066: Hea V2 [squashed]
This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Mason James <mtj@kohaaloha.com>
2017-05-24 13:44:59 +12:00
..
Acquisition Revert "Bug 17564: Fix t/db_dependent/Acquisition/OrderUsers.t" 2016-12-31 14:43:38 +13:00
Auth Bug 15756: Some tests for haspermission in C4::Auth 2016-02-23 20:54:42 +00:00
Authorities Bug 17913: [16.05.x] Authority merge fix 2017-02-24 09:42:24 +13:00
Biblio Bug 15572: Add tests for TransformHtmlToMarc 2016-01-25 07:09:50 -07:00
Circulation Bug 18242: 16.05 version - 16.11.x adaptation - Old::Checkouts vs OldIssues 2017-05-08 14:11:00 +12:00
Creators Bug 18044: Add a test 2017-02-16 07:23:50 +13:00
Exporter Bug 17644 - t/db_dependent/Exporter/Record.t fails 2016-12-12 14:54:12 +13:00
Holds Bug 17574: Remove itemtype-related warnings from LocalHoldsPriority.t 2016-12-12 14:54:09 +13:00
Items Bug 16607: Remove CPL/MPL from two unit tests 2016-06-24 07:28:45 +02:00
Koha Bug 15451: Add the 2 new modules Koha::CsvProfile[s] 2017-04-30 23:20:00 +12:00
LDAP
…
Labels Bug 14468: (QA followup) remove useless diags 2015-10-27 10:38:15 -03:00
Letters
…
Members Bug 18009 - IssueSlip.t test fails if launched between 00:00 and 00:59 2017-01-29 00:52:07 +13:00
OAI Bug 16172: fix OAI Server tests broken by bug 15946 2016-03-31 16:51:38 +00:00
Patron Bug 17603: Remove itemtype-related Borrower_Discharge.t warnings 2016-12-12 14:54:09 +13:00
Record Bug 13642 - Add Unit test for C4::Record::marc2dcxml 2016-01-27 06:23:08 +00:00
Reports Bug 12787 [QA Followup] - Retain change to Reports_Guided.t 2016-04-22 00:54:36 +00:00
Reserves Bug 13592: Add an option to charge for any hold placed 2015-12-31 15:59:05 +00:00
SIP Bug 15006: Centralize timeout logic and allow zero client timeout 2016-08-01 11:52:51 +02:00
Search Bug 17323: Column search_history.time cannot be null 2016-12-31 15:19:46 +13:00
Serials Bug 7728: QA fixes 2017-04-24 11:08:33 +12:00
Template/Plugin Bug 12787: Reorganise t/db_dependent files 2016-04-22 00:54:35 +00:00
Utils Bug 18094: Add tests to highlight the problem 2017-05-04 02:52:33 +12:00
XSLT_Handler
…
api/v1 Bug 16271: Allow more filters on /api/v1/holds 2016-08-01 11:11:13 +02:00
data
…
selenium Bug 13691: Add changes for 3.22 2016-08-01 11:41:11 +02:00
www Bug 10171: (QA followup) Make search tests use the right form 2016-05-04 14:14:50 +00:00
00-strict.t
…
Accounts.t Bug 16607: Remove CPL/MPL from two unit tests 2016-06-24 07:28:45 +02:00
Acquisition.t Bug 16934: Add test for ModOrder 2016-08-01 13:31:23 +02:00
AdditionalField.t Bug 10855: Remove additional field values when subscription is removed 2015-10-02 15:10:31 -03:00
Amazon.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
AudioAlerts.t Bug 11431: (QA followup) remove diags 2015-11-06 15:08:37 -03:00
Auth.t Bug 18275: Regression test 2017-03-18 02:24:06 +13:00
Auth_with_cas.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
Auth_with_ldap.t Bug 17615 - Fix unit tests 2017-01-25 14:53:46 +13:00
AuthorisedValues.t Bug 10363: Use Koha::AuthorisedValue[s] in the admin page 2015-10-19 12:46:27 -03:00
AuthoritiesMarc.t
…
BackgroundJob.t Bug 17426: Do not set AutoCommit in tests 2016-10-22 15:10:09 +02:00
Barcodes.t Bug 15878 - Updated unit tests for hbyymmincr barcodes 2016-05-23 17:42:10 +00:00
Barcodes_ValueBuilder.t Bug 15081: (QA followup) Fix issue with Barcodes_ValueBuilder.t 2015-11-04 12:32:57 -03:00
Biblio.t Bug 16431: Use Koha::Cache to cache marc subfield structure 2016-08-01 10:34:57 +02:00
Bookseller.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
Branch.t Bug 15629: Fix tests - Koha::Library does not have ->categories but ->get_categories 2016-02-24 03:55:07 +00:00
Breeding.t
…
Budgets.t Bug 15009 [QA Followup] - Update number of tests run 2016-04-29 13:01:07 +00:00
Category.t Bug 15081: (followup) Make test files using TestBuilder handle their transactions 2015-11-04 12:32:57 -03:00
Charset.t
…
Circulation.t Revert "Bug 14598: [QA Follow-up] Small changes" 2016-12-31 14:35:57 +13:00
ClassSource.t Bug 16860: Remove the comment about the warning 2016-08-01 10:50:08 +02:00
ColumnsSettings.t Bug 16177: Fix tests for ColumnsSsettings.t 2016-04-01 19:08:23 +00:00
Context.t Bug 16813: Allow empty string for OPACBaseURL 2016-08-01 13:31:21 +02:00
Contract.t
…
CourseReserves.t Bug 15530 - Add Unit Test 2016-01-27 00:58:14 +00:00
DecreaseLoanHighHolds.t Bug 17573: Remove itemtype-related warnings from DecreaseLoanHighHolds.t 2016-12-12 14:54:09 +13:00
Filter_MARC_ViewPolicy.t Bug 15870: Follow up to address comment #20 2016-03-24 20:15:40 +00:00
Fines.t Bug 13590: Add ability to charge fines at start of charge period 2015-10-22 14:51:24 -03:00
FrameworkPlugin.t Bug 14199: Unify marc orgcode plugins, not overwriting existing values 2016-02-26 14:04:28 +00:00
Heading_MARC21.t
…
Hold.t Bug 16609: Catch warning from Koha::Hold in Hold.t 2016-06-15 09:04:06 +02:00
Holds.t Bug 16717: Improve variable naming 2016-06-24 07:29:30 +02:00
HoldsQueue.t Fix t/db_dependent/HoldsQueue.t tests 2017-05-05 21:13:46 +12:00
Holidays.t Bug 16376: (regression tests) 2017-01-03 14:22:37 +13:00
ILSDI_Services.t Bug 16155: (QA followup) fix small bug in t/db_dependent/ILSDI_Services.t 2016-05-04 13:47:58 +00:00
ImportBatch.t Bug 9259: Use is instead of is_deeply 2016-08-01 10:59:55 +02:00
Installer.t Bug 17234: Need to separate KEY and FOREIGN KEY checks 2017-01-25 13:19:12 +13:00
Items.t Bug 17512: Improve handling dates in C4::Items 2017-02-23 09:36:21 +13:00
KitchenSinkPlugin.kpz Bug 17368: Update KitchenSinkPlugin.kpz 2016-10-22 15:12:40 +02:00
Koha.t Bug 16157: Move the selected flag from GetAuthorisedValues to the templates 2016-04-07 00:16:09 +00:00
Koha_Authority.t Bug 15578: Insert authority records before processing tests 2016-01-23 18:28:49 +00:00
Koha_Database.t
…
Koha_Elasticsearch. Bug 17372: Standardize Elasticsearch paths 2016-10-22 13:56:11 +02:00
Koha_Elasticsearch_Indexer.t Bug 17372: Standardize Elasticsearch paths 2016-10-22 13:56:11 +02:00
Koha_Misc_Files.t
…
Koha_SearchEngine_Elasticsearch_Search.t Bug 18068: ES - Fix location and (home|holding)branch facets 2017-02-23 10:26:47 +13:00
Koha_borrower_modifications.t Revert "Bug 17494: Prevent duplicate tokens from getting stored" 2016-12-31 14:44:57 +13:00
Languages.t Bug 16866: Catch warning t/db_dependent/Languages.t 2016-08-01 11:33:45 +02:00
Letters.t Bug 17441: [QA Follow-up] Return value of SendAlerts 2016-10-22 14:55:21 +02:00
Linker_FirstMatch.t Bug 16868: Silence error t/db_dependent/Linker_FirstMatch.t 2016-08-05 09:00:34 +02:00
Log.t Bug 15939: Action logs - Do not default dates to today 2016-03-02 00:53:14 +00:00
MarcModificationTemplates.t Bug 15669: Add a test for GetModificationTemplates 2016-03-03 22:11:44 +00:00
Members.t Bug 16819: Unit tests for C4::Members::DelMember 2016-08-01 11:32:24 +02:00
MungeMarcPrice.t Bug 15084: Replace C4::Budgets::GetCurrencies with Koha::Acquisition::Currencies->search 2016-03-03 20:39:01 +00:00
NewsChannels.t Bug 16550: Clean the tests 2016-05-23 17:37:48 +00:00
Overdues.t Bug 13624 - Remove columns branchcode, categorytype from table overduerules_transport_types 2015-12-31 12:47:16 +00:00
Passwordrecovery.t Bug 18025: Fix test 2017-02-23 10:26:47 +13:00
Patron.t Bug 14577 [QA Followup] 2016-04-29 11:54:23 +00:00
PatronLists.t Bug 16452: Remove the warnings raised by PatronLists.t 2016-05-05 20:44:47 +00:00
Patrons.t Bug 17742: Use TestBuilder to create the library and patron category 2017-01-25 12:47:57 +13:00
Plugins.t Bug 16502: Replace a few other ok-calls by is-calls 2016-06-24 07:25:41 +02:00
QueryParser.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
Ratings.t Bug 14878: Tests - Create the branchcodes when needed 2015-11-06 15:51:51 -03:00
Record.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
RecordProcessor_EmbedSeeFromHeadings.t Bug 15380: Move Koha::Authority to Koha::MetadataRecord::Authority 2015-12-31 18:50:41 +00:00
Reports.t
…
Reserves.t Bug 14968 - Provides unit test 2016-08-01 10:33:02 +02:00
Review.t Bug 15548: Move new patron related code to Patron* 2016-03-03 14:38:26 -07:00
RotatingCollections.t Bug 15629: Koha::Libraries - Remove ModBranch 2016-02-24 03:55:06 +00:00
Search.t Revert "Bug 17917: [QA Follow-up] Reprove Search.t" 2017-01-27 17:07:56 +13:00
Serials.t Bug 18460: Fix undefined itemtype warning in Serials.t 2017-04-30 23:19:59 +12:00
Serials_2.t Bug 14878: Tests - Create the branchcodes when needed 2015-11-06 15:51:51 -03:00
Service.t
…
ShelfBrowser.t Bug 14878: Tests - Create the branchcodes when needed 2015-11-06 15:51:51 -03:00
Sitemapper.t Bug 17920: t/db_dependent/Sitemapper.t fails because of permissions 2017-01-25 15:05:56 +13:00
Stats.t
…
SuggestionEngine_ExplodedTerms.t Bug 16869: Silence and catch warnings in t/db_dependent/SuggestionEngine_ExplodedTerms.t 2016-08-01 10:43:57 +02:00
Suggestions.t Bug 16325: Add a test for SearchSuggestions when searching for STATUS='' 2016-05-05 20:46:01 +00:00
Tags.t
…
Templates.t Bug 18010: Unit test for gettemplate 2017-04-04 03:45:18 +12:00
TestBuilder.t Bug 17726: [QA Follow-up] Add test descriptions 2017-02-22 21:58:09 +13:00
Upload.t Bug 16501: Remove some unneeded warns in Upload.t 2016-05-16 17:46:25 +00:00
UsageStats.t Bug 18066: Hea V2 [squashed] 2017-05-24 13:44:59 +12:00
Virtualshelves.t Bug 14544: [QA Follow-up] Added some test descriptions 2015-11-05 10:00:31 -03:00
XISBN.t Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
XSLT_Handler.t
…
check_sysprefs.t Revert "Bug 17917: [AMENDED] Fix path issue in t/db_dependent/check_sysprefs.t" 2017-01-27 17:07:44 +13:00
db_structure.t Bug 16173: db_structure.t shouldn't have a fixed number of tests 2016-04-01 19:14:53 +00:00
default_search_class.pl Bug 11998: Use t::lib::Mocks::mock_preference in tests 2016-03-15 07:08:28 +00:00
rollingloans.t
…
sysprefs.t Bug 11998: Use Koha::Cache to cache sysprefs 2016-03-15 07:08:28 +00:00
zebra_config.pl Bug 15082: t/db_dependent/Search.t shouldn't mock the DB connection 2015-10-29 14:10:23 -03:00