Koha-clms/Koha
Jonathan Druart 63a1adff90 Bug 18066: Hea V2
This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
  hea-ws  - https://github.com/joubu/hea-ws/commits/v2
  hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.

More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.

Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).

With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).

The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
  1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
  2. Link the Privacy_Policy.md from the Hea interface
  3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.

You can find screenshot of this whole enhancement on bug 18066, comment 2.

What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
  * UsageStatsGeolocation
  * UsageStatsLibrariesInfo
  * UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations

How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.

TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?

Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
  CREATE DATABASE hea
  CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
  GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
  FLUSH PRIVILEGES;
3/ Fill the DB with some data
  mysql hea < hea-app/sql/schema.sql
  mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
  % cd hea-app
  % perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
  % perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
  DocumentRoot "/path/to/hea-ws"
  ServerName "hea.koha-community.org"
  <Directory "/">
    Options +ExecCGI
    Require all granted
    AddHandler cgi-script .pl
  </Directory>
</VirtualHost>

And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.

6/ Launch the hea-app
  % cd hea-app
  % edit README.md # to install the missing modules
  % cp environments/config.yml environments/development.yml
  % edit environments/development.yml # to fill the DB info
  % perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data

7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
  % perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified

B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2017-03-23 00:10:46 +00:00
..
Account Bug 17586: Move ->get_balance to Koha::Account->balance 2016-12-16 14:50:18 +00:00
Acquisition Bug 13726: Make Koha::Acq::Bookseller using Koha::Object 2016-12-30 11:54:32 +00:00
ArticleRequest Bug 14610 - Add and update modules 2016-10-26 12:15:13 +00:00
Authority Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Biblio Bug 17196: Move marcxml out of the biblioitems table 2017-01-13 13:49:26 +00:00
BiblioUtils Bug 17196: [QA Follow-up] Adjust some text on marcxml 2017-01-13 13:49:30 +00:00
Cache Bug 16088: Introduce Koha::Cache::Memory::Lite to cache the language 2016-06-04 07:43:22 +00:00
Config Bug 11998: Use Koha::Cache to cache sysprefs 2016-03-15 07:08:28 +00:00
Edifact BUG 17414 Add new GIR codes in Edifact 2016-10-11 14:14:12 +00:00
Exceptions Bug 17767: Make Koha::Patron::Modification handle extended attributes 2016-12-28 13:53:02 +00:00
Exporter Bug 17088: [Follow-up] Use Logger for failed exports 2016-10-10 12:24:00 +00:00
ExternalContent bug_16034 Koha::ExternalContent::OverDrive - a wrapper around WebService::ILS::Overdrive::Patron 2017-02-21 19:58:20 +00:00
Filter/MARC Bug 16365: Selectively introduce GetMarcStructure() "unsafe" variant for better performance 2016-09-09 10:43:22 +00:00
Indexer Revert Bug 11081 - Port Koha::Contrib::Tamil indexer into Koha code base 2016-03-01 16:56:35 +00:00
Item Bug 16686: Add Koha::Object based class for Koha::Item::Transfer[s] 2016-09-02 13:56:03 +00:00
List Bug 15065: (follow-up) update license header 2013-10-14 21:50:10 +00:00
MetadataRecord Bug 16708 - (QA followup) Fix pod copy/paste error 2016-06-24 11:59:40 +00:00
Middleware Bug 16520 (Follow up) Rebase fixes 2016-10-11 12:37:17 +00:00
Misc Bug 16011: $VERSION - Remove use vars qw(); 2016-03-24 17:20:39 +00:00
Notice Bug 17959 - Add the Koha::Notice::Message[s] packages (message_queue table) 2017-02-17 11:48:44 +00:00
Number Bug 16768 - Add official number format for Switzerland: 1'234'567.89 2016-06-24 14:00:03 +00:00
OAI/Server Bug 17196: Move marcxml out of the biblioitems table 2017-01-13 13:49:26 +00:00
Old Bug 17796: Move Koha::OldIssue[s] to Koha::Checkout[s] 2016-12-20 15:13:29 +00:00
Patron Bug 18025 - Expired password recovery links cause sql crash 2017-02-17 11:24:39 +00:00
Plugins Bug 17935: Follow up for C4::Plugin... 2017-02-14 14:12:51 +00:00
QueryParser/Driver Bug 15258: Fix Perl scripts declaring unused variables 2015-12-30 17:24:45 -07:00
REST Bug 17992: REST api: Remove the use of ->unblessed from Cities controller 2017-03-03 16:57:40 +00:00
RecordProcessor Bug 16011: $VERSION - Remove the $VERSION init 2016-03-24 17:20:28 +00:00
SMS Bug 15548 [QA Followup] - More new uses of Koha::Borrower 2016-03-03 14:39:00 -07:00
Schema/Result Bug 18066: Hea V2 2017-03-23 00:10:46 +00:00
SearchEngine Bug 17169: Add a facet for ccode fields to ElasticSearch 2017-03-22 19:24:22 +00:00
Serial Bug 7677: QA followup 2016-09-12 09:13:57 +00:00
Sitemapper Bug 11190: (QA followup) for tests on jenkins 2015-08-31 10:32:34 -03:00
Subscription Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
SuggestionEngine Bug 9978: Replace license header with the correct license (GPLv3+) 2015-04-20 09:59:38 -03:00
Template/Plugin Bug 16530: Add a new method to the Branches TT Plugin to avoid c/p 2017-03-03 18:34:36 +00:00
Tmp Bug 17962: Add Koha::Tmp::Order[s] 2017-02-17 11:43:46 +00:00
Util Bug 17302: [Follow-up] Make Normalize.pm undef-resistent 2016-09-25 14:04:54 +00:00
Account.pm Bug 17894 - Update pay() and use it internally for WriteOffFee 2017-01-19 11:15:25 +00:00
AdditionalField.pm Bug 10855: Remove additional field values when subscription is removed 2015-10-02 15:10:31 -03:00
ArticleRequest.pm Bug 17634: Fix circ dependency with require instead of use 2016-11-16 10:57:56 +00:00
ArticleRequests.pm Bug 14610 - Add and update modules 2016-10-26 12:15:13 +00:00
AudioAlert.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
AudioAlerts.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
AuthUtils.pm Bug 16818: External auth redirect broken under Plack 2016-08-04 19:42:44 +00:00
AuthorisedValue.pm Bug 17935: Adjust some POD lines, fix a few typos 2017-02-14 14:12:50 +00:00
AuthorisedValueCategories.pm Bug 17216: Add Koha::AuthorisedValueCategory|ies classes 2016-10-11 07:30:30 +00:00
AuthorisedValueCategory.pm Bug 17216: Add Koha::AuthorisedValueCategory|ies classes 2016-10-11 07:30:30 +00:00
AuthorisedValues.pm Bug 17642: Do not explode if no authorised value exist 2016-11-18 15:52:01 +00:00
Authorities.pm Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Authority.pm Bug 16708: Fix authority reindex for ElasticSearch 2016-06-24 11:59:40 +00:00
Biblio.pm Bug 17630: Add the Koha::Biblio->holds method 2016-12-16 14:51:53 +00:00
BiblioFramework.pm Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
BiblioFrameworks.pm Bug 14889: Fix typo in Koha/BiblioFrameworks.pm 2016-03-03 20:32:47 +00:00
BiblioUtils.pm Bug 12748: (QA followup) fix several tiny QA tools warnings 2016-04-26 20:20:13 +00:00
Biblioitem.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Biblioitems.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Biblios.pm Bug 14610 - Add and update modules 2016-10-26 12:15:13 +00:00
Cache.pm Bug 17935: Adjust some POD lines, fix a few typos 2017-02-14 14:12:50 +00:00
Caches.pm Bug 17935: Adjust some POD lines, fix a few typos 2017-02-14 14:12:50 +00:00
Calendar.pm Bug 16376 - Koha::Calendar->is_holiday date truncation creates fatal errors for TZ America/Santiago 2016-12-09 15:25:01 +00:00
Checkout.pm Bug 17973: Add the Koha::Checkout->item method 2017-02-17 12:11:30 +00:00
Checkouts.pm Bug 14757 - Allow the use of Template Toolkit syntax for slips and notices 2016-07-08 13:47:42 +00:00
Cities.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
City.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Config.pm Bug 11921: (followup) Don't die on non-existent koha-conf.xml 2016-09-09 13:56:13 +00:00
CsvProfile.pm Bug 15451: Add the 2 new modules Koha::CsvProfile[s] 2016-07-22 17:18:34 +00:00
CsvProfiles.pm Bug 15451: Add the 2 new modules Koha::CsvProfile[s] 2016-07-22 17:18:34 +00:00
Database.pm Bug 18215: Resolve warning on $tls in Database.pm 2017-03-06 14:10:33 +00:00
DateUtils.pm Bug 16848: Prevent invalid warning to be carped from output_pref 2016-08-10 13:43:34 +00:00
EDI.pm Bug 16288: Remove Reference to obsolete EAN syspref 2016-05-16 17:14:56 +00:00
Edifact.pm Bug 7736: Support Ordering via Edifact EDI messages 2016-04-01 20:03:17 +00:00
Email.pm Bug 9530: Adding a new Koha::Email module 2014-10-27 10:38:14 -03:00
Exceptions.pm Bug 14048: Add RefundLostItemFeeRule table and classes 2016-07-07 10:37:01 -06:00
ExternalContent.pm bug_16034 Koha::ExternalContent::OverDrive - a wrapper around WebService::ILS::Overdrive::Patron 2017-02-21 19:58:20 +00:00
FrameworkPlugin.pm Bug 9006: Remove AUTOLOAD in C4::Context 2015-07-20 10:11:41 -03:00
Hold.pm Bug 14642: Make sure the log will be written only if the hold has been deleted 2016-08-17 18:43:13 +00:00
Holds.pm Bug 17327: (QA followup) Remove Carp::Always which is not used 2016-09-24 20:55:17 +00:00
I18N.pm Bug 11848: (follow-up) update license statment 2014-05-05 04:51:42 +00:00
IssuingRule.pm Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
IssuingRules.pm Bug 17783: Prevent crash when providing an undefined value 2016-12-23 12:01:57 +00:00
Item.pm Bug 17974 (QA Followup) 2017-02-17 12:11:54 +00:00
ItemType.pm Bug 17589 - Improper method type in Koha::ItemType(s) 2016-11-08 17:12:27 +00:00
ItemTypes.pm Bug 17589 - Improper method type in Koha::ItemType(s) 2016-11-08 17:12:27 +00:00
Items.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Libraries.pm Bug 15758: Koha::Libraries - Move onlymine to C4::Context::only_my_library 2016-09-08 14:36:04 +00:00
Library.pm Bug 15446 [QA Followup] - Update new classes 2016-02-12 12:35:57 -07:00
LibraryCategories.pm Bug 15446 [QA Followup] - Update new classes 2016-02-12 12:35:57 -07:00
LibraryCategory.pm Bug 15629: Koha::Libraries - Remove GetBranchesInCategory (1) 2016-02-24 03:55:05 +00:00
Linktracker.pm Bug 8917 : [ Sign followup ] Track clicks to links external to Koha 2013-03-07 11:14:25 -05:00
Localization.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Localizations.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Logger.pm Bug 17088: [Follow-up] Adjust POD in Logger.pm 2016-10-10 12:24:00 +00:00
MarcSubfieldStructure.pm Bug 17249: Remove GetKohaAuthorisedValuesFromField - Add classes MarcSubfieldStructure[s] 2016-10-21 15:13:55 +00:00
MarcSubfieldStructures.pm Bug 17249: Remove GetKohaAuthorisedValuesFromField - Add classes MarcSubfieldStructure[s] 2016-10-21 15:13:55 +00:00
MetaSearcher.pm Bug 16769: Uniformise calls to Koha::Cache->set_in_cache 2016-10-10 17:03:54 +00:00
MetadataIterator.pm Bug 12478 - authorities can now be stored in ES 2016-04-26 20:20:03 +00:00
MetadataRecord.pm Bug 17201: (bug 16431 follow-up) Remove occurrence of marcfromkohafield 2016-09-02 14:00:11 +00:00
News.pm Bug 14757 - Allow the use of Template Toolkit syntax for slips and notices 2016-07-08 13:47:42 +00:00
NewsItem.pm Bug 14757 - Allow the use of Template Toolkit syntax for slips and notices 2016-07-08 13:47:42 +00:00
NorwegianPatronDB.pm Bug 16011: $VERSION - Remove comments 2016-03-24 17:20:29 +00:00
Object.pm Bug 17932: Add a TO_JSON method to Koha::Object(s) 2017-02-17 15:32:59 +00:00
Objects.pm Bug 17932: Add a TO_JSON method to Koha::Object(s) 2017-02-17 15:32:59 +00:00
Patron.pm Revert "Bug 12461 - Add patron clubs feature" 2017-02-26 20:41:27 -05:00
Patrons.pm Bug 18169: Make 'before' param non mandatory for Koha::Patrons->anonymise_issue_history 2017-03-03 18:16:54 +00:00
Plugins.pm Bug 17461: Warn about plugins that can't be loaded 2017-02-17 17:19:22 +00:00
Rating.pm Bug 17089: Mark out the boundaries of ratings 2016-09-09 09:31:30 +00:00
Ratings.pm Bug 17089: Koha::Ratings - Remove GetRating 2016-09-09 09:31:29 +00:00
RecordProcessor.pm Bug 15870: potential follow up to comment #8 2016-03-24 20:15:40 +00:00
RefundLostItemFeeRule.pm Bug 14048: (QA followup) Default to 1 if no default rule 2016-07-07 10:37:01 -06:00
RefundLostItemFeeRules.pm Bug 14048: (QA followup) Default to 1 if no default rule 2016-07-07 10:37:01 -06:00
Review.pm Bug 15839: Koha::Reviews - Remove approvereview & unapprovereview 2016-09-09 10:29:57 +00:00
Reviews.pm Bug 15839: Koha::Reviews - Add Koha::Review[s] classes 2016-09-09 10:29:31 +00:00
Schema.pm Bug 11891: (follow-up) remove custom storage class 2014-06-19 11:55:03 -03:00
SearchEngine.pm Bug 12478: make things using SimpleSearch use the new version 2016-04-26 20:20:07 +00:00
SearchField.pm Bug 12748: (QA followup) fix several tiny QA tools warnings 2016-04-26 20:20:13 +00:00
SearchFields.pm Bug 12748: (QA followup) fix several tiny QA tools warnings 2016-04-26 20:20:13 +00:00
SearchMarcMap.pm Bug 12748: (QA followup) fix several tiny QA tools warnings 2016-04-26 20:20:13 +00:00
SearchMarcMaps.pm Bug 12748: (QA followup) fix several tiny QA tools warnings 2016-04-26 20:20:13 +00:00
Serial.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Serials.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
SimpleMARC.pm Bug 16011: $VERSION - Remove the $VERSION init 2016-03-24 17:20:28 +00:00
Sitemapper.pm Bug 11190: (QA followup) for tests on jenkins 2015-08-31 10:32:34 -03:00
Subscription.pm Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Subscriptions.pm Bug 15446: (follow-up) Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Suggestion.pm Bug 14757 - Allow the use of Template Toolkit syntax for slips and notices 2016-07-08 13:47:42 +00:00
SuggestionEngine.pm Bug 9978: Replace license header with the correct license (GPLv3+) 2015-04-20 09:59:38 -03:00
Suggestions.pm Bug 14757 - Allow the use of Template Toolkit syntax for slips and notices 2016-07-08 13:47:42 +00:00
Token.pm Bug 17110: Lower CSRF expiry in Koha::Token 2016-09-09 13:40:41 +00:00
UploadedFile.pm Bug 17501: [Follow-up] QA Requests 2017-01-20 14:20:07 +00:00
UploadedFiles.pm Bug 17501: [Follow-up] QA Requests 2017-01-20 14:20:07 +00:00
Uploader.pm Bug 17501: Rename Upload to Uploader 2017-01-20 14:20:06 +00:00
Virtualshelf.pm Bug 17094: Make Koha::Virtualshelf methods return Koha::Objects-based objects 2016-10-11 13:14:46 +00:00
Virtualshelfcontent.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Virtualshelfcontents.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Virtualshelfshare.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Virtualshelfshares.pm Bug 15446: Rename Koha::Object[s]->type with _type 2016-02-12 12:35:57 -07:00
Virtualshelves.pm Bug 16551: Display the name of lists to the search results at the OPAC 2016-09-25 15:43:07 +00:00
XSLT_Handler.pm Bug 13262 - Add parameters to XSLT Handler transform method 2015-02-05 15:24:29 -03:00