This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
hea-ws - https://github.com/joubu/hea-ws/commits/v2
hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.
More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.
Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).
With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).
The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
2. Link the Privacy_Policy.md from the Hea interface
3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.
You can find screenshot of this whole enhancement on bug 18066, comment 2.
What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
* UsageStatsGeolocation
* UsageStatsLibrariesInfo
* UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations
How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.
TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?
Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
CREATE DATABASE hea
CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
FLUSH PRIVILEGES;
3/ Fill the DB with some data
mysql hea < hea-app/sql/schema.sql
mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
% cd hea-app
% perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
% perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
DocumentRoot "/path/to/hea-ws"
ServerName "hea.koha-community.org"
<Directory "/">
Options +ExecCGI
Require all granted
AddHandler cgi-script .pl
</Directory>
</VirtualHost>
And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.
6/ Launch the hea-app
% cd hea-app
% edit README.md # to install the missing modules
% cp environments/config.yml environments/development.yml
% edit environments/development.yml # to fill the DB info
% perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data
7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
% perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified
B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit 63a1adff90)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended to include items and deleteditems.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 69a24812ed)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 3e1f21ae0d)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
The loading of file admin/searchengine/elasticsearch/mappings.yaml
specifies 'type' as empty, which fails with Mysql 5.7+ which is more
stringent.
Also, forcing an empty value into a boolean also fails.
Both issues arise when updatedatabase.pl hit 3.23.00.050.
NOTE: both issues could also be resolved by actually setting values in
the load file. This doesn't make this solution incorrect, though.
To Test/reproduce:
-1) Happens with Mysql 5.7.4. Maybe earlier, but certainly at
that point. Use a Xenial kohadevbox to more easily test.
0) Find a database on 3.22 or earlier, save it.
a) place
sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
into the [mysqld] section of /etc/mysql/mysql.conf.d/mysqld.cnf
b) restart you mysql server
c) drop the db and recreate it
d) checkout the 3.22.x branch
e) do a web install
f) remove the added sql_mode
g) restart the mysql server
1) Set your code base to master
2) run updatedatabase.pl
3) See the errors on 3.23.00.050
4) Apply the patch
5) Reload the 3.22 db.
a) repeat steps 0(a)-0(g)
b) don't forget the caching issues
6) succeed with updatedatabase.pl
7) drop the db and recreate it
8) run the web installer
9) notice no issues either.
10) run koha qa test tools
NOTE: This bug only solved the upgrade portion.
I added the kohastructure.sql change as well.
I confirmed that all the code changes were
triggered with this test plan.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mehdi Hamidi <mehdi.hamidi@inlibro.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 77337a2ab5)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
We want to update it if it's empty, not equal to ';'
nopaste vim mode is to blame
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 11bae489bc)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Will fix
ERROR 1136 (21S01) at line 57: Column count doesn't match value count at row 2
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 2ce4b1635d)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c7dc340d63)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Adds missing notices to the fr-CA web installer:
- MEMBERSHIP_EXPIRY
- PASSWORD_RESET
Renames RESERVESLIP to HOLD_SLIP
TEST PLAN
---------
1) Apply the first patch
2) prove xt/sample_notices.t
-- it should fail for fr-CA.
3) Apply the second patch
4) prove xt/sample_notices.t
-- it should pass.
5) run all tests (prove t; prove xt)
-- they should generally pass
6) run koha qa test tools
NOTE: Split test patch from fix patch, so could prove problem
exists easily.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Caitlin Goodger <caitlingoodger.student@wegc.school.nz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit b37b22a4fd)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 10459 has been backported and the DB entry (add borrowers.updated_on and
deletedborrower.updated_on) is now played in
- 16.06.00.027
- 16.05.00.002
- 3.22.08.001
This will raise a MySQL warning if the column already exists.
DBD::mysql::db do failed: Duplicate column name 'updated_on'
Since bug 17234 we have now a subroutine (C4::Installer::column_exists) to test
if a column exists.
When a DB entry modifying the DB structure is backported, it HAS TO check if the
column, constraint or table exists before executing the query.
Test plan:
git checkout 3.22.x (16.05.x or 16.11.x)
install Koha
git checkout master
execute the installer
=> Without this patch you will get a warning when adding borrowers.updated_on)
=> With this patch, you should not get it
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 2b303129b8)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
In the previous patch we use the constraint_exists subroutine to verify
if an index or a foreign key exists.
But the `SHOW INDEX` query does not return foreign keys (as its name
suggests!).
We need another subroutine foreign_key_exists to check the FK existence.
I have found that because t/db_dependent/TestBuilder.t fails on
oai_sets_biblios, because oai_sets_biblios_ibfk_1 has not been removed.
Test plan:
0/ Do not apply this patch
1/ Use a 3.20 DB
2/ update the DB
3/ SHOW CREATE TABLE oai_sets_biblios
will display oai_sets_biblios_ibfk_1
Apply the patch and repeat 1, 2, 3
=> Will not display oai_sets_biblios_ibfk_1
It has been removed as expected.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 05fdd855c8)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0b15c5e2cc)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
These 2 subroutines will help us deal with the absense of ALTER IGNORE
TABLE
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c08922180e)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 16276 added two more ALTER IGNORES. This removes them.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 4ea484677e)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
The doc says: "As of MySQL 5.7.4, the IGNORE clause for ALTER TABLE is removed and its use produces an error."
This fix replaces ALTER IGNORE with ALTER in updatedatabase.pl
To TEST, try an upgrade from 3.18 to 3.22 after installing the latest mysql (at least 5.7.4, by the doc).
Some will fail with error
DBD::mysql::db do failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IGNORE TABLE aqbasket
ADD KEY authorisedby (authorisedby)' at line 1 [for Statement "
ALTER IGNORE TABLE aqbasket
ADD KEY authorisedby (authorisedby)
"] at ./installer/data/mysql/updatedatabase.pl line 10563.
0) Find a database on 3.18, save it.
1) Set your code base to 3.22(or master)
2) run updatedatabase.pl
3) See the errors.
4) Apply the patch
5) Reload the 3.18 database
6) succeed with updatedatabase.pl
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit acf425bbbc)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
This reverts commit 7ac1bc3537.
On a kohadevbox:
Error flock (1) on
'/home/vagrant/kohaclone/installer/data/mysql/atomicupdate/bug_xxxx.perl':
No locks available at installer/data/mysql/updatedatabase.pl line 13637.
This is because Path::Tiny->slurp[_utf8] uses flocks which does not work
with NFS
Test plan:
Execute a .perl atomic update file on a kohadevbox
=> Without this patch you should get the flock error
=> With this patch the update should work fine
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c9e95aa22e)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
- Adds "Dear..."
- Fixes capitalization
- Removes 'E-mail' from the description as you could also have
a print template
To test:
- Make sure the SQL has no errors
- Install the letters and proof read them
If you agree - sign off!
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
(This issue has been raised on bug 17216 comments 93-103)
On bug 11944 (3.19.00.006) we updated the default COLLATE for all our tables to
utf8_unicode_ci but not the marc_subfield_structure table.
Indeed we want to keep tagsubfield a utf8_bin (to allow lowercase and uppercase
of the same letter for subfields).
We should have set the default collate to utf8_unicode_ci for further changes.
This patch updates the DB entry 3.19.00.006 to set the default COLLATE to this
table (for people upgrading from prior to 3.19.00.006) and set this
default COLLATE on 16.06.00.033 (for people upgrading from after 3.19.00.006).
The error is:
DBD::mysql::db do failed: Can't create table `koha_kohadev`.`#sql-306_9f9` (errno: 150 "Foreign key constraint is incorrectly formed") [for Statement "
ALTER TABLE marc_subfield_structure
MODIFY COLUMN authorised_value VARCHAR(32) DEFAULT NULL,
ADD CONSTRAINT marc_subfield_structure_ibfk_1 FOREIGN KEY (authorised_value) REFERENCES authorised_value_categories (category_name) ON UPDATE CASCADE ON DELETE SET NULL;
"] at installer/data/mysql/updatedatabase.pl line 13175.
Upgrade to 16.06.00.033 done (Bug 17216 - Add a new table to store authorized value categories)
Test plan:
1/ git checkout v3.18.00
2/ Do an install
3/ git checkout master;
4/ perl installer/data/mysql/updatedatabase.pl
=> Without this patch, you get the error
=> With this patch applied you will not get it and the default COLLATE for
marc_subfield_structure will be correctly set. Make sure tagsubfield is
still utf8_bin
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Will put some notes on the Bugzilla report.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
- wrong assignation
- reference to 'serials' table rather than 'serial'
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This db rev reapplies dbrev 3.15.00.049 in case that one failed due to
existence of issn_idx.
Test plan:
If you do not have index issn_idx, recreate it:
ALTER TABLE biblioitems ADD INDEX issn_idx (issn (255))
Run this db rev. You should see: Removed issn_idx.
Rerun the dbrev. You should see: Everything is fine.
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
When dbrev 3.15.00.049 failed, the still existing issn_idx must have
been the cause.
This patch only adjusts the old db rev.
Still thinking if we also need a new db rev to fix isbn and issn for
those installs where 3.15.00.049 failed too. (In that case the fields
may not have been changed to mediumtext and no indexes were added.)
Test plan:
Ideally, run an upgrade from something older than 3.15.00.49.
But running the added sql command from the command line might convince
you too.
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Adds
- ReturnpathDefault
- ReplytoDefault
to the sysprefs.sql file.
To test:
- Run the web installer in any language
- Check all preferences are installed without error
- Check prefs listed above now exsit in your
systempreferences table
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 14686 added in a dbrev:
(13, 'upload_general_files', 'Upload any file'),
(13, 'upload_manage', 'Manage uploaded files');
But these were not added in userpermissions.sql somehow :)
So, what now?
This patch:
[1] adds them to userpermissions.sql as should have been done,
[2] adds a dbrev to add them for newer installs that did not run the
14686 dbrev.
Test plan:
[1] Run this sql statement:
DELETE FROM permissions WHERE code = 'upload_general_files' OR
code = 'upload_manage'
[2] Run the db rev.
[3] Check if you see Tools/Upload (with sufficient perms).
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Updates sample files with German translations:
- Sample notices for article requests
To test:
- Make sure the German web installer installs all
sample files correctly.
- Bonus: check German translations.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Tested german installer/translation
No errors
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Amended patch:
- Authorised value HSBND_FREQ
is done by bug 17576 (and correctly translated in German)
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
branchcode and is_html columns are present in this file
Test plan:
mysql < installer/data/mysql/fr-CA/obligatoire/sample_notices.sql
should not return any errors
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
It's "Ceština" and should be "Čeština"
Test plan:
1) apply the patch
1.1) update database
2) install Czech translation for Koha
3) note the language name in language switcher (in staff client and
opac) is right: "Čeština"
Signed-off-by: radiuscz <radek.siman@centrum.cz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch adds a new circulation rule (no_auto_renewal_after) to block/allow
auto renewals after a given delay.
For instance, if the issue date is 10 days before today, and
no_auto_renewal_after is set to 10, tomorrow the issue won't be auto
renewed.
Test plan:
0/ Execute the update DB entry
Note: You will have to manually change data in your DB, make sure you
have access to the sql cli.
1/ Define a rule with no_auto_renewal_after (10 for instance) and
norenewalbefore (5 for instance).
(This new rule will behave the same as norenewalbefore: the unit depends
on the lengthunit value).
The automatic renewals will be done from 5 to 10 days ahead.
2/ Modify the issues.issuedate, to simulate a checkout in the past:
UPDATE issues
SET issuedate = "yyyy-mm-dd hh:mm:ss"
WHERE itemnumber = YOUR_ITEMNUMBER;
with issuedate = 2 days before for instance
3/ Execute the automatic renewals cronjob script (misc/cronjobs/automatic_renewals.pl)
Confirm that the issue has not been renewed (too soon)
4/ Repeat step 2 with a due date set as 11 days before
5/ Execute the automatic renewals cronjob script (misc/cronjobs/automatic_renewals.pl)
Confirm that the issue has not been renewed (too late)
6/ Repeat step 2 with a due date set as 7 days before
7/ Execute the automatic renewals cronjob script (misc/cronjobs/automatic_renewals.pl)
Confirm that the issue has been renewed (issues.renewals has been
incremented and date_due has been updated according your circ rules).
Sponsored-by: University of the Arts London
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Katrin Fischer
Jonathan Druart
pongtawat
Lari Taskula
Blou
root
Caitlin Goodger
Mark Tompsett
Frédéric Demians
Kyle M Hall
Marcel de Rooy
Nick Clemens
Alex Sassmannshausen
Josef Moravec
Tomas Cohen Arazi