That breaks randomly tests on Jenkins (Koha/Patrons.t)
It certainly affects data in other languages but I did not find a quick
and easy way to modify this file (replace all guarantorid values with
NULL)
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Easy fix
Signed-off-by: Mason James <mtj@kohaaloha.com>
fr-CA is not up-to-date:
- RESERVESLIP have been renamed with HOLD_SLIP
- PASSWORD_RESET and MEMBERSHIP_EXPIRY are not defined
Signed-off-by: Mason James <mtj@kohaaloha.com>
This string patch fixes typos in kohastructure.sql as described in initial
comment.
To test:
- Carefully verify that string changes make sense
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 279126d710)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 7ff2fd6e9a)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended to include items and deleteditems.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 69a24812ed)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 3e1f21ae0d)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
This patch is the Koha part of the Hea v2 project.
You can find the (testing) code for the server at
hea-ws - https://github.com/joubu/hea-ws/commits/v2
hea-app - https://github.com/joubu/hea-app/commits/v2
They contain the different pull requests made over the last 6 months.
More information on Hea at https://wiki.koha-community.org/wiki/KohaUsageStat_RFC
The goal of this commit message is to provide an overview of what could
be a new version of Hea.
Prior to these changes, the Hea database was filled with 1 line per Koha
installation. System preferences were filled by the libraries and a
cronjob (share_usage_with_koha_community.pl) collected these values to send
them to a webservice (hea-ws/upload.pl).
With the need to collect more data we would want to collect data at the library
level (branch) and not at the installation level.
For instance the geolocation, the url or the country can be different from one
library to another, even if managed from the same Koha installation.
The Hea DB has been upgraded to reflect that change (see hea-app/sql/schema.sql).
The hidden goal of this patch is to make Hea sexier and explain
better to libraries how it can be useful to share their information
with the Koha community. I guess the main problem is the lack of
communication and explanations about what we are doing we these data.
To fill this gap I'd like to (TODO)
1. Communicate on the ML about this new version of Hea (once it got
pushed and backported)
2. Link the Privacy_Policy.md from the Hea interface
3. Get help from a native English speaker to add
popup/help/info/whatever on "Home › Administration › Usage statistics",
to clearly explain what happens (and what will not happen!) when an option or
another is set.
You can find screenshot of this whole enhancement on bug 18066, comment 2.
What this patch does:
- Create a new branches.geolocation DB field
- Add 3 new sysprefs:
* UsageStatsGeolocation
* UsageStatsLibrariesInfo
* UsageStatsPublicID
- Integrate the Leaflet JS library to get a fancy map to pick
geolocations
How does it works:
On the new administration page where statistics to share are configured,
there are several new things. It is now possible to share information either
per Koha installation or libraries. If UsageStatsLibrariesInfo is set,
the info at library level (url, name, country, geolocation) will be
sent to the Hea webservice. If it is not set, you can decide to fill
UsageStatsLibraryUrl, UsageStatsLibraryName, UsageStatsCountry,
UsageStatsGeolocation to share these information. Note that even if the
data are retrieved at installation level, it's better to fill the prefs
as well: On the Hea website the different libraries defined for a given
Koha installation could be displayed on the same page.
This page is a public page which will be attributed to every
installation (with the pref UsageStatsPublicID). On this page all the
info available publicly will be displayed.
TODO later:
- Add a button on the administration page to delete the info shared
publicly. It will be easy to show that the info are no longer displayed
on the public page.
- Add an icon per Koha installation to get a better "public page"
- Any suggestions?
Test plan:
We will need to test hea-ws, hea-app and the Koha-side code to test the
whole enhancement.
1/ To start, clone the hea-ws and hea-app project and checkout the
'master' branch (*not* 'v2')
2/ Create the hea database and user
CREATE DATABASE hea
CREATE USER 'hea'@'localhost' IDENTIFIED BY 'hea';
GRANT ALL PRIVILEGES ON hea.* TO 'hea'@'localhost';
FLUSH PRIVILEGES;
3/ Fill the DB with some data
mysql hea < hea-app/sql/schema.sql
mysql hea < hea-app/sql/sql/mock-data.sql
4/ Checkout the 'v2' branch for both hea-ws and hea-app
5/ Execute the upgrade DB script
% cd hea-app
% perl -p -i -e 's/REPLACE_ME/hea/' sql/upgrade.pl # Fill the DB info
% perl sql/upgrade.pl
Now the DB is using the v2 structure. That means we have 1 installation
row per library previously defined. 1 library row has also been created.
5/ Configure hea-ws
% echo '192.168.50.1 hea.koha-community.org' >> /etc/hosts
<VirtualHost *:80>
DocumentRoot "/path/to/hea-ws"
ServerName "hea.koha-community.org"
<Directory "/">
Options +ExecCGI
Require all granted
AddHandler cgi-script .pl
</Directory>
</VirtualHost>
And enable it with a2ensite, then restart apache.
The copy the database.yml.sample to database.yml and edit it to fill the
DB info.
6/ Launch the hea-app
% cd hea-app
% edit README.md # to install the missing modules
% cp environments/config.yml environments/development.yml
% edit environments/development.yml # to fill the DB info
% perl bin/app.pl
Then hit localhost:3000
You should see a local version of Hea with sample data
7/ Back to Koha side
A. We will test that the webservice still works with previous version of Koha (without v2)
a. Do not configure Hea
% perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
Then hit localhost:3000
=> Nothing added
b. Configure Hea on admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> New library added
c. Modify the Hea configuration
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Info are modified
B. Not we will test that it works with the new version (much more fun ;))
% git checkout hea-v2 # koha
a. Configure Hea using /admin/usage_statistics.pl
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
=> Check the result on localhost:3000
b. Share libraries's info
perl misc/cronjobs/share_usage_with_koha_community.pl -f -v
c. Continue to play a bit and share the info.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Mason James <mtj@kohaaloha.com>
The loading of file admin/searchengine/elasticsearch/mappings.yaml
specifies 'type' as empty, which fails with Mysql 5.7+ which is more
stringent.
Also, forcing an empty value into a boolean also fails.
Both issues arise when updatedatabase.pl hit 3.23.00.050.
NOTE: both issues could also be resolved by actually setting values in
the load file. This doesn't make this solution incorrect, though.
To Test/reproduce:
-1) Happens with Mysql 5.7.4. Maybe earlier, but certainly at
that point. Use a Xenial kohadevbox to more easily test.
0) Find a database on 3.22 or earlier, save it.
a) place
sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
into the [mysqld] section of /etc/mysql/mysql.conf.d/mysqld.cnf
b) restart you mysql server
c) drop the db and recreate it
d) checkout the 3.22.x branch
e) do a web install
f) remove the added sql_mode
g) restart the mysql server
1) Set your code base to master
2) run updatedatabase.pl
3) See the errors on 3.23.00.050
4) Apply the patch
5) Reload the 3.22 db.
a) repeat steps 0(a)-0(g)
b) don't forget the caching issues
6) succeed with updatedatabase.pl
7) drop the db and recreate it
8) run the web installer
9) notice no issues either.
10) run koha qa test tools
NOTE: This bug only solved the upgrade portion.
I added the kohastructure.sql change as well.
I confirmed that all the code changes were
triggered with this test plan.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mehdi Hamidi <mehdi.hamidi@inlibro.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 77337a2ab5)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 10459 has been backported and the DB entry (add borrowers.updated_on and
deletedborrower.updated_on) is now played in
- 16.06.00.027
- 16.05.00.002
- 3.22.08.001
This will raise a MySQL warning if the column already exists.
DBD::mysql::db do failed: Duplicate column name 'updated_on'
Since bug 17234 we have now a subroutine (C4::Installer::column_exists) to test
if a column exists.
When a DB entry modifying the DB structure is backported, it HAS TO check if the
column, constraint or table exists before executing the query.
Test plan:
git checkout 3.22.x (16.05.x or 16.11.x)
install Koha
git checkout master
execute the installer
=> Without this patch you will get a warning when adding borrowers.updated_on)
=> With this patch, you should not get it
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 2b303129b8)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
In the previous patch we use the constraint_exists subroutine to verify
if an index or a foreign key exists.
But the `SHOW INDEX` query does not return foreign keys (as its name
suggests!).
We need another subroutine foreign_key_exists to check the FK existence.
I have found that because t/db_dependent/TestBuilder.t fails on
oai_sets_biblios, because oai_sets_biblios_ibfk_1 has not been removed.
Test plan:
0/ Do not apply this patch
1/ Use a 3.20 DB
2/ update the DB
3/ SHOW CREATE TABLE oai_sets_biblios
will display oai_sets_biblios_ibfk_1
Apply the patch and repeat 1, 2, 3
=> Will not display oai_sets_biblios_ibfk_1
It has been removed as expected.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 05fdd855c8)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit 0b15c5e2cc)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
These 2 subroutines will help us deal with the absense of ALTER IGNORE
TABLE
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit c08922180e)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Conflicts:
installer/data/mysql/updatedatabase.pl
The doc says: "As of MySQL 5.7.4, the IGNORE clause for ALTER TABLE is removed and its use produces an error."
This fix replaces ALTER IGNORE with ALTER in updatedatabase.pl
To TEST, try an upgrade from 3.18 to 3.22 after installing the latest mysql (at least 5.7.4, by the doc).
Some will fail with error
DBD::mysql::db do failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IGNORE TABLE aqbasket
ADD KEY authorisedby (authorisedby)' at line 1 [for Statement "
ALTER IGNORE TABLE aqbasket
ADD KEY authorisedby (authorisedby)
"] at ./installer/data/mysql/updatedatabase.pl line 10563.
0) Find a database on 3.18, save it.
1) Set your code base to 3.22(or master)
2) run updatedatabase.pl
3) See the errors.
4) Apply the patch
5) Reload the 3.18 database
6) succeed with updatedatabase.pl
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit acf425bbbc)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 14686 added in a dbrev:
(13, 'upload_general_files', 'Upload any file'),
(13, 'upload_manage', 'Manage uploaded files');
But these were not added in userpermissions.sql somehow :)
So, what now?
This patch:
[1] adds them to userpermissions.sql as should have been done,
[2] adds a dbrev to add them for newer installs that did not run the
14686 dbrev.
Test plan:
[1] Run this sql statement:
DELETE FROM permissions WHERE code = 'upload_general_files' OR
code = 'upload_manage'
[2] Run the db rev.
[3] Check if you see Tools/Upload (with sufficient perms).
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Mason James <mtj@kohaaloha.com>
[1] Renames the dbrev version to XXX; adding a my for $sth, although
not strictly necessary it feels better.
[2] Circulation.t stumbles over:
Undefined subroutine &C4::Circulation::GetItem called at ... line 1283.
Somehow this is related to use_ok instead of a regular use.
Worked around this by require_ok and adding a regular use.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mason James <mtj@kohaaloha.com>
If the 'statistics' table contains circulation information about items
that are no longer on the DB it will raise a warning. This patch
makes the updatedatabase.pl script use the deleteditems and deletedbiblioitems
tables to get information for those items on the statistics table.
To reproduce:
- Have your sample DB contain some circulation data on the statistics table
- Make sure some of them have NULL itemtype:
> UPDATE statistics SET itemtype = NULL WHERE type='return';
- Check the upgrade query catches them:
> SELECT s.itemnumber, i.itype, b.itemtype FROM ( SELECT DISTINCT itemnumber FROM statistics WHERE ( type = "return" OR type = "localuse" ) AND itemtype IS NULL ) s LEFT JOIN ( SELECT itemnumber,biblionumber, itype FROM items UNION SELECT itemnumber,biblionumber, itype FROM deleteditems ) i ON (s.itemnumber=i.itemnumber) LEFT JOIN ( SELECT biblionumber, itemtype FROM biblioitems UNION SELECT biblionumber, itemtype FROM deletedbiblioitems ) b ON (i.biblionumber=b.biblionumber);
+------------+-------+----------+
| itemnumber | itype | itemtype |
+------------+-------+----------+
| 732 | BK | BK |
| 731 | BK | BK |
+------------+-------+----------+
2 rows in set (0.00 sec)
- Delete the items, and some biblio too.
- Re-run the query
=> SUCCESS: Same results
- Go reset to NULL the itemtypes
> UPDATE statistics SET itemtype = NULL WHERE type='return';
- Run the updatedatabase.pl script:
$ sudo koha-shell koahdev ; cd kohaclone
$ perl installer/data/mysql/updatedatabase.pl
=> SUCCESS: No warnings
Note: It is possible that on production sites, if the sysadmin is cleaning the
deleted{items|biblioitems|biblio} tables, there will be warnings. This is expected
as they need to know some data lacks information.
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mason James <mtj@kohaaloha.com>
This patch introduces an updatedatabase.pl entry that takes care of
updating existing statistics rows.
It does so by looping on the statistics rows, collecting itemnumber occurences
that are have NULL itemtypes.
It then chooses the right itemtype following what is proposed on bug 14651, and
then updates the rows in statistics using the calculated itemtype.
Regards
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Mason James <mtj@kohaaloha.com>
It's "Ceština" and should be "Čeština"
Test plan:
1) apply the patch
1.1) update database
2) install Czech translation for Koha
3) note the language name in language switcher (in staff client and
opac) is right: "Čeština"
Signed-off-by: radiuscz <radek.siman@centrum.cz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
(cherry picked from commit de2ba75c25)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Some installer files still use WTHDRAWN instead of WITHDRAWN as an AV
category.
Test plan:
git grep WTHDRAWN installer
should not return any results
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit ce076e0e75)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Some installer files use COU and LAN instead of COUNTRY and LANG.
To uniformise these AVs, it would be good to use the same code.
Note that I do not think we should update existing installations
Carefully compared code.
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit 75a900d553)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
What we currently have:
Koha/ElasticSearch.pm
Koha/ElasticSearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm
What we want:
Koha/SearchEngine/Elasticsearch.pm
Koha/SearchEngine/Elasticsearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm
Test plan:
% git grep -i Koha::ElasticSearch
% git grep ElasticSearch|grep -v Catmandu::Store::ElasticSearch
should not return any result
Do a full reindex and search for records
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit df3df9cf71)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
The yaml file is not used to populate ES mapping tables (search_field,
search_marc_map and search_marc_to_field) when doing a fresh install.
We need to insert them, otherwise ES will be unusable.
Test plan:
Create a new install and confirm that the ES tables (search_field,
search_marc_map and search_marc_to_field) are correctly populated.
Bonus points: Use an older DB (prior to 3.23.00.050), execute the
updatedatabase.pl script and confirm that the ES table are correctly
populated
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit a36b643019)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
remove a 'my'
next if not defined subscriptions.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit 6ff836ab31)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
A recent change in the 'subscription' table structure highlighted a
problem in a DBRev upgrade (3.23.00.006). As it adds a new column, when
upgrading from (say) 3.20.00.000 the code/schema (correctly) expects the
subscription.itemtype column to exist. But it is not created until DBRev
16.06.00.025.
To reproduce:
- Have a clean 3.20.00 DB loaded into kohadevbox
- Checkout current master
- Run:
$ perl installer/data/mysql/updatedatabase.pl
=> FAIL: The upgrade procedure fails due to missing colum.
The solution: rewrite the updatedatabase.pl entry using plain DBI
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit f11f07a703)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
In a database update we alter the letter table, making a column
(branchcode) not be able to contain NULL values.
However if it already does, this SQL fails. So before changing the
column we change the NULL values
[Tue Sep 20 19:08:10 2016] updatedatabase.pl: DBD::mysql::db do failed:
Invalid use of NULL value [for Statement "
[Tue Sep 20 19:08:10 2016] updatedatabase.pl: ALTER TABLE letter MODIFY
COLUMN branchcode varchar(10) NOT NULL DEFAULT ''
[Tue Sep 20 19:08:10 2016] updatedatabase.pl: "] at
/home/mtompset/kohaclone/installer/data/mysql/updatedatabase.pl line
12638.
Reproduce:
on an ubuntu 16.04 LTS machine with MySQL 5.7 install git koha
development environment.
...
git checkout -b my_3.18.x origin/3.18.x
drop database koha_library;
create database koha_library;
quit;
web install all sample data
(ignore some issues with sample data, and one with mandatory)
git checkout -b bug_17234 origin/master
git bz apply 17292
git bz apply 17320
git bz apply 17260
git bz apply 17234
web install
Two warning messages are expected output from upgrade process, but
the above message is not.
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Did not follow the test plan, but the changes make sense
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
(cherry picked from commit 0599ab0076)
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Tomas Cohen Arazi
Jonathan Druart
Mason James
Marc Véron
pongtawat
Lari Taskula
Katrin Fischer
Blou
Marcel de Rooy
Tomas Cohen Arazi
Kyle M Hall
Frédéric Demians
Josef Moravec
Mark Tompsett