The SIP fields in the database are too small for the templates of some configurations, and are truncated during upgrade resulting in invalid templates. It's likely we need more than 255 characters for some templates. We should convert all sip template fields to TEXT fields.
Test Plan:
1) For each template field on a sip account attempt to save a template like:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY
2) Note error preventing you from doing so
3) Apply this patch
4) Run updatedatabase.pl
5) Repeat step 1
6) Note you can now do so without error!
Signed-off-by: Charlie Arthur <carthur@slolibrary.org>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 7693360a0f)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
The categorycode column comment in the categories table had a typo:
'idenfity' instead of 'identify'. This fixes it in kohastructure.sql
and adds an atomic update to correct existing installations.
Test plan:
1. Apply this patch
2. Run:
$ updatedatabase
=> SUCCESS: The atomic update is applied
3. Verify:
echo "SHOW FULL COLUMNS FROM categories WHERE Field='categorycode';" | koha-mysql kohadev
=> SUCCESS: The comment reads 'unique primary key used to identify the patron category'
4. Sign off :-D
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 894e144303)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
- Add order_by => ['name'] to record_sources fetch in marc-overlay-rules.pl
so filter dropdown options are consistently sorted
- Add COMMENT to record_sources.is_system column in kohastructure.sql
to match convention of other columns in that table
- Remove commented-out Data::Dumper debug lines from biblios.t
- Remove redundant explicit object deletes inside subtests that are
already wrapped in txn_rollback; the rollback undoes them anyway
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
(cherry picked from commit 6aff5f361a)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
This changes the structure of record source with an is_system column which controls whether this value can be deleted.
Also predefined values are inserted to reflect the default hard coded values from 'Record overlay rules' sources
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
(cherry picked from commit 86443b8437)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
Mysql doesn't allow text type columns to have a default.
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
(cherry picked from commit 7cebccb76f)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
To fix misc4dev/run_tests.pl --run-db-compare-only
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit e7855c0c01)
Signed-off-by: Jacob O'Mara <Jacob.omara@openfifth.co.uk>
Test plan:
1) Run 'updatedatabase'
2) No errors should be thrown
Sponsored-by: Cheshire West and Chester Council
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 366ce3f08e)
Signed-off-by: Chloé Zermatten <chloe.zermatten@openfifth.co.uk>
This should pass:
perl /kohadevbox/misc4dev/run_tests.pl --instance=kohadev --db-password=password --run-db-compare-only --compare-with=HEAD
This is a squash of the following:
1) Fix charset in kohastructure.sql
2) Fix NULL => DEFAULT NULL
3) Reorder tables in kohastructure.sql
4) typos
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This follow-up removes the security vulnerability of storing plaintext
passwords in the sip_accounts table. The SIP2 system now relies entirely
on Koha's secure password authentication.
Changes:
- Remove plaintext password comparison from login_core() and telnet_transport()
- Remove login_password column from sip_accounts table
- Update database schema files and atomic update
- Remove password field from API definitions and Vue form
- Update all tests to remove login_password references
Authentication now works in two stages:
1. Access control: Check if userid exists in sip_accounts (allowlist)
2. Authentication: Validate password against hash in borrowers table
This maintains SIP access control while eliminating plaintext password
storage. Staff users authenticate with their Koha password (securely hashed).
Test plan:
1. Run updated tests to verify they pass
2. Verify SIP2 authentication works with staff user credentials
3. Confirm no plaintext passwords are stored
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Sponsored-by: ByWater Solutions
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch updates the database to include a record_table field in the
additional_field_values table and adds an index on that field.
This should facilitate more performant joins.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new boolean field 'po_is_basketname' to the vendor_edi_accounts
table to allow configuration of how basket names are used in EDIFACT.
Changes:
- Added atomicupdate for existing installations
- Updated kohastructure.sql for new installations
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Sponsored-by: Westminster City Council <https://www.westminster.gov.uk/>
Sponsored-by: Royal Borough of Kensington and Chelsea <https://www.rbkc.gov.uk/>
Sponsored-by: OpenFifth <https://openfifth.co.uk/>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Add UNIQUE KEY entries to table definition
The UNIQUE KEY are required as a hold group 'might have' a target hold, not 'has many' target holds.
This update and the other things this patch updates in atomicupdate + kohastructure should fix the error thrown by command:
perl /kohadevbox/misc4dev/run_tests.pl --instance=kohadev --db-password=password --run-db-compare-only --compare-with=HEAD
This command no longer produces any errors.
Also in this patch:
Renamed table from 'hold_group_target_holds' to 'hold_groups_target_holds' and move it alphabetically to be after 'hold_groups' in kohastructure.sql
This is cleaner, as it matches the 'hold_groups' related table, and allows for alphabetical sorting on kohastructure.sql that makes sense i.e. hold_groups is created first, then hold_group_target_holds is created second with an FK to the former.
All of this passes:
prove ./t/db_dependent/Koha/Hold.t
prove ./t/db_dependent/Reserves/HoldGroup.t
prove ./t/db_dependent/Circulation*
prove ./t/db_dependent/Hold*
prove ./t/db_dependent/Reserves*
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
If a hold that is a hold group target, i.e. it's 'waiting' or 'in transit', is reverted it needs to remove itself as its own hold group target
Tests added here as well:
prove t/db_dependent/Koha/Hold.t
Signed-off-by: Anneli Österman <anneli.osterman@koha-suomi.fi>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Hold group target hold:
The hold that is set as the hold group's target. If a hold group has a
hold target, subsequent check-ins that would 'trap' items to the
remaining holds in the group are instead skipped.
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Anneli <anneli.osterman@koha-suomi.fi>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The default sort order for Koha's patron search includes many columns, some of which are not indexed:
[% CASE 'name-address' %] {
"data":"me.surname:me.preferred_name:me.firstname:me.middle_name:me.othernames:me.street_number:me.address:me.address2:me.city:me.state:me.postal_code:me.country",
Depending on the number of patrons in the database, and the specific configuration of the server, this can end up generating a query that creates a temp table in the hundreds of gigabytes of data or more. Not only does this risk crashing a database server with insufficient disk space, it's also incredibly slow.
We've found a compound index prevents the issue and also makes the search much faster. InndoDB indexes have a fixed maximum length which is too small ( 3072 bytes ) to encompass the full data for all the fields we need to sort by.
The following works well as a comprise to keep the index length within the maximum:
CREATE INDEX idx_borrowers_sort_order ON borrowers (
surname(100), -- 400
preferred_name(80), -- 320
firstname(80), -- 320
middle_name(50), -- 200
othernames(50), -- 200
streetnumber(20), -- 80
address(100), -- 400
address2(75), -- 300
city(75), -- 300
state(40), -- 160
zipcode(20), -- 80
country(40) -- 160
);
Test Plan:
1) Apply this patch
2) Run updatedatabase.pl
3) Perform a patron search
4) No change in behvior should be noted!
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds database migration to move EDI transport configurations
from vendor_edi_accounts to the new file_transports system introduced
in Bug 39190.
Changes:
- Add file_transport_id column to vendor_edi_accounts with foreign key
- Add 'local' transport type to file_transports enum
- Remove old transport fields from vendor_edi_accounts in kohastructure
- Migration script converts existing EDI configurations to file transports
- Handles FILE -> local transport type conversion
Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
As suggested by David, this one adds an index on the new column
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Update status field to store longtext so we can sore more details
information about connection statuses from test connection
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch updates the database schema to add booking_id foreign key
columns to both issues and old_issues tables, enabling the linkage
between checkouts and the bookings that generated them.
Schema changes:
- Add booking_id column to issues and old_issues tables
- Add foreign key constraints referencing bookings table
- Preserve referential integrity with appropriate CASCADE actions
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch explicitly defines indexes for foreign keys.
This isn't really needed for the atomic update, but we've included
it for consistency.
It is needed in kohastructure.sql, because MySQL/MariaDB silently/implicitly
adds these indexes for foreign keys, which means that the database
audit tool will show a discrepancy between kohastructure.sql and
the installed DB schema.
Also fix a discrepancy between atomic update and kohastructure.sql
Test plan:
0. Apply the patch
1. Use both methods to update the DB schema
2. Go to http://localhost:8081/cgi-bin/koha/about.pl?tab=database
3. Note that about.pl isn't telling you to drop the indexes
for reserves_ibfk_hg and old_reserves_ibfk_hg now
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Kyle M Hall
Tomás Cohen Arazi
Martin Renvoize
Jan Kissig
Michael Hafen
Jonathan Druart
Matt Blenkinsop
Lucas Gass
Pedro Amorim
Pedro Amorim
Martin Renvoize
Jake Deery
Paul Derscheid
David Cook
Aleisha Amohia