By defining a default policy in the Perl module, we can easily update
it over time. Using koha-conf.xml will just be for overrides, so people
can manage their own there.
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
report-uri is deprecated, replaced by report-to and Reporting-Endpoints.
To test:
1. prove t/db_dependent/Koha/Middleware/ContentSecurityPolicy.t
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the default configuration to log4perl.conf.
Note we are passing a non-standard interface 'csp' to Koha::Logger. This is to
avoid log entries from propagating up to api-error.log.
To test:
1. prove t/db_dependent/api/v1/public/csp_reports.t
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add /api/v1/public/csp-reports endpoint to receive and log
Content-Security-Policy violation reports from browsers.
When CSP is configured with a report-uri directive pointing to this
endpoint, browsers will POST violation reports whenever a CSP rule
is violated. These reports are logged using Koha's logging system
for administrator review.
Features:
- Public endpoint (no authentication required, as browsers send
these reports automatically)
- Logs violations at WARN level with key details (violated directive,
blocked URI, document URI, source location)
- Logs full report at DEBUG level for detailed analysis
- Accepts both application/csp-report and application/json content types
Configuration example (in koha-conf.xml csp_header_value):
...; report-uri /api/v1/public/csp-reports
Log4perl configuration to capture CSP violations:
log4perl.logger.api.csp = WARN, CSP
log4perl.appender.CSP = Log::Log4perl::Appender::File
log4perl.appender.CSP.filename = /var/log/koha/csp-violations.log
log4perl.appender.CSP.layout = PatternLayout
log4perl.appender.CSP.layout.ConversionPattern = [%d] %m%n
Also updates koha-conf.xml documentation with CSP configuration guidance.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This change adds "style-src-attr 'unsafe-inline';" to the default CSP policy.
The reason is that we have a lot of inline styles throughout Koha,
and it will take a long time to clean those up. Also, the WYSIWYG/TinyMCE
editor injects inline styles, so it might not be possible to forbid
unsafe inline styles in Koha.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Good to have a test :)
Test plan:
Run xt/check_apache_conf.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
Run xt/check_apache_conf.t again. Should pass now.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds missing RewriteRules for ERM, Preservation and Acquisition modules in koha-httpd.conf to restore staff client functionality.
Test plan:
1. Apply patch
2. Restart Apache
3. Access acquisitions and ERM pages
4. Confirm functionality restored
Signed-off-by: Vivek Kumar <vivek.kumar@bestbookbuddies.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Bug 41167: (follow-up) Correct rewrite rules in koha-httpd.conf
This follow-up patch fixes issues from the initial patch for Bug 41167:
- Adds missing asterisks (*) in rewrite rule patterns
- Removes erroneous RewriteRules from the OPAC virtualhost
- Ensures all ERM, Preservation, and Acquisition rewrite rules are correctly placed under the intranet virtualhost
Test plan:
1. Apply patch
2. Restart Apache
3. Access ERM, Preservation, and Acquisitions modules
4. Confirm pages load correctly without 404 or routing errors
Signed-off-by: Vivek Kumar <vivek.kumar@bestbookbuddies.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new configuration entry in `koha-conf.xml` for making
it possible to set a different `batch_size` for `koha-es-indexer`.
The current implementation has a default of *10* that can be overridden
using the CLI parameter `--batch_size N`.
Unfortunately, if you restart the server or the service without passing
the parameter, it will be set back to 10. This patch solves that by
adding a new config entry and making the script use it. The passed CLI
parameter will keep overridding the configured value.
To test:
1. Run:
$ ktd --shell
k$ koha-es-indexer --restart --batch_size 100 kohadev
k$ ps -ef | grep elastic
=> SUCCESS: No errors. Notice '100' is passed to the called perl script
2. Run:
k$ koha-es-indexer --restart kohadev
k$ ps -ef | grep elastic
=> FAIL: Notice '10' is the new value
3. Apply this patch
4. Run:
k$ reset_all
5. Repeat 1 and 2
=> SUCCESS: '100' is kept.
6. Try standalone `--start` too
7. Sign off :-D
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
See comment30 and comment36.
Result of two sed's:
git grep -l test_system | xargs sed -i -e "s/test_system/debug_mode/g"
git grep -l TEST_SYSTEM | xargs sed -i -e "s/TEST_SYSTEM/DEBUG_MODE/g"
The next patch will take care of the debugger options in the plack and
z3950 responder scripts.
Test plan:
Read the patch. (Testing in next patch.)
git grep -i test_system
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
Look at changes in this patch.
Note that we already tested these settings with former patches.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds more module search paths to the zebra-biblios-dom.cfg and zebra-authorities-dom.cfg, so that a larger range of CPU architectures will work with Zebra search unit tests.
This will not change Debian package installs, but it will fix any git-based installs.
TO TEST (you'll need access to an ARM-based CPU, and, preferably, an x86-64 machine):
ON THE ARM-BASED MACHINE:
a) start ktd, then ktd --shell
b) in ktd, run: prove -v t/db_dependent/Search.t
c) notice how the test now fails
APPLY PATCH
d) rerun step b
e) notice how the test now passes
ON THE X86-64-BASED MACHINE:
f) run steps a-e
g) notice how the test passes, both before and after applying the patch
SIGN OFF
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
Just read the patch.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Note that --errlog should refer to all output from the wrapper daemon
process only like respawning processes etc.
And that --output should refer to all output from the daemonized
client process (so in our case the SIP server).
Moving a few sip.log occurrences to sip-output in that regard.
Test plan:
Restart koha-sip.
Login to sip server.
Restart koha-sip.
Verify that you see Net::Server messages and traces from the login in the
-output log. The restarts are not found however in -error. See my comment
on the BZ report.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds the Log4perl configurations required to match the
existing prior use of Log4perl in EDI code.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Some libraries ask for the ability to scan/browse the call numbers index.
Test plan:
==========
1. Have an installation with items with callnumbers (in ktd with standard
test data set: add some callnumbers to the items).
2. Go to Advanced search > More options; choose Call number in the first
drop list and mark Scan indexes. Enter one of the call numbers you
assigned to the items and perform the search. You should get no
results.
3. Apply the pach; restart all; when using ES:
koha-elasticsearch --rebuild -r -d kohadev
4. Repeat p. 2. You should get a list with at least one call number.
The link should lead you to the record(s) with items with the selected
call number.
This should work for ES and for Zebra, but for Zebra you would need to
replace /etc/koha/zebradb/ccl.properties with the repository version; and
you will get one token results on the list, like for other indexes.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This came up with a SIP vendor recently - when reading responses they were only getting part of a patron's information.
It seemed to have been caused by patrons with multiple restrictions and a new line in the response
This patch also does minor refactoring to write_msg to pass the server variable that contains
all the previously passed parameters, and removes the never used "file" parameter.
Test Plan:
1) Add multiple debarments to a patron
2) Run a sip patron information response, note the presence of newlines
3) Apply this patch
4) Add the new option convert_control_characters to your SIP account,
set the value to " -- "
5) Restart all the things!
7) Run a sip patron information response, note the newlines have been replaced!
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Those are leftovers, we should have removed them on bug 18984.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Looks good!
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change adds the ability to enable and configure TCP keepalive
support for the SIP server using SIPconfig.xml.
For the sake of backwards compatibility, it defaults to disabled
and additional parameters default match typical kernel defaults.
Technical detail can be found in the perldoc for C4/SIP/SIPserver.pm
Test plan:
0. Apply the patch
1. koha-sip --restart kohadev
2. apt-get update && apt-get install tcpdump
3. In one window, run "tcpdump -A -n -v -i any 'port 6001'"
4. In another window, run the following:
echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v
5. Note in tcpdump output that after the initial flood of packets,
nothing more is received
6. vi /etc/koha/sites/kohadev/SIPconfig.xml
7. In the "server-params" element, add attributes like the following:
custom_tcp_keepalive='1'
custom_tcp_keepalive_time='10'
custom_tcp_keepalive_intvl='5'
8. koha-sip --restart kohadev
9. In one window, run "tcpdump -A -n -v -i any 'port 6001'"
10. In another window, run the following:
echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v
11. Note in tcpdump output that after the initial flood of packets,
ACK packets are sent out every 10+ seconds for the idle connection
Signed-off-by: Tadeusz „tadzik” Sośnierz <tadeusz@sosnierz.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.
If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.
Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds some commented Elasticsearch security configuration,
which shows how to use username/password with HTTPS.
Test plan:
0. Apply patch
1. cp debian/templates/koha-conf-site.xml.in /etc/koha/koha-conf-site.xml.in
2. koha-create --create-db test
3. vi /etc/koha/sites/test/koha-conf.xml
4. Note that the comments for userinfo and use_https are in the koha-conf.xml
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Works as advertised.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We keep OPEN when people still use log_file or setsid.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Having double dashes inside a commmented block is not valid XML. This
patch restores it, with an added message explaining it
To test:
1. Run:
$ xmllint etc/z3950/config.xml
=> FAIL: You get:
etc/z3950/config.xml:5: parser error : Double hyphen within comment: <!--
<config>
<z3950_responder_options>
<z3950_responder_options>--add-item-status k -t 5</z3950_responder_options
2. Apply this patch
3. Repeat 1
=> SUCCESS: All good!
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds the <config> node that the z3950 responder starter script is looking for in the z3950/config.xml to the example code.
To test:
- verify that the <config> </config> is around the commented z3950_additional_options suggestion in the etc/z3950/config.xml file
- copy the config stanza to the live file: /etc/koha/sites/kohadev/z3950/config.xml
- restart_all
- ps aux | grep z3950
- confirm the script has restarted
- confirm the options: --add-item-status k -t 5 have been passed through
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch enabled the restart by default.
After a poll at hackfest24 we opted to enable this by default and the RM
requested I add the patch to the bug so we don't forget ;)
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds the ability to disable the automated plack restart we
introduce with this patchset via configuration.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Koha serves static .js files as application/javascript (if /etc/mime.types
says to) and serves them compressed, but output_with_http_headers uses the
currently-correct text/javascript mimetype, and Koha doesn't compress that.
Test plan:
1. Set the preference EnableAdvancedCatalogingEditor to Enable.
2. Open the browser Web Developer Tools to the Network tab
3. Load Cataloging - Advanced editor
4. Click on the line for the framework?frameworkcode=&callback=define load
5. Note the content-type text/javascript, no Content-Encoding line, and
the size of 1.9MB
6. Apply the patches from bug 36463 if they haven't been pushed, then this
patch, and reset_all
7. Repeat steps 1-4, and note a Content-Encoding: gzip header and a
Transferred size around 160KB
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch simply adds application/json to the mod_deflate configuration
To test:
1 - Open the netowrk tab in firefox
2 - Load http://localhost:8081/api/v1/libraries
3 - Not the transferred size, and note no 'Content-Encoding: gzip" header
4 - Apply patch, reset_all (or edit /etc/koha/apache-shared.conf)
5 - Reload
6 - Note smaller size, note gzip header
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Find or add a record with title: Chevilly-Larue, L'Haÿ-les-Roses, Fresnes, Rungis [par] Sté éditions et de publicité L.F.B.
2 - Search for 'L'Hay-les-Roses'
3 - No results
4 - Apply patch, copy the file:
sudo cp /kohadevbox/koha/etc/zebradb/etc/word-phrase-utf.chr /etc/koha/zebradb/etc/word-phrase-utf.chr
5 - Restart all, Reindex
restart_all
sudo koha-rebuild-zebra -v -f kohadev
6 - Search again
7 - Success!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/
This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml
If blank we default to 5000.
To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.
Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev
8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The SIP circulation status specifies that a 12 means an item is lost, and 13 means an item is missing. In Koha, missing items are simply a type of lost item so we never send a 13. This is an important distinction for some SIP based inventory tools. It would be good to be able to specify when lost status means "missing" at the SIP login level.
Test Plan:
1) Apply this patch
2) prove t/db_dependent/SIP/Transaction.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>