This adds a shell script which is able to turn html files
into pdf and print them on a printer
It takes a directory where pdf files is laid, a css filename,
a host for printer and a Printername
printoverdues : generates PDF files from html files in directorys and prints them
usage :
printoverdues.sh directory [css [printer_host [printername]]]
- directory directory to use to apply html2pdf transform
- css css file to apply to html
- printer_host Network Name or IP of the printer (port possibly included)
- printer_name printername
Note that css printerhost and printername are optional parameters
Note that this script uses xhtml2pdf command
xhtml2pdf command comes with pisa (a python library)
To install you need setuptools library for python
then type easy_install pisa
A new Koha Offline Circulation client has been written by Kyle M Hall in C++/Qt4.
Unfortunately, it requires an SQLite3 databse, where the PHP/Gtk client needs an SQLite2 database.
This update adds the switches --sqlite2 and --sqlite3 to the script to output either format.
Those bugs must have been introduced by merge?
- Overdue to all libraries with overdue rules doesn't work
- Overdue to a specific library doesn't work also
Enabled the -n (nomail) option, which was previously doing nothing. In
addition, I have added an -itemscontent option to allow for <<items.content>>
to be used in the notices for DUE and PREDUE.
This add two new options to overdue_notices.pl to select only overdues for few categorycodes, or to exclude few categorycodes.
Conflicts solved misc/cronjobs/overdue_notices.pl
xsl strings are now taken into account.
This is a first draft.
Some parameters of the functions will be taken into account
and <xsl> tags are not translated
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
This update the way Member is added and editing so that import and Edition
could be best automatized
GetMember evolves and allow ppl to serach on a hash of data
Adding SQLHelper A new package to deal with INSERT UPDATE and SELECT
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Adding some new options to bulkmarcimport :
-k idtagsubfield in order to store the id of the file record into another field
-match tagsubfield,index
-a to import authorities
-l logfilename to store logs
Bug Fixing : C4/Charset.pm
Charset was incorrect for UNIMARC Authorities
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
The script already has a param -b for batch mode, which should silence informational
messages, but it missed a couple. This patch fixes that.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
This patch, permit to take care of similar entries, if there is some diacritics or not.
(cherry picked from commit 776c177e3debedaf08fec65fbf8111675ccc93e7)
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Allows temporary locations corresponding to 'in processing' and 'shelving'
so that newly-created items, and newly-returned items do not show
immediately as a available. Three new system preferences govern the usage
of these features.
NewItemsDefaultLocation. If system pref NewItemsDefaultLocation is set to a location code,
all newly catalogued items will be set to the location set in this preference.
Location code must be a valid LOC authorized value type.
InProcessingToShelvingCart. if the system pref InProcessingToShelvingCart is turned on,
any items run through returns.pl with a location code for 'PROC', will be modified to
have a new location code of 'CART'.
ReturnToShelvingCart. If the syspref ReturnToShelvingCart is turned on,
all items returned other than confirmed holds will have a new location code of 'CART'.
Any item issued is automatically taken of the shelving cart.
Adds a cron script shelf_to_cart.pl which should be run hourly.
Updates all items with a location of CART to the item's permanent location.
The original location code is stored in the new items column 'permanent_location'.
Original Author: PTFS Contractor <dbavousett@ptfs.com>
This work co-sponsored by
Middletown Township Public Library, Middletown, NJ USA and
East Brunswick Public Library, East Brunswick, NJ USA
Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Numbers in perl with leading zeros are interpreted in octal
Ensure that comparisons are done using string operators
or where appropriate use the MARC::Field method
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
mergeauthority and ModAuthority were working on two separate directories.
So that no authority would ever be merged via cronjob or commandline script
when MergeAuthoritiesOnUpdate is disable
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
Fixed the holds queue job so that it correctly
ignores hold requests that are not yet scheduled
to be filled.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
This little script establishes a framework for database cleanup on some regular
schedule. Initial implementation provides for brute truncation of the sessions
table, and selective-by-age cleanup of the zebraqueue.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
The standard license statement in the header is fine; please
don't confuse things by doing anything different.
Signed-off-by: Galen Charlton <gmcharlt@gmail.com>
It appears, in Ryan's patch, that he wants to throw a warning to the log if
the directory specified in --out is not present. (Further messages will
be given when the open-or-die occurs a few lines later.) However, it was
throwing the warning if --out was not specified at all, which is
undesirable. This patch modifies that bit to check for the presence of
whatever directory is going to be used, either --out, ENV{TMPDIR}, or /tmp.
As before, if the write to the directory fails for any reason--including
its' non-existence--that is handled later, but this message will help
inform the troubleshooter.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
With this patch, rebuild_zebra can re-index a whole Koha DB
quickly:
rebuild_zebra -r -b -nosanitize
Biblio (authority) records are dump directly in a file
from marcxml field without beeing transformed into
MARC::Record object and corrected.
DOCUMENTATION:
rebuild_zebra.pl new paramater:
-nosanitize export biblio/authority records directly from DB marcxml
field without sanitizing records. It speed up
dump process but could fail if DB contains badly
encoded records. Works now only with -x and -b
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch adds the MARC21 subdivsion record tags (18x) to the
block which recognizes and assigns authtypecodes to imported
authority records.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Fixes a hang of the staging import tool when it
attempts to process a MARC21 record that claims
that it's UTF-8 when it is not. The staging import
will now attempt to fix the character encoding of such
records.
Also added a FIXME to bulkmarcimport.pl, which because
of its use of MARC::Batch will skip over such records -
better than the original hang of the staging import, but
worse than the staging import's new ability to fix such
records.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The Offline Circ tool by Kyle Hall uses PHP's SQLite, which is SQLite
v2. Gnope, which Kyle links, ships with libSQLite v2.
Let's not count on libsqlite3 not being installed for perl. If it is
installed DBD::SQLite will use it, where DBD::SQLite2 will not.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
When marcxml cannot be parsed into a MARC::Record object, the biblio is
undisplayable and it obviously breaks many features in Koha. This script
can test to parse every marcxml, and alert on failures. Optionally, the
marcxml can be replaced from the marc field.
See extensive perldoc for details.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The problem is that we do not ensure that the issues table has valid
borrowernumber in each line. This is exacerbated by Getoverdues()
returning data sorted BY borrowernumber. So one NULL borrowernumber
in issues prevented ALL fines from being assessed. The actual error
from fines.pl cron log is:
No branchcode argument to new. Should be C4::Calendar->new(branchcode => $branchcode)
at /home/user/kohaclone/misc/cronjobs/fines.pl line 98
This patch deals only with getting fines to avoid crashing. It does
not fix the underlying data integrity problem.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This has bearing on bugs 2905, 2665, 2514 and other "wide character" crashes
related to diacritics and Unicode. This should help open the door for reliable
input of diacriticals via acquisitions.
MARC21_utf8_flag_fix.pl diagnoses and fixes existing problems with MARC data
affected by the bug.
Adding SetMarcUnicodeFlag to TransformKohaToMarc prevents the bug from corrupting
further data.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Acquisitions process seems to be adding records with incorrect
representation of the MARC encoding in leader/09. It should be
'a' meaning UTF-8, for all Koha's internalized records, but in
many cases it appears blank (for MARC-8). This script diagnoses
and repairs the value in the leader, depending on runtime options.
The symptom of this problem is that high-value UNICODE characters
in the record will cause Koha to crash whenever it tries to parse
the MARCXML, giving a "Wide character" fatal. While we work on
fixing the input, this script will fix the existing data.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
If series title was previously taken as collectiontitle
then you should launch
misc/maintenance/UNIMARC_fix_collectiontitle.pl
misc/batchRebuildBiblioTables.pl
[RM note: applies only to UNIMARC users - MARC21
users should not run the batch jobs.]
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
execute_query now refactored, returns reliable results, does
zero presentation-layer crap. Arguments reduced, client scripts
adapted to new API and performance improved. Text::CSV now used
to generate CSV output, ensuring portability, encoding and accuracy.
Replaced tools/runreport.pl with misc/cronjobs/runreport.pl:
~ security fixed
~ documentation improved
~ TODO: finish sendmail option.
Bug 3077 also fixed.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The old location of runreport.pl was under tools, leaving it exposed
to web requests. This is a security flaw since it does NOT check any
Auth and allows the user to request any Saved Report be run. This is
not a problem under misc/crontab/ and it suggests the more appropriate use.
Guided.pm is not fixed here (see bug 3066), but it is prepared to be fixed
and made compatible with runreport as detailed in the perldoc.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Adds three new switches:
-idmap <filename> - optional output file of
map of source record ID numbers
to Koha biblionumber
-x - if idmap is supplied, MARC tag
to get source record ID from
-y - if idmap is supplied, MARC subfield
to get source record ID from
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Pulled the GetMarcFromKohaField outside the loop, since we only need to
check it once. Same for $item_sth. Added safety checks for success of
delete_field and GetMarcItem, with warnings on failure.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Removed cronjob, which was written to work around
a bug in 2.2 that no longer applies and is specific
to a single library in any event.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch just fix the script which export to csv the overdue, and field the missing fields
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This does not fix all problems recorded in 2883 (see all the FIXME's), but
it does improve the script's basic feedback to an intelligible level.
It also adjusts the documentation and examples to correct bogus usage
instructions.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
* removed ersataz YAML::XS dependency
* use 'return' instead of 'return undef'
* minor language changes
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Improve URLs checker script in the way (half way) pointed out by Galen:
- A C4::URL::Checker class handle URL checking. This class is not yet
in a separate file in C4 directory. This class would be easily
extended to accomodate authorities URLs checking.
- Script output can now be formatted in CSV or HTML. HTML version
link directly to MARC biblio record editor.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Item type was not retrieved in a query, leading to a case
where an item could be selected by build_holds_queue.pl
to fill a hold request even where forbidden by the
library and item type-level policy.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This includes part of a patch from Henri-Damien Laurent
that could not be applied because Chris and Joe patches
happened to win the race.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
* use item branch instead of patron's branch to
look up the applicable hold policies - this makes
requesting in the OPAC consistent with the intranet.
* when generating pick list using build_holds_queue.pl, only match items
to patrons if request is allowed.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Add "use warnings", remove unused variables and unnecessary finish/disconnect
at the end. This script could be improved to run only on tables that need to
be altered instead of touching all of them. It should also probably contain
warnings to the effect that it does not rescue your DATA that was forced into
whatever encoding the table used previously.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This reverts commit def09f5a21.
As I emailed to the patches list Oct 06, 2008:
I suggest we need to revert Josh' commit def09f5a21.
The effect on the crontab example is to invalidate the lines being executed. The lines were apparently copied in from a cron source, not crontab, despite the header describing it NOT being an example for cron. It also runs longoverdue twice, instead of fines.pl.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The problem was that the TmplTokenizer create global symbols to store file
handlers so the objects aren't destroyed by the garbage collector.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
As rss.pl is not a CGI script, moved it to join the
other cronjobs. Full documentation of the script
is in misc/cronjobs/rss/rss.pl, but to summarize:
[1] rss.pl is run on the command line to produce
an RSS XML document. The output should be
placed in a directory accessible to the OPAC
(or staff) web interface so that users can download
the RSS feed. An example of usage:
misc/cronjobs/rss.pl lastAcquired.conf
Normally rss.pl should be run periodically (e.g., daily)
to keep the feed up-to-date.
[2] The configuration file (e.g., lastAcquired.conf) lists
* name of the template file to use
* path of output file
* SQL query
rss.pl runs the SQL query, then feeds the output of the
query through the template to produce the output file.
[3] The template file (e.g., lastAcquired.tmpl) uses
HTML::Template syntax like any of the HTML
templates for the web interface.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Add the phrase 'if ( $verbose_logging )' to the two print statements
concerning the skipping of biblio or authority records.
I recently had to split biblio and authority index updating in my cron
script ( had some really big records so had to add the -x switch which
should only be used on biblios accourding to the help ). So I noticed
that rebuild_zebra.pl printed messages that it was skipping biblios or
authorities.
This patch is to conditionalize those prints based on the verbose
logging switch.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Reduce the size of the SQLite database produced by
create_koc_db.pl by emiting only the columns
actually used by Kyle Hall's offline circ client.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This commit adds two options to the overdue_notices.pl script.
The notices are now generated in one of two modes: default or triggered.
In default mode, every time the script runs, a notice is generated if an item
is overdue within a notice date range. This mode is suitable for a weekly cron
(or manual run), but the notice triggers should be syncronized with the cron script such that
the script is run once per notice date range. Otherwise, duplicate messages will
be generated ( or no notice generated if a notice date range passes without the script being run).
Triggered mode is suitable for a nightly cron job: a notice is generated only on those
days that an overdue item enters a notice date range. This mode is susceptible to message
loss when the cron script fails to run, or when notice triggers are edited.
A second flag is added: --list-all, which causes items.content to list all currently overdue items.
Without this flag, only items that are overdue within the given notice's date range are listed on
that notice.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Restoring the functionality of these sysprefs as part of the
rework of build_holds_queue.pl:
* StaticHoldsQueueWeight is an ordered list of library codes.
The order of the libraries in this list determines the order
of libraries in which available items will be chosen to be
targeted to holds. Note that if this list does not include
a library, items from that library will never be targeted to
holds and staff will never be asked to retrieve items for
title-level holds from those libraries.
* RandomizeHoldsQueueWeight is a Boolean parameter, indicating
whether to randomize the libraries in the StaticHoldsQueueWeight
list.
If neither RandomizeHoldsQueueWeight nor StaticHoldsQueueWeight is
set, then the libraries will be used to target items to holds in
alphabetical order by library code.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This reduces the output of the script and zebraidx, and creates a -v
command line switch which will increase the logging to their former
states.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Most Perl scripts (as opposed to modules) do
not need to require Exporter.
No user-visible or documentation changes.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
The offline circulation tool can use all of the patron information, not just their ID
numbers. This patch populates the offline circulation database with patron information.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
CalcFine returned values that mismatched expectations in fines.pl.
fines.pl refactored: added debugging, prevent needless recreation of
Calendar objects by storing them in hash by branch.
Still outstanding problems with fines, including the output of a field
that has no other references in Koha (so is always undef) and the
incorrect description of FinesMode.
Calendar exported "new" erroneously. I also cleaned up the queries to
avoid needlessly compiling additional statement handles.
Please test and consider application to 3.0 maintenance.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This is the script that generates the patron and holdings database for the offline circ module. I
touched up a few parts of it:
* fixed the database syntax to squash a bug or two
* extracted some methods for readability and reuse
* added some documentation
* added a --file and --force command line argument
* made it die if DBD::SQLite is not present. That module is required for this feature.
Here's an example crontab line that will generate the borrowers.db file each midnight.
0 0 * * * create_koc_db.pl --file /tmp/borrowers.db
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This script generates a sqlite database of patron and holdings data for the offline circulation tool.
It's a cronjob, so it should go into misc/cronjobs
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
SQL fix to properly ensure that if an item is on
the hold shelf, it will not be used for request
targeting, which applies only to items that are
not already on the hold shelf.
Prior to this fix, when checking out an item on the
hold shelf that fills patron A's hold request, it
was possible for that request to not be marked as
filled if another patron had an item-level request
on the item; the second patron's request was
incorrectly targeted by that item.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This batch job now automatically populates the
holds request targeting table hold_fill_targets.
This patch is essentially a revamp of the job,
and includes fixes for the following bugs:
2281 (holds queue report including unavailable items)
2331 (holds queue report not working with item-level
holds)
2332 (holds queue script should attempt to fill
many requests as possible).
Several functions in this batch job are candidates
for being moved to C4::Reserves:
GetBibsWithPendingHoldRequests()
GetPendingHoldRequestsForBib()
GetItemsAvailableToFillHoldRequestsForBib()
MapItemsToHoldRequests()
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
This patch deprecates the NoReturnSetLost system preference, which, as it
turns out, was not implemented anyway. New longoverdue script allows one to
specify on the commandline system-wide delays for changing items to different
lost statuses, and optionally charge for the item.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
merge works on the fly now.
But for an obscure reason, merge_authority.pl fails to update database when lanched on command line.
Adding one table to LOCK for noZebra UPDATE in Biblio.pm
You should remove C4::Search from merg_authority.pl
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Correct bug in output of overdue notices that has
items.content displaying only the number of columns
in each items.content structure (currently, 4).
Patch inspired by patch submitted by Paul Poulain.
[LL bug 31]
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
This patch changes all $foo eq undef's to !defined($foo). It also makes misc/spellcheck_suggest/build_spellcheck_suggest.pl have proper syntax.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The lines for each item in the overdue notices were not separated by newlines. This cause them to
all be smashed together. I'm putting a newline between them.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
misc/maintenance/fix_accountlines_date.pl runs through the accountlines table and converts every date in the 'description' column from us or metric format (specified on the command line) to the format in the 'dateformat' syspref.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
These three programs have been replaced by misc/cronjobs/overdue_notices.pl:
overduenotices-30.pl
overduenotices-csv.pl
overduenotices.pl
This patch deletes them.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
This patch adds the misc/cronjobs/overdue_notices.pl script that is intended to replace
overduenotices.pl, overduenotices-30.pl and overduenotices-csv.pl. It adds messages to
the message_queue to be sent later (by process_message_queue.pl). It also marks borrowers
as debarred if their issues become too overdue.
It is intended to be run from cron nightly with usage something like:
0 2 * * * misc/cronjobs/overdue_notices.pl
C4::Members:
- improved documentation on ModMember
- made ModMember return a useful value (the return value of the database call)
- added a DebarMember method
- adding t/lib/KohaTest/Members/DebarMember.pm to test ModMember
misc/cronjobs/overdue_notices.pl
- designed to replace overduenotices.pl, overduenotices-30.pl, and overduenotice-csv
Changes to C4::Letters:
- EnqueueLetter now lets you pass in to_address and from_address which can override defaults
- _send_message_by_email pays attention to these defaults.
- now handles attachments with MIME::Lite
C4::Overdues
- added GetBranchcodesWithOverdueRules
- added t/lib/KohaTest/Overdues/GerBranchcodesWithOverdueRules.pm to test that.
circ/overdue.pl
- replaced call to obsolete overduenotices-csv.pl with call to overdue_notices.pl
KohaTest:
- added three helper methods: random_phone, random_email, random_ip
- these can be used to populate example records
- you can now pass an optional lengh to random_string
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
el-GR-i-staff-t-prog-v-3000000.po
es-ES-i-staff-t-prog-v-3000000.po
hy-Armn-i-staff-prog-v-3000000.po
ru-RU-i-staff-t-prog-v-3000000.po
tr-TR-i-staff-t-prog-v-3000000.po
uk-UA-i-staff-t-prog-v-3000000.po
zh-Hans-CN-i-staff-t-prog-v-3000000.po
These will require extensive modifs to apply correctly due to
syntax problems.
I've added methods to to C4::Letters to manage the database table
message_queue. This will let us keep track of messages sent
via email, sms, and rss to patrons. That way, we can show the history,
deal with failures, and reconstruct an RSS feed when needed.
misc/cronjobs/overduenotics.pl has been added. It prepares advance notices
and item due notices and stages messages to be sent in the message_queue
table.
C4::Overdues::Getoverdues now takes two optional arguments to tell it how
old of overdues to fetch.
Also, a C4::Circualtion::getUpcomingDueIssues method was added that
advance_notices.pl uses.
misc/cronjobs/process_message_queue.pl has been added. It sends the email
or SMS messages out of the message queue.
The C4::SMS module didn't work at all, and it has been rebuilt to use
an external perl module from CPAN, SMS::Send.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Prior to this patch, rebuild_zebra.pl -z was effectively
hanging on to a lock on the zebraqueue table, preventing
other scripts from inserting new entries into the table.
This had the effect of causing circulation operations
to time out.
Refactored by having rebuld_zebra.pl pull the active
queue into memory, then mark entries done by zebraqueue.id.
Consequently, rebuild_zebra.pl should no longer
block adding new entries into zebraqueue.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Created a new script, sync_items_to_marc_bib.pl,
to replace the item tags embedded in the MARC bib
records with fresh versions taken from the items table.
This script should be run as follows:
maintenance/sync_items_to_marc_bib.pl --run-update
Assuming that you're using Zebra, rebuild_zebra.pl -b -z
or rebuild_zebra.pl -b -r should be run after running
this script.
This script should be run if you have used
link_bibs_to_authorities.pl prior to the first
patch for bug 2258. It can also be used if there
is any reason to suspect that the embedded item tags
do not reflect the items table.
With this script I am creating a maintenance/ subdirectory of
misc/ to hold scripts that are meant to fix problems
in the database but are not (or should not be, anyway) necessary
for regular use.
Documentation: add to documentation for server side scripts
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
If a MARC bib is modified by this batch job,
do not duplicate the item tags embedded in
it (e.g., 952 for MARC21). When modifying
a bib record, any embedded item tags must
be removed before calling ModBiblio - perhaps
this should be moved to ModBiblio itself.
Also removed an error in the job's help text.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
During link_bibs_to_authorities.pl run, a bib was
observed that had both biblioitems.marc and biblioitems.marcxml
set to blank or NULL. Added check to verify that a
defined value is returned from GetMarcBiblio(); may follow
up with additional patches once it's determined how
the corrupt bib came to be loaded.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
[1] Increase sleep interval between checks of zebraqueue
from 0.01 seconds to 0.50.
[2] Batch up commits of changes to the zebraqueue table
[3] If the same record appears multiple times in the queue,
handle only once.
[4] Properly postpone failures to process record deletes to
avoid spinning.
[5] Correct how queue entries are marked done - avoid skipping
an authority record update, e.g., if it has the same
ID number as a bib that was updated.
[6] Added a FIXME about a possible later enhancement to
batch up updates so that Zebra isn't told to commit
after each record.
No documentation changes.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
from circulation dashboard, creates new sysprefs, assigns
the sysprefs to the proper tab in sysprefs (Circulation),
updatedatabase changes to do the previous, and fixing one
redundent limit in the query for build_holds_queue.pl
Note: still need to address item-level holds
the tmp_holdsqueue table. This is an alternative holds
targeting workflow that is more suitable for multi-location
libraries than the default holds picklist report.
Note to documentation writers: this summary should be
added to any holds documentation as an overview of
the avaialable methods for holds fulfillment.
This alternative holds workflow assumes an
expectation that the system should target a specific
item for a given hold request, attempt to fulfill the
hold with that item, and if unable to fulfill, select
an available item at another location to fulfill the
hold.
This is quite different than the default Koha behavior
which uses a 'broadcast' method of hold fulfillment.
How it works:
This script weights available locations for holds based
on options specified in two system preferences:
StaticHoldsQueueWeight
Allows the library to specify a list of library
location codes -- if used alone, it will rank the
list statically, selecting the top-ranking available
location to be added to the picklist.
RandomizeHoldsQueueWeight
If RandomizeHoldsQueueWeight and StaticHoldsQueueWeight
are set, the list of library codes in the
StaticHoldsQueueWeight syspref are randomized rather
than statically ranked. If RandomizeHoldsQueueWeight
alone is set, the list of all available library codes
is used to randomize the weight.
If neither syspref is set, the list is statically
ranked according to how they are pulled out of the system
database.
NOTE: This has not yet been tested with item-level holds
this script had quite serious issues :
- it would not use mindays and maxdays variables
- It would send latin1 where utf8 was expected
- It would send data without text delimiters (; was chosen if title contains ; it would have been a problem " used as delimiters now)
- It would write a file when it was not asked
Now stores the results in a string before printing it.
New option added to store result into a file : -o filename
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enhanced the ability of catalogers to specify how
bib and item records should be added, replaced, or
ignored during a staging import.
When an import batch of bib records is staged and commit,
the user can now explicitly specify what should occur
when an incoming bib record has, or does not have, a match
with a record already in the database. The options are:
if match found (overlay_action):
create_new (just add the incoming record)
replace (replace the matched record with the incoming one)
use_template (option not implemented)
ignore (do nothing with the incoming bib; however, the
items attached to it may still be processed
based on the item action)
if no match is found (nomatch_action):
create_new (just add the incoming record)
ignore (do nothing with the incoming bib; in this
case, any items attached to it will be
ignored since there will be nothing to
attach them to)
The following options for handling items embedded in the
bib record are now available:
always_add (add the items to the new or replaced bib)
add_only_if_match (add the items only if the incoming bib
matches an existing bib)
add_only_if_add (add the items only if the incoming bib
does *not* match an existing bib)
ignore (ignore the items entirely)
With these changes, it is now possible to support the following use cases:
[1] A library joining an existing Koha database wishes to add their
items to existing bib records if they match, but does not want
to overlay the bib records themselves.
[2] A library wants to load a file of records, but only handle
the new ones, not ones that are already in the database.
[3] A library wants to load a file of records, but only
handle the ones that match existing records (e.g., if
the records are coming back from an authority control vendor).
Documentation changes:
* See description above; also, screenshots of the 'stage MARC records
for import' and 'manage staged MARC records' should be updated.
Test cases:
* Added test cases to exercise staging and committing import batches.
UI changes:
* The pages for staging and managing import batches now have
controls for setting the overlay action, action if no match,
and item action separately.
* in the manage import batch tool, user is notified when they
change overlay action, no-match action, and item action
* HTML for manage import batch tool now uses fieldsets
Database changes (DB rev 076):
* added import_batches.item_action
* added import_batches.nomatch_action
* added 'ignore' as a valid value for import_batches.overlay_action
* added 'ignored' as a valid value for import_records.status
* added 'status' as a valid value for import_items.status
API changes:
* new accessor routines for C4::ImportBatch
GetImportBatchNoMatchAction
SetImportBatchNoMatchAction
GetImportBatchItemAction
SetImportBatchItemAction
* new internal functions for C4::ImportBatch to
determine how a given bib and item are to be
processed, based on overlay_action, nomatch_action,
and item_action:
_get_commit_action
_get_revert_action
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
If running Zebra, try to locate zebrasrv and zebraidx
so that koha-zebra-ctl.sh can point to a functioning
zebrasrv.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
note 995 for items is hardcoded, so it's really for UNIMARC only. The script exit if you're not UNIMARCflavour
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The daemon control scripts (koha-zebra-ctl.sh, koha-zebraqueue-ctl.sh,
and koha-pazpar2-ctl.sh) are now copied and installed in a
runnable fashion for a 'dev'-mode install. By default
they are installed in the bin subdirectory of the runtime
directory.
Also:
* the control scripts now work if the EUID is other
than root (as would be expected for a 'dev' or 'single'
install).
* Split the SCRIPT_DIR installation target into
SCRIPT_DIR (scripts to copy regardless of install mode)
and SCRIPT_NONDEV_DIR (scripts to copy to SCRIPT_DIR
unless the install mode is 'dev').
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
rebuild_zebra.pl will now mark all zebraqueue entries
of the affected record type(s) done when run in
normal mode to index all records (as opposed to running
it with -z to just process the zebraqueue). This prevents
any running zebraqueue_daemon processes from attempting
to reindex the same records, redundantly.
The new -y swtich overrides this new behavior; in other words, if
running rebuild_zebra.pl without -z, you can specify
-y to *not* mark zebraqueue done.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Add a new parameter -o to begin importing input file after skiping
n records.
* Enclose input file reading in an eval directive to avoid abording
import if few records are corrupted: they are now skipped.
* Help formating.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Add a explanation on DBD::mysql installation without test suite.
Add /misc/translator/install-code.pl script that creates templates
for specified language codes.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Accidentally introducing a circular reference in a
MARC::Record object does not lead to goodness, particularly
if you export lots and lots of them.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The -z option, when used in conjunction with -a and/or -b,
selects the records to reindex from the zebraqueue table.
Both record updates and record deletes are handled.
-z is cannot be used with -s or -r: the updated records
must always be freshly exported, and if zebraqueue
is to be processed, it's assumed that you don't want
to drop the Zebra index first.
This means that rebuild_zebra.pl -b -a -x can be
used as a cronjob to update the indexes periodically; it
is believed that this will offer much better indexing
performance on some setups as compared to zebraqueue_daemon.pl,
which uses Z39.50 extended services to send record updates
to Zebra.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
At moment using both -a (index authorities) and
-x (export records as MARC XML) is not allowed -
if the Zebra authority database is using the DOM
filter, zebraidx will not be able to process the
exported records correctly.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
1. Logic to fix up record IDs, UNIMARC 100 field,
and record leader now in separate functions.
2. Removed (incorrect) logic to save corrected record
in database.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The version of MARC::Batch->new() distributed with version
2.0.0 of MARC::Record, if given a file name, will
open it using the ':utf8' layer. This results in an
incorrect character conversion when processing records
in the MARC-8 character encoding.
To avoid this, batch jobs that use MARC::Batch now
open the file themselves, then pass the file handle
to MARC::Batch->new().
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The new tables have the same structure and constraints
as the tables they archive with the following exceptions:
* borrowernumber and biblionumber in old_reserves can be
NULL
* the FK constraints (e.g., for itemnumber) on old_reserves
set the child column to NULL if the parent row is deleted
instead of deleting the child row.
* there is no FK constraint on old_issues.branchcode, allowing
a branch to be deleted without changing archived requests.
Some miscellaneous cleanup was done as part of this patch:
* GetMemberIssuesAndFines (C4::Members) now uses bind variables
* fixed POD for GetMemberIssuesAndFines
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
INTRANET_TMPL_DIR
INTRANET_WWW_DIR
OPAC_TMPL_DIR
OPAC_WWW_DIR
KOHA_CONF_DIR
OPAC_TMPL_DIR
OPAC_WWW_DIR
PAZPAR2_CONF_DIR
ZEBRA_CONF_DIR
For each directory on the list, if an existing file
installed file is different from the version
coming from the new package, it is copied to a backup file
whose name will have the suffix "_koha_<old_version_number>"
or "_upgrade_backup", depending on whether
the --prev-install-log option was used or not
during 'perl Makefile.PL'.
The directories whose files were backed up were
chosen because they contain configuration and
HTML files that a non-programmer Koha user
is likely to change. Consequently, the backup
files produced by 'make upgrade' are not a substitute
for doing a complete backup of one's Koha
installation before upgrading.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* IsStringUTF8ish - determine if scalar contains a string in UTF8
* MarcToUTF8Record - convert MARC blob or MARC::Record to UTF8
* SetMarcUnicodeFlag - set appropriate MARC21 or UNIMARC field to
indicate that record is in UTF-8.
Design points of this module include:
* No dependencies on other C4 modules, making it easier to add
more test cases
* All character conversion code in one place
* Single entry point for doing a character conversion on a
MARC record
* Capture of errors and warnings produced by Text::Iconv
and MARC::Charset
* Start of support for guessing the source character set of
a MARC record.
Several functions were moved from other scripts
or modules to C4::Charset:
* C4::Koha->FixEncoding (expanded and renamed
MarcToUTF8Record)
* C4::Koha->char_decode5426
* fMARC8ToUTF8 from bulkmarcimport.pl (renamed
_marc_marc8_to_utf8)
Several batch jobs were adjusted to use MarcToUTF8Record instead of
FixEncoding.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Fixed syntax errors preventing compilation; however,
unsure whether this is a dead utility that should be
removed outright.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Needed to restore OpenSearch capabilities, and did the following while
I was at it:
* add support for unAPI: http://unapi.info/
* add basic support for COinS and OpenURL:
http://ocoins.info;
http://www.niso.org/committees/committee_ax.html
* ^^ Gives us Zotero Support!
* adding some XSLT stylesheets for handling additional transformations
NOTE: English and MARC21 specific unfortunately
* adding back opensearch/rss feed <link>s for autodiscovery
TODO: after the installation, to get the Zebra system running on an external
port it's necessary to hand-edit the configs. I'm looking into Virtual Hosts
which could solve that problem (run on both the socket and a port).
Need to add better error handling to the unapi and opensearch scripts
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Replace C4::Biblio::AddBiblioAndItems with two
things:
* An option to C4::Biblio::AddBiblio to defer writing
biblioitems.marc and biblioitems.marcxml. This
option was created to give a significant
speed boost to bulkmarcimport.pl, but is *not*
recommended for general use.
* C4::Items::AddItemBatchFromMarc
This refactoring removes the need to have functions
in C4::Biblio and C4::Items that call each other's
private functions.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* Move CheckItemPreSave to C4::Items (from C4::Biblio)
* Modified C4::Biblio::AddBiblioAndItems to use appropriate
internal routines from C4::Items
* Moved GetItemnumberFromBarcode to C4::Items
* Removed duplicate C4::Biblio::_koha_new_items
* Removed disused C4::Biblio::MARCitemchange
Currently AddBiblioAndItems is a special routine that
uses private subs from both C4::Biblio and C4::Items.
This needs to be refactored.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Prior to this fix, the status fields had three 'off' values, NULL, "",
and 0. I've reduced it to two in the db, removing the option for NULL, and
setting the default value to 0, however, we need to verify that we don't ever
write out as "" as this needlessly complicates the indexing process,
critical for searching or limiting by status (e.g., availability). Also,
queries that attempt to write a NULL value to one of these fields will fail
(based on my tests).
This patch includes the following changes:
* Updated the database definition for notforloan, damaged, itemlost, and
wthdrawn in kohastructure.sql to forbid NULL and default to 0; MySQL
can't forbid other values (such as empty ""), so this has to be handled
at the application layer and REQUIRES further patching.
* Fixed the 'limit by availability' query node in Search.pm to use a
much less confusing definition of 'available'
* Added code to set values to 0 where they are NULL or empty ( "" ) for
notforloan, damaged, itemlost or wthdrawn in both the MARC and the items
table:
* Biblio.pm -> AddBiblioAndItems
* catalogue/updateitem.pl
* SEE NOTE BELOW, REQUIRES UPDATE TO THE REST OF KOHA'S ITEM MGT!
* Removed code in bulkmarcimport.pl that sets notforloan status depending
on item-level or bib-level itemtype -- that flag is designed to be set
only to override the notforloan setting for the item's (or bib's,
depending on the syspref) assigned itemtype (it doesn't need to override
to 'for loan', only to 'not for loan').
added $dbh->do("truncate zebraqueue"); when operation is 'delete'
* I updated some notes in catalogue/updateitem.pl as to why ModItem can't be
used -- we don't have _a_ place where we can change the item and marc :/
I've tested the following:
bulkmarcimport.pl..........................MARC/items OK
Staged Records Import......................NOT OK
updateitem.pl (via moredetail.pl)..........MARC/items OK
circulation.pl.............................NOT OK
returns.pl.................................NOT OK
addbiblio.pl...............................NOT OK
additem.pl.................................NOT OK
Basically, there isn't a single place to apply this patch that will
update both item data and MARC data in one place ... a future patch
needs to address this issue.
Signed-off-by: Galen Charlton <galen.charlton@liblime.com>
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
the unimarc stuff has been moved to marc_defs directory and the
lang specific is in lang_defs
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Enabled automatic conversion of MARC-8 records to
UTF-8. Record is converted if its Leader/09 contains
a blank and the -s (skip) option hasn't been supplied
on the command-line. Any record that cannot be converted
to UTF-8 is skipped.
Also now use Unicode Normalization Form C (NFC) for
records converted from MARC-8.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Introduced new C4::Biblio function CheckItemPreSave,
which checks for duplicate barcodes and invalid
branch codes. Not yet sure whether this function
needs to be exported or whether it will just be
used internally to C4::Bibli.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
Changes to improve speed of MARC bib and item
imports:
[1] Turn off autocommit and commit database
transactions in larger batches.
[2] Introduce a new C4::Biblio function (AddBiblioAndItems)
to combine AddBiblio and AddItems -- this is faster
because we are not parsing the MARC XML of the biblio
every time we add an item.
[3] Introduce FasterTransformMarcToKoha, which is much
faster than TransformMarcToKoha. The new version,
which will replace the old one once it has been
fully tested, scans through each field in the
MARC record just once, instead of potentially
dozens of times.
[4] Remove code in bulkmarcexport that moved the
item tags to separate MARC::Record objects.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* adding zebra daemons for managing server and queue processes
* improvements to the README.debian file
* Fixes to Search.pm since last series of commits broke zebra-based
searching (again)
* moving some files to new misc/bin and misc/cronjobs
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
[1] Use File::Temp to create and manage
export directory if -d is not specified.
[2] Added usage message.
[3] Code that attempts to fix up Zebra
configuration files changed so that it
is invoked only if --munge-config option
is supplied; this code will ultimately
either be removed or moved to a separate
script -- the sorts of errors that it
tries to fix should no longer be appearing
in a standard install.
[4] Fixed Win32 portability problem when removing
temporary directory.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* rewrite-config.PL now puts in installed location
of koha-conf.xml in C4/Context.pm so that
correct config can be found even when
KOHA_CONF is not set. Note that setting KOHA_CONF
will still override path set by installer.
* changed references from koha.xml to koha-conf.xml
The correct field is 'items.wthdrawn', not 'items.withdrawn'.
Spelling may be corrected in post-3.0 version.
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
There are only 2 UNIMARC specific files (.abs and .chr), they have been moved to etc/zebradb
The rebuild_zebra.pl takes all config file from this location now.
the misc/zebra/ can be removed (and will be soon)
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The authoritative data for the items is the items table, not the MARC data.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
this time it's when a biblio don't have biblionumber, has a 100$a field, and it's invalid.
1 biblio in my 300 000 DB (and it was biblio 294 359, of course !)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
the API of TransformMarcToKoha has changed.
+
For a reason I don't understand (& appear a conception bug), the biblioitemnumber is not shipped in the hash
and it's needed by the UPDATE. So it has to be reintroduced manually.
That's how it's done in Biblio.pm/Sub ModItem.
I think 3.2 will see the end of biblioitems (& it's replacement by something else)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
now, adding both on the fly when needed.
(had 2 biblios like that in a 290 000 DB, but was enought to have M::F::X complaining & diing !)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
* now properly deletes deleted authorities from index
* now actually ignores duplicate work
* standardized on recordDelete instead of delete_record
for bibs and recordDelete for authorities
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
As part of this, modified two routines in C4::ImportBatch
to support a callback for monitor progress of import
processing.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
(authorities that don't have a 001 field containing authid)
also comment some code when exporting biblios (NOT tested, hdl,pls confirm this commit)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
The zebraque_start scripts works, but it not friendly when you try to track what he did
as records are deleted after done.
This commit changes the behaviour : 2 columns are added done & timestamp.
done is set to 1 when a line have been done. And the timestamp contains the timestamp of the last action (either line creation if done=0 or operation if done=1)
should be helpfull to track problem.
the table will grow, but i'll add soon a DELETE FROM zebraqueue WHERE timestamp > 30 days or something like that
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
- the quotemeta was wrong (and introduced some bugs in diacritics)
- fixing some bugs that appear only sometimes : the union was done including weight, which is wrong & resulted in missing some results (when various weighting)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
This Bug came out of the fact that indexes were used a wrong way in record.abs
Please consider correcting record_usmarc.abs
Indeed indexes were used :
melm NNN$X Myattribute !:w,!:p
This prooved not to work on indexes.
It took only default (w) index for Myattribute.
So Please, correct your record.abs, reindex, and sorting by publication date will be fine.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
If 100$a repeated, the scripts failed to handle that correctly
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
(just spelling, but very important, without them, authorities won't work)
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
- reindenting
- upcasing SQL
- the script at least compiles...
... but it does seem not work yet
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
- pay.pl fixed, the librarian can see patron fines & mark them paid
- fines2.pl fixed, the script now calculate the fines correctly from the finerules :
* itemtype / patron category if it exist
* itemtype / default if needed
- renamed misc/fines.pl as fines-sanop.pl
- added 2 systempreferences:
* MaxFine: the max amount a patron can be charged for an item
* NoReturnSetLost: how many days of late before a non returned item is marked lost & the patron charged for the full cost of replacement
(those values where hardcoded in fines2.pl
- C4/Circulation/Fines.pm has been removed (unused duplicate of C4/Overdues)
Note that SANOP feature about notify levels have NOT been ported here. I think they are too specific now, and the code is poorly written
I've renammed fines.pl to fines-sanop.pl to point that it is specific.
Thus, all notify_id related features are not used by anything (and always 0)
Can be interesting to reintroduce them, but that will probably be a large work...
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
nobody probably ever enter "." on search term, but just in case...
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
This option uses the iso2709 version of the MARC record instead of the XML one
(biblioitems.marc vs biblioitems.marcxml)
No change if the parameter is not set.
Signed-off-by: Chris Cormack <crc@liblime.com>
Signed-off-by: Joshua Ferraro <jmf@liblime.com>
- adding browser and nozebra table definition to kohastructure & updatedatabase
- bumping to 3.00.00.005
Signed-off-by: Chris Cormack <crc@liblime.com>
- use biblio instead of marc_biblio,
- better check that biblionumber is correctly stored
- fix an buggy API call when ModMarcBiblio
Signed-off-by: Chris Cormack <crc@liblime.com>
local LC call number as used at many North American academic libraries.
Correcting authoritytypecode values where my typo used the number '0' as
part of the value in one table and the letter 'O' in another table.
Improved seealso column indexing values.
Signed-off-by: Chris Cormack <crc@liblime.com>
part of the value in one table and the letter 'O' in another table.
Improved seealso column indexing values.
Signed-off-by: Chris Cormack <crc@liblime.com>
This script allows you to update your database/records after a bulkmarckimport integration.
Currectly it reformats biblioitems.isbn and bibliotitems.marcxml on deleting any '-' on isbn.
- updating templates to have tmpl_process3.pl running without any errors
- adding a drupal-like css for prog templates (with 3 small images)
- fixing some bugs in circulation & other scripts
- updating french translation
- fixing some typos in templates
- support for authorities
- some bugfixes in ordering and "CCL" parsing
- support for authorities <=> biblios walking
Seems I can do what I want now, so I consider its done, except for bugfixes that will be needed i m sure !
* adding 3 subs in Biblio.pm
- GetNoZebraIndexes, that get the index structure in a new systempreference (added with this commit)
- _DelBiblioNoZebra, that retrieve all index entries for a biblio and remove in a variable the biblio reference
- _AddBiblioNoZebra, that add index entries for a biblio.
Note that the 2 _Add and _Del subs work only in a hash variable, to speed up things in case of a modif (ie : delete+add). The effective SQL update is done in the ModZebra sub (that existed before, and dealed with zebra index).
I think the code has to be more deeply tested, but it works at least partially.
- changing nozebra table to have biblionumber,title-ranking; (; is the entry separator. Now, if a value is several times in an index, it is stored only once, with a higher ranking (the ranking is the number of times the word appeard for this index)
- improving search to have ranking value (default order). The ranking is the sum of ranking of all terms. The list is ordered by ranking+title, from most to lower
- add nozebra table management on biblio editing
- the index table content is hardcoded. I still have to add some specific systempref to let the library update it
- manage pagination (next/previous)
- manage facets
WHAT works :
- NZgetRecords : has exactly the same API & returns as zebra getQuery, except that some parameters are unused
- search & sort works quite good
- CQL parser is better that what I thought I could do : title="harry and sally" and publicationyear>2000 not itemtype=LIVR should work fine
Abiding by Name Convention.
Using Members wherever it should be used.
Borrower is only used for borrower Categories.
+ GetBorrowersWhoHaveNeverBorrowed
and lists like that.
- checkaccount and getborraccountno => GetBorrowerAcctRecord
Many changes in names,
some changes in function signature.
Will be detailed in a mail to kohadevel.
All subs have be cleaned :
- removed useless
- merged some
- reordering Biblio.pm completly
- using only naming conventions
Seems to have broken nothing, but it still has to be heavily tested.
Note that Biblio.pm is now much more efficient than previously & probably more reliable as well.
== Biblio.pm cleaning (useless) ==
* some sub declaration dropped
* removed modbiblio sub
* removed moditem sub
* removed newitems. It was used only in finishrecieve. Replaced by a Koha2Marc+AddItem, that is better.
* removed MARCkoha2marcItem
* removed MARCdelsubfield declaration
* removed MARCkoha2marcBiblio
== Biblio.pm cleaning (naming conventions) ==
* MARCgettagslib renamed to GetMarcStructure
* MARCgetitems renamed to GetMarcItem
* MARCfind_frameworkcode renamed to GetFrameworkCode
* MARCmarc2koha renamed to TransformMarcToKoha
* MARChtml2marc renamed to TransformHtmlToMarc
* MARChtml2xml renamed to TranformeHtmlToXml
* zebraop renamed to ModZebra
== MARC=OFF ==
* removing MARC=OFF related scripts (in cataloguing directory)
* removed checkitems (function related to MARC=off feature, that is completly broken in head. If someone want to reintroduce it, hard work coming...)
* removed getitemsbybiblioitem (used only by MARC=OFF scripts, that is removed as well)
cvs -z3 -d:ext:kados@cvs.savannah.nongnu.org:/sources/koha co -P koha
find koha.precrash -type d -name "CVS" -exec rm -v {} \;
cp -r koha.precrash/* koha/
cd koha/
cvs commit
This should in theory put us right back where we were before the crash
subfields in the 09o or 999 placeholer for 090 Locally Assigned LC-Type Call
Call Number found in copy catalogued records. The locally assinged original
090 call number remains visible in tab 0 for informational purposes as often
the only place to find what an LC Call number for the material might be.
This file now combines koha configuration and zebra configuration. Old koha.conf variable names are retained. They go between the tags <config>. Do not change the tag names for zebra part. Use zebrasrv -f /etc/koha.xml to start your server. The virtual server configuration details can be found in YAZ documentation
only allows numbers up to 127 - needs to allow bigger numbers if the sub you're making
is for a daily subscription like a newspaper where the number of issues in a year
will be greater than 127
head repo (ie, copies over the right files, creates some symlinks,
etc.). It's not finished yet, but getting there (I'll be testing it
extensively today and tomorrow).
It now responds to:
-n : the number of records to import.
-commit : the number of records to wait before performing a 'commit' operation
ALSO: IMPORTANT: I took out the char_encoding as this should be handled by
MARC::File::XML now, unless I'm mistaken.
recordId: (bib1,Identifier-standard) just after the comma. Adam agreed it was a bug, and it should be solved soon. But now we are aware, we can avoid putting the space !
In this commit you have all what is needed to setup a working zebra DB in Unimarc :
* collection.abs is UNIMARC specific and must be rewritten for MARC21, in marc21 directory
* pdf.properties is to be copied unmodified in the marc21 directory (can also be put somewhere else)
* rebuild_zebra.pl is SLOW, but 1 step reindexing tool, using ZOOM
* rebuild_zebra_idx is FAST, but 2 step reindexing tool, and does not use zebra. run it, it will create all biblios XML files in /zebra/biblios directory, then zebraidx update biblios in your zebra directory
* zebra.cfg is the zebra config file ;-)
* test_cql2rpn.pl is a script that will query the database and show the results. Works for me, just change the query at the beginning to get answers you expect.
What has to be done :
* benchmarking : it seems the zebraidx update is faster than lightning (400biblios/sec : 10 000biblios in 25seconds), while ZOOM indexing is slow (something like 25biblios/second) More benchmarking could be done.
* completing collection.abs for UNIMARC. I'll take care of it.
* modifying Biblio.pm to use ZOOM instead of the "zebraidx through exec" running actually. I'll take care of it also.
* modify the search API & tools & screens. I'll let the ball to someone else (chris ?) for this. I agree SearchMarc.pm can be dropped and replaced by something else (maybe a new-and-clean Search.pm package)
Seems not to break too many things, but i'm probably wrong here.
at least, new features/bugfixes from 2.2.5 are here (tested on some features on my head local copy)
- removing useless directories (koha-html and koha-plucene)
* go to koha cvs home directory
* in misc/zebra there is a unimarc directory. I suggest that marc21 libraries create a marc21 directory
* put your zebra.cfg files here & create your database.
* from koha cvs home directory, ln -s misc/zebra/marc21 zebra (I mean create a symbolic link to YOUR zebra directory)
* now, everytime you add/modify a biblio/item your zebra DB is updated correctly.
NOTE :
* this uses a system call in perl. CPU consumming, but we are waiting for indexdata Perl/zoom
* deletion still not work
* UNIMARC zebra config files are provided in misc/zebra/unimarc directory. The most important line being :
in zebra.cfg :
recordId: (bib1,Local-number)
storeKeys:1
in .abs file :
elm 090 Local-number -
elm 090/? Local-number -
elm 090/?/9 Local-number !:w
(090$9 being the field mapped to biblio.biblionumber in Koha)
how it works :
* create the table marc_Tword with the following structure :
CREATE TABLE `marc_Tword` (
`word` varchar(80) NOT NULL default '',
`usedin` text NOT NULL,
`tagsubfield` varchar(4) NOT NULL default '',
PRIMARY KEY (`word`,`tagsubfield`)
) TYPE=MyISAM;
* open a console & type export PERL5LIB & export KOHA_CONF as usual.
* fill this table with misc/build_marc_Tword.pl. Warning, this script uses a very very consumming but very fast method to fill the table : it does everything in memory, then write everything. Another method is provided (& commented), but it's 100x times slower (really !)
* open opac-search.pl and replace use C4::SearchMarc; by use C4::SearchMarcTest; as the API hasn't changed, it will work immediatly.
* go to opac-search (advanced search) & search whatever you want. Should work fine.
LIMITS :
* build_marc_Tword has problem with extended chars (accented ones mainly). So don't be afraid if you get sql errors. They are not a problem for a POC
* search works always order by title, whatever you choose.
* search works only search WORDA and WOARDB, not yet WORDA or WORDB or WORDA except WORDB.
Make the generated pot file (i.e., result of "create") look more "real",
but using msgmerge to reformat the output
Script failed to create intermediate directories if the directory of the
target does not exist and the parent of that directory does not exist
either. This should fix that.
The biblio & the items are separated.
The bug was that the item was still in the biblio (thus being added twice : as "biblio" and as "item")
thanks to dana for pointing the bug.
the user can select sql-datas to add to it's DB
can be populated by any SQL
french UNIMARC sample being added really soon
- framework for monography
- framework for URL
- framework for ...
- Personal UNIMARC autority
...
table with a reduced number of tags (only those
tags that should be searched) allowing for
faster and more accurate searching when used
with the SearchMarc routines. Make sure that
the MARCaddword routine in Biblio.pm will index
characters >= 1 char; otherwise, searches like
"O'brian, Patrick" will fail as the search
routines will seperate that query into "o",
"brian", and "patrick". (If "o" is not in the
database the search will fail)
spurious context to be recognized. In particular, the bugs fixed are:
1. Failure to recognize INPUT element at the end, e.g., if the input has
the form "Item number:%S", then the pattern was recognized as only
"Item number".
2. Failure to remove matching <foo></foo> tags if the pattern contains
INPUT or TMPL_VAR; e.g., if the input has the form "<h1>%s %s</h1>",
the form would not be simplified to "%s %s".
Unfortunately, fixing these 2 bugs will cause about 40 fuzzies to appear.
Amazon and store them in a ratings table in the Koha db
ratings.sql is the table definition
My next task is to extend it to work for cd's and dvds as well.
fines.pl will levy fines and place them on patrons accounts, it will also then attempt to notify
the borrowers.
Either by email, sms, fax etc depending on what the patron has set as preferred contact.
Email and letters are currently working fine, fax, and sms need to be worked on.
Test with caution in your library, you dont want to be bombarding patrons with emails :)
- If a string is enclosed by a tag, remove that tag from the extracted string
- Generate automatic comments to provide more information for the translator
- A couple bug fixes
It seems to be still correct, and it is no longer complaining about syntax
errors when seeing commented-out HTML (esp. with TMPL_* directives).
Don't try to translate stuff between <title>...</title> too, the stuff in
the middle is supposed to be PCDATA.
(Actually, xgettext.pl did not know how to handle them in the files listed
in the list of files.)
If the po file is empty (corrupted), $href->{'""'} will be undefined.
We just blindly dereferenced this null value without checking.
Minor comment update
Removed forced backups and the comment about interrupting xgettext.pl
corrupting the po file, now that we seem to be detecting the situation.
when minor whitespace changes occur in the original templates; it also
makes the strings much easier to read (e.g., instead of "foo\n\n\t\t bar",
xgettext.pl will now always generate "foo bar" and tmpl_process3.pl will
understand it to be the same as the original string).
Early termination of analysis if we encounter some strings, such as </h1>
or | or ||, in order to avoid extracting strings that are unnecessarily
long and which doesn't add any meaningful context.
to be combined together into one string. This seems to have the desired
effect (that "<b>foo</b> bar" type strings are now recognized in one piece).
However, "<h1>foo</h1>\nexplanation"-type things may now also be (arguably
wrongly) recognized as one piece.
either iso8859-1 or utf8, msgmerge(1) won't crap out. The code is ugly;
the conversion table is hard-coded, and in some place not very appropriate.
However, this does fix the case where a few strings containing French
characters can't be translated. As a side effect, tmpl_process3 can now
also be used for French or other languages using iso8859-1.
gives better strings. (Always allowing combinations gives havoc, we
currently avoid this by allowing combination only if the first and last
tokens are both TEXT.)
word order is too different than the word order of the target language to
yield meaningful translations.
The new scripts use a different translation file format (namely standard
gettext-style PO files).
This seems to reasonably work (e.g., producing an empty en_GB translation
then installing seems to not corrupt the "translated" files), but it likely
will still contain some bugs. There is also little documentation, but try
to run perldoc on the .p[lm] files to see what's there. There are also some
spurious warnings (both from bugs in the new scripts and from buggy third-
party Locale::PO module).
warnings aren't pedantic because (1) if it's a templating directive, it
might expand into something containing a real < and/or >, and (2) if it
contains >, the browser will close the current tag, and (3) if it contains
< and the browser knows what "SGML closed start tags" are (e.g., Mozilla),
the browser will also close the current tag.
tags like "<b foo" at the end of the file seems to be discarded silently by
Mozilla, even in quirks mode. We now display a warning for these (in case
these ever come up by accident).
This is a script for the use of printing, and emailing overdue notices.
The script must be run from the command line
Documentation is contained in the script
It loads an iso2709 file into the ACTIVE Database. My 1st tests on sample files shows it works quite fine.
Feel free to try it.
How it works ? enter ./bulkmarcimport in a console, and look.
Note : i'm proud, because it's less than 100 lines long => 99 lines !!!
the index.html shows just 2 links : 1 to intranet.html, the other to opac.html.
during buildrelease, intranet.html is moved to intranet document root and renamed to index.html, same thing for opac.html with opac documentroot.
with this trick, you can have a "cvs" install, and a clean "official" install
koha-tmpl directory in the koha module.
buildrelease also now uses an absolute path to find the buildrelease script for
guessing the current version number.
get "use strict" to do something useful regarding these global variables.
Because of this change, a number of accessor functions have been created to
let installer.pl get/set some of these global variables. (We should not use
the $Install:: notation for the global variables either, as that will also
make "use strict" do nothing about these variables.)
POD updates
Chris Cormack
Henri-Damien LAURENT
Paul Poulain
Kyle M Hall
Frédéric Demians
Nahuel ANGELINETTI
David Birmingham
Matthias Meusburger
J. David Bavousett
Ricardo Dias Marques
Sébastien Hinderer
Galen Charlton
Colin Campbell
James Winter
Ryan Higgins
Joe Atzberger
Brian Harrington
Galen Charlton
Michael Hafen
J. David Bavousett
Daniel Sweeney
Mason James
Brian Harrington
Marc Chantreux
Pianohacker
Andrew Moore
Vincent Danjean
Joshua Ferraro
Southeastern University
Jesse Weaver
Chris Nighswonger
Owen Leonard
Henri-Damien LAURENT
Ryan Higgins
Mason James
Ryan Higgins
Mason James
Daniel Bünzli
Chris Cormack
Paul POULAIN
paul
Chris Nighswonger
Mason James
thd
Chris Catalfo
MJ Ray
thd
tipaul
toins
hdl
kados
rangi
btoumi
rych
tgarip1957
bob_lyon
sushi
slef
joshferraro
acli
shedges
kb2qzv
wolfpac444
tonnesen
arensb
pate