Bug 17196: Move marcxml out of the biblioitems table

Two discussions on koha-devel lead to the same conclusion:
biblioitems.marcxml should be moved out this table
- biblio and biblioitems
http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html
- biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html

There are several goals to do it:
- Performance
As Paul Poulain wrote, a simple query like
  SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear;
takes more than 10min on a DB with more than 1M bibliographic records
but only 3sec (!) on the same DB without the biblioitems.marcxml field
Note that priori to this patch set, the biblioitems.marcxml was not
retrieved systematically, but was, at least, in
C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders
- Flexibility
Storing the marcxml in a specific table would allow use to store several
kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats (marcflavour)
- Clean code
It would be a first step toward Koha::MetadataRecord for bibliographic
records (not done in this patch set).

Test plan:
- Update the DBIC Schema
- Add / Edit / Delete / Import / Export bibliographic records
- Add items
- Reindex records using ES
- Confirm that the following scripts still work:
    * misc/cronjobs/delete_records_via_leader.pl
    * misc/migration_tools/build_oai_sets.pl
- Look at the reading history at the OPAC (opac-readingrecord.pl)
- At the OPAC, click on a tag, you must see the result

Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108.

Signed-off-by: Mason James <mtj@kohaaloha.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This commit is contained in:
Jonathan Druart 2016-08-25 11:41:56 +01:00 committed by Kyle M Hall
parent fdc15cb762
commit c6e488f4af
18 changed files with 166 additions and 70 deletions

View File

@ -41,6 +41,8 @@ use C4::Debug;
use Koha::Caches;
use Koha::Authority::Types;
use Koha::Acquisition::Currencies;
use Koha::Biblio::Metadata;
use Koha::Biblio::Metadatas;
use Koha::SearchEngine;
use Koha::Libraries;
@ -157,7 +159,7 @@ Biblio.pm contains functions for managing storage and editing of bibliographic d
=item 2. as raw MARC in the Zebra index and storage engine
=item 3. as MARC XML in biblioitems.marcxml
=item 3. as MARC XML in biblio_metadata.metadata
=back
@ -181,7 +183,7 @@ Because of this design choice, the process of managing storage and editing is a
=item 2. _koha_* - low-level internal functions for managing the koha tables
=item 3. Marc management function : as the MARC record is stored in biblioitems.marcxml, some subs dedicated to it's management are in this package. They should be used only internally by Biblio.pm, the only official entry points being AddBiblio, AddItem, ModBiblio, ModItem.
=item 3. Marc management function : as the MARC record is stored in biblio_metadata.metadata, some subs dedicated to it's management are in this package. They should be used only internally by Biblio.pm, the only official entry points being AddBiblio, AddItem, ModBiblio, ModItem.
=item 4. Zebra functions used to update the Zebra index
@ -189,7 +191,7 @@ Because of this design choice, the process of managing storage and editing is a
=back
The MARC record (in biblioitems.marcxml) contains the complete marc record, including items. It also contains the biblionumber. That is the reason why it is not stored directly by AddBiblio, with all other fields . To save a biblio, we need to :
The MARC record (in biblio_metadata.metadata) contains the complete marc record, including items. It also contains the biblionumber. That is the reason why it is not stored directly by AddBiblio, with all other fields . To save a biblio, we need to :
=over 4
@ -201,20 +203,6 @@ The MARC record (in biblioitems.marcxml) contains the complete marc record, incl
=back
When dealing with items, we must :
=over 4
=item 1. save the item in items table, that gives us an itemnumber
=item 2. add the itemnumber to the item MARC field
=item 3. overwrite the MARC record (with the added item) into biblioitems.marcxml
When modifying a biblio or an item, the behaviour is quite similar.
=back
=head1 EXPORTED FUNCTIONS
=head2 AddBiblio
@ -230,7 +218,7 @@ framework code.
This function also accepts a third, optional argument: a hashref
to additional options. The only defined option is C<defer_marc_save>,
which if present and mapped to a true value, causes C<AddBiblio>
to omit the call to save the MARC in C<bibilioitems.marcxml>
to omit the call to save the MARC in C<biblio_metadata.metadata>
This option is provided B<only>
for the use of scripts such as C<bulkmarcimport.pl> that may need
to do some manipulation of the MARC record for item parsing before
@ -1318,11 +1306,12 @@ sub GetMarcBiblio {
}
my $dbh = C4::Context->dbh;
my $sth = $dbh->prepare("SELECT biblioitemnumber, marcxml FROM biblioitems WHERE biblionumber=? ");
my $sth = $dbh->prepare("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber=? ");
$sth->execute($biblionumber);
my $row = $sth->fetchrow_hashref;
my $biblioitemnumber = $row->{'biblioitemnumber'};
my $marcxml = StripNonXmlChars( $row->{'marcxml'} );
my $marcxml = GetXmlBiblio( $biblionumber );
$marcxml = StripNonXmlChars( $marcxml );
my $frameworkcode = GetFrameworkCode($biblionumber);
MARC::File::XML->default_record_format( C4::Context->preference('marcflavour') );
my $record = MARC::Record->new();
@ -1351,17 +1340,24 @@ sub GetMarcBiblio {
my $marcxml = GetXmlBiblio($biblionumber);
Returns biblioitems.marcxml of the biblionumber passed in parameter.
Returns biblio_metadata.metadata/marcxml of the biblionumber passed in parameter.
The XML should only contain biblio information (item information is no longer stored in marcxml field)
=cut
sub GetXmlBiblio {
my ($biblionumber) = @_;
my $dbh = C4::Context->dbh;
my $sth = $dbh->prepare("SELECT marcxml FROM biblioitems WHERE biblionumber=? ");
$sth->execute($biblionumber);
my ($marcxml) = $sth->fetchrow;
my $dbh = C4::Context->dbh;
return unless $biblionumber;
my ($marcxml) = $dbh->selectrow_array(
q|
SELECT metadata
FROM biblio_metadata
WHERE biblionumber=?
AND format='marcxml'
AND marcflavour=?
|, undef, $biblionumber, C4::Context->preference('marcflavour')
);
return $marcxml;
}
@ -3216,9 +3212,6 @@ sub _koha_modify_biblio {
my ($biblioitemnumber,$error) = _koha_modify_biblioitem_nonmarc( $dbh, $biblioitem );
Updates biblioitems row except for marc and marcxml, which should be changed
via ModBiblioMarc
=cut
sub _koha_modify_biblioitem_nonmarc {
@ -3505,9 +3498,19 @@ sub ModBiblioMarc {
$f005->update(sprintf("%4d%02d%02d%02d%02d%04.1f",@a)) if $f005;
}
$sth = $dbh->prepare("UPDATE biblioitems SET marcxml=? WHERE biblionumber=?");
$sth->execute( $record->as_xml_record($encoding), $biblionumber );
$sth->finish;
my $metadata = {
biblionumber => $biblionumber,
format => 'marcxml',
marcflavour => C4::Context->preference('marcflavour'),
};
# FIXME To replace with ->find_or_create?
if ( my $m_rs = Koha::Biblio::Metadatas->find($metadata) ) {
$m_rs->metadata( $record->as_xml_record($encoding) );
} else {
my $m_rs = Koha::Biblio::Metadata->new($metadata);
$m_rs->metadata( $record->as_xml_record($encoding) );
$m_rs->store;
}
ModZebra( $biblionumber, "specialUpdate", "biblioserver", $record );
return $biblionumber;
}

View File

@ -421,7 +421,6 @@ sub GetPatronInfo {
my $branchname = $library ? $library->branchname : '';
# Remove unwanted fields
delete $item->{'marcxml'};
delete $item->{'more_subfields_xml'};
# Add additional fields

View File

@ -694,7 +694,7 @@ Looks up what the patron with the given borrowernumber has borrowed.
C<&GetPendingIssues> returns a
reference-to-array where each element is a reference-to-hash; the
keys are the fields from the C<issues>, C<biblio>, and C<items> tables.
The keys include C<biblioitems> fields except marc and marcxml.
The keys include C<biblioitems> fields.
=cut
@ -714,7 +714,6 @@ sub GetPendingIssues {
}
}
# must avoid biblioitems.* to prevent large marc and marcxml fields from killing performance
# FIXME: namespace collision: each table has "timestamp" fields. Which one is "timestamp" ?
# FIXME: circ/ciculation.pl tries to sort by timestamp!
# FIXME: namespace collision: other collisions possible.

View File

@ -158,7 +158,6 @@ Returns a count and a list of overdueitems for a given borrowernumber
sub checkoverdues {
my $borrowernumber = shift or return;
# don't select biblioitems.marcxml... too slow on large systems
my $sth = C4::Context->dbh->prepare(
"SELECT biblio.*, items.*, issues.*,
biblioitems.volume,

View File

@ -19,7 +19,7 @@ while (1) {
print "No item ($in)";
next;
}
for (qw(marc marcxml)) {
for (qw(marc marcxml)) { # Letting it just in case but should not longer be useful
$item->{$_} = 'suppressed...';
}
my $queue = $item->hold_queue();

44
Koha/Biblio/Metadata.pm Normal file
View File

@ -0,0 +1,44 @@
package Koha::Biblio::Metadata;
# This file is part of Koha.
#
# Koha is free software; you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation; either version 3 of the License, or (at your option) any later
# version.
#
# Koha is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with Koha; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
use Modern::Perl;
use Carp;
use Koha::Database;
use base qw(Koha::Object);
=head1 NAME
Koha::Metadata - Koha Metadata Object class
=head1 API
=head2 Class Methods
=cut
=head3 type
=cut
sub _type {
return 'BiblioMetadata';
}
1;

50
Koha/Biblio/Metadatas.pm Normal file
View File

@ -0,0 +1,50 @@
package Koha::Biblio::Metadatas;
# This file is part of Koha.
#
# Koha is free software; you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation; either version 3 of the License, or (at your option) any later
# version.
#
# Koha is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with Koha; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
use Modern::Perl;
use Carp;
use Koha::Database;
use Koha::Biblio::Metadata;
use base qw(Koha::Objects);
=head1 NAME
Koha::Biblio::Metadatas - Koha Metadata Object set class
=head1 API
=head2 Class Methods
=cut
=head3 type
=cut
sub _type {
return 'BiblioMetadata';
}
sub object_class {
return 'Koha::Biblio::Metadata';
}
1;

View File

@ -92,8 +92,9 @@ sub next {
my $marc;
my $row = $self->{rs}->next();
return if !$row;
if ( $row->marcxml ) {
$marc = MARC::Record->new_from_xml( $row->marcxml );
my $marcxml = C4::Biblio::GetXmlBiblio( $row->get_column('biblionumber') );
if ( $marcxml ) {
$marc = MARC::Record->new_from_xml( $marcxml );
}
else {
confess "No marcxml column returned in the request.";

View File

@ -43,14 +43,14 @@ sub new {
}
my $max = $repository->{koha_max_count};
my $sql = "
(SELECT biblioitems.biblionumber, biblioitems.timestamp, marcxml
(SELECT biblioitems.biblionumber, biblioitems.timestamp
FROM biblioitems
";
$sql .= " JOIN oai_sets_biblios ON biblioitems.biblionumber = oai_sets_biblios.biblionumber " if defined $set;
$sql .= " WHERE timestamp >= ? AND timestamp <= ? ";
$sql .= " AND oai_sets_biblios.set_id = ? " if defined $set;
$sql .= ") UNION
(SELECT deletedbiblio.biblionumber, null as marcxml, timestamp FROM deletedbiblio";
(SELECT deletedbiblio.biblionumber, timestamp FROM deletedbiblio";
$sql .= " JOIN oai_sets_biblios ON deletedbiblio.biblionumber = oai_sets_biblios.biblionumber " if defined $set;
$sql .= " WHERE DATE(timestamp) >= ? AND DATE(timestamp) <= ? ";
$sql .= " AND oai_sets_biblios.set_id = ? " if defined $set;

View File

@ -36,6 +36,7 @@ use Getopt::Long;
use C4::Biblio;
use C4::Items;
use Koha::Database;
use Koha::Biblio::Metadatas;
my $delete_items;
my $confirm;
@ -68,23 +69,23 @@ This script has the following parameters :
exit();
}
my $schema = Koha::Database->new()->schema();
my @biblioitems = # Should be replaced by a call to C4::Search on zebra index
# Record-status when bug 15537 will be pushed
$schema->resultset('Biblioitem')->search( { marcxml => { LIKE => '%<leader>_____d%' } } );
my @metadatas = # Should be replaced by a call to C4::Search on zebra index
# Record-status when bug 15537 will be pushed
Koha::Biblio::Metadatas->search( { format => 'marcxml', marcflavour => C4::Context->preference('marcflavour'), metadata => { LIKE => '%<leader>_____d%' } } );
my $total_records_count = @biblioitems;
my $total_records_count = @metadatas;
my $deleted_records_count = 0;
my $total_items_count = 0;
my $deleted_items_count = 0;
foreach my $biblioitem (@biblioitems) {
my $biblionumber = $biblioitem->get_column('biblionumber');
foreach my $m (@metadatas) {
my $biblionumber = $m->get_column('biblionumber');
say "RECORD: $biblionumber" if $verbose;
if ($delete_items) {
my $deleted_count = 0;
my $biblioitem = Koha::Biblioitem->find( $biblionumber );
foreach my $item ( $biblioitem->items() ) {
my $itemnumber = $item->itemnumber();

View File

@ -70,8 +70,10 @@ my $mappings = GetOAISetsMappings;
# Get all biblionumbers and marcxml
print "Retrieving biblios... " if $verbose;
my $query = qq{
SELECT biblionumber, marcxml
FROM biblioitems
SELECT biblionumber, metadata
FROM biblio_metadata
WHERE format='marcxml'
AND marcflavour = ?
};
if($length) {
$query .= "LIMIT $length";
@ -80,7 +82,7 @@ if($length) {
}
}
my $sth = $dbh->prepare($query);
$sth->execute;
$sth->execute( C4::Context->preference('marcflavour') );
my $results = $sth->fetchall_arrayref({});
print "done.\n" if $verbose;

View File

@ -90,8 +90,9 @@ foreach my $issue ( @{$issues} ) {
getitemtypeimagelocation( 'opac',
$itemtypes->{ $issue->{$itype_attribute} }->{imageurl} );
}
if ( $issue->{marcxml} ) {
my $marcxml = StripNonXmlChars( $issue->{marcxml} );
my $marcxml = C4::Biblio::GetXmlBiblio( $issue->{biblionumber} );
if ( $marcxml ) {
$marcxml = StripNonXmlChars( $marcxml );
my $marc_rec =
MARC::Record::new_from_xml( $marcxml, 'utf8',
C4::Context->preference('marcflavour') );

View File

@ -595,8 +595,7 @@ if ($tag) {
$query_cgi = "tag=" .$tag . "&" . $query_cgi;
my $taglist = get_tags({term=>$tag, approved=>1});
$results_hashref->{biblioserver}->{hits} = scalar (@$taglist);
my @biblist = (map {GetBiblioData($_->{biblionumber})} @$taglist);
my @marclist = (map { (C4::Context->config('zebra_bib_index_mode') eq 'dom')? $_->{marcxml}: $_->{marc}; } @biblist);
my @marclist = map { C4::Biblio::GetXmlBiblio( $_->{biblionumber} ) } @$taglist;
$DEBUG and printf STDERR "taglist (%s biblionumber)\nmarclist (%s records)\n", scalar(@$taglist), scalar(@marclist);
$results_hashref->{biblioserver}->{RECORDS} = \@marclist;
# FIXME: tag search and standard search should work together, not exclusively

View File

@ -58,7 +58,7 @@ $dbh->do("INSERT INTO biblio (frameworkcode, author, title, datecreated) VALUES
my $biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$bib_title'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$itemtype')");
my $biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")

View File

@ -57,7 +57,7 @@ $dbh->do("INSERT INTO biblio (frameworkcode, author, title, datecreated) VALUES
my $biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$bib_title'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$itemtype')");
my $biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")

View File

@ -72,7 +72,7 @@ $dbh->do("INSERT INTO biblio (frameworkcode, author, title, datecreated) VALUES
my $biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$bib_title'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$right_itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$right_itemtype')");
my $biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")

View File

@ -87,8 +87,8 @@ $dbh->do("INSERT INTO biblio (frameworkcode, author, title, datecreated)
VALUES ('SER', 'Koha test', '$TITLE', '2011-02-01')");
my $biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$TITLE'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype)
VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype)
VALUES ($biblionumber, '$itemtype')");
my $biblioitemnumber = $dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")
or BAIL_OUT("Cannot find newly created biblioitems record");
@ -227,11 +227,9 @@ $biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE tit
$dbh->do(qq{
INSERT INTO biblioitems (
biblionumber,
marcxml,
itemtype
) VALUES (
$biblionumber,
'',
'$itemtype'
)
});
@ -439,7 +437,7 @@ $dbh->do("
$biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$TITLE'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$itemtype')");
$biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")
@ -501,7 +499,7 @@ $dbh->do("
$biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$TITLE'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$itemtype')");
$biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")
@ -542,7 +540,7 @@ $dbh->do("INSERT INTO biblio (frameworkcode, author, title, datecreated) VALUES
$biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$TITLE'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$itemtype')");
$biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")
@ -654,7 +652,7 @@ $dbh->do("INSERT INTO biblio (frameworkcode, author, title, datecreated) VALUES
$biblionumber = $dbh->selectrow_array("SELECT biblionumber FROM biblio WHERE title = '$TITLE'")
or BAIL_OUT("Cannot find newly created biblio record");
$dbh->do("INSERT INTO biblioitems (biblionumber, marcxml, itemtype) VALUES ($biblionumber, '', '$itemtype')");
$dbh->do("INSERT INTO biblioitems (biblionumber, itemtype) VALUES ($biblionumber, '$itemtype')");
$biblioitemnumber =
$dbh->selectrow_array("SELECT biblioitemnumber FROM biblioitems WHERE biblionumber = $biblionumber")

View File

@ -241,14 +241,14 @@ sub construct_objects_needed {
# ---------- 3 biblio items -------------------------
$query = '
INSERT INTO biblioitems
(biblionumber, itemtype, marcxml)
VALUES (?,?,?)';
(biblionumber, itemtype)
VALUES (?,?)';
$insert_sth = $dbh->prepare($query);
$insert_sth->execute( $biblionumber1, 'Book', '' );
$insert_sth->execute( $biblionumber1, 'Book' );
my $biblioitemnumber1 = $dbh->last_insert_id( undef, undef, 'biblioitems', undef );
$insert_sth->execute( $biblionumber2, 'Music', '' );
$insert_sth->execute( $biblionumber2, 'Music' );
my $biblioitemnumber2 = $dbh->last_insert_id( undef, undef, 'biblioitems', undef );
$insert_sth->execute( $biblionumber3, 'Book', '' );
$insert_sth->execute( $biblionumber3, 'Book' );
my $biblioitemnumber3 = $dbh->last_insert_id( undef, undef, 'biblioitems', undef );
# ---------- 3 items -------------------------