Commit Graph

12 Commits

Author SHA1 Message Date
Martin Renvoize 4deb199163
Bug 42656: (QA follow-up) Make error status persistence consistent, with a full trace, across FTP, SFTP and Local
Two related inconsistencies remained across the three transport
backends' error-handling paths:

1) Only Koha::File::Transport::SFTP's _abort_operation() persisted an
   "errors" status snapshot to the status column on every failed
   operation. FTP never did, and Local had no _abort_operation() at
   all - so an SFTP transport's displayed status could reflect a
   single transient failure from real usage (e.g. an EDI cron job)
   while the exact same failure on FTP/Local left the stored status
   untouched (stale, possibly still showing the last explicit
   test_connection() result). This is exactly the kind of unclear
   status reporting bug 42656 is about.

2) FTP's _abort_operation() carried its extra detail under an
   operation_params key, while SFTP (and Local's plain add_message
   calls) used path. The change_directory() success payload had the
   same split: FTP used directory, SFTP and Local used path. The
   admin template's change_directory row display depended on FTP's
   mismatched key names, which meant it silently failed to show the
   tested directory for SFTP transports.

Adds a shared _record_error($operation, \%payload) to the
Koha::File::Transport base class - add_message() plus status
persistence - and routes all three backends' _abort_operation() (Local
gains one) through it. Renames FTP's operation_params/directory keys
to path throughout, matching SFTP and Local, and simplifies the
template to read operation.detail.path unconditionally instead of
ORing two differently-named keys.

_record_error() persists the full trace of operations recorded on the
transport object so far this session (every success leading up to the
failure), not just the failing operation in isolation - the same
"full trace" shape TestTransport::process() already builds for an
explicit test_connection() run. A long-lived transport used across a
multi-step operation (e.g. an EDI download loop touching several files
before one fails) will therefore show every successful step leading up
to the failure. It also logs the failure - including the built trace -
via Koha::Logger at warn level, so it's visible in the Koha logs
without needing to query the transport's status column.

Test plan:
1. prove -r t/db_dependent/Koha/File/ t/db_dependent/Koha/BackgroundJob/TestTransport.t t/db_dependent/Koha/Edifact/Transport.t t/db_dependent/api/v1/file_transports.t
   => All tests successful (48 tests total), including:
      - new "_abort_operation persists status consistently" subtests
        in FTP.t, SFTP.t and Local.t that reload the transport from
        the DB after a simulated failure and confirm the status
        column was updated with the correct operation code and path
        detail;
      - a new "_record_error() tests" subtest in Transport.t proving
        the persisted trace includes a prior success message plus the
        failure (not just the failure alone), and that a warn-level
        log line mentioning the operation and error was emitted.
2. koha-qa.pl -v 2 --more-tests
   => [PASS], no critic/pod warnings.

Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
2026-08-06 16:05:21 +00:00
Martin Renvoize 7a750c7e83
Bug 43088: Add a consistent current_directory() accessor to Koha::File::Transport
FTP, Local and SFTP list_files() were unified to return the same flat
shape, but consumers still had no portable way to ask a transport for
its current remote directory - the underlying libraries' own methods
of that name don't agree on what "cwd" means. Net::SFTP::Foreign's
cwd() is a pure read-only accessor, but Net::FTP's cwd() is a mutator:
called with no argument it defaults to "/", issues a real CWD command,
and returns a boolean rather than a path.

The koha-plugin-transport-browser plugin discovered this the hard way:
it read $transport->{connection}->cwd directly to capture the initial
working directory, which happened to work for SFTP but silently reset
the FTP connection to root and captured "1" (a boolean) as the "path"
on every request, breaking directory navigation for FTP transports.

Add a public current_directory() method to the Koha::File::Transport
base class, following the same connect-on-demand pattern as
list_files()/upload_file()/etc, with a _current_directory() hook each
subclass implements using its own real read-only accessor: pwd() for
FTP, cwd() for SFTP, and the tracked {current_directory} (falling back
to download_directory) for Local. Consumers should use this instead of
reaching into transport internals.

Test plan:
1. prove t/db_dependent/Koha/File/Transport.t
        t/db_dependent/Koha/File/Transport/FTP.t
        t/db_dependent/Koha/File/Transport/Local.t
        t/db_dependent/Koha/File/Transport/SFTP.t
   => All tests successful, including the new current_directory()
      tests subtest in each transport test file, which mock the
      underlying libraries (Net::FTP / Net::SFTP::Foreign) to assert
      each backend reads its own real read-only accessor.
2. koha-qa.pl -v 2 --more-tests
   => [PASS] for all four changed modules, no critic/pod warnings.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
2026-08-03 13:00:33 +00:00
Martin Renvoize b75d07acd5
Bug 40811: Implement dual API for file transports with simplified auto-management
This patch implements a dual API design for Koha::File::Transport, providing
both a simplified auto-managing API and a traditional explicit API for maximum
flexibility and ease of use.

DUAL API DESIGN
===============

Simplified API (Recommended):
  - Automatic connection management (no connect/disconnect needed)
  - Flexible per-operation directory control via options hashref
  - Stateless operations safe for concurrent usage

  Example:
    $transport->upload_file($local, $remote, { path => '/custom/' });
    $transport->download_file($remote, $local);  # Uses download_directory
    $transport->list_files({ path => '/incoming/' });

Traditional API (Explicit Control):
  - Manual connection/directory management when needed
  - Stateful directory operations via change_directory()
  - Ideal for multiple operations in the same directory

  Example:
    $transport->change_directory('/work/');
    $transport->upload_file($local, 'file1.txt');
    $transport->upload_file($local, 'file2.txt');
    my $files = $transport->list_files();

The APIs can be mixed - calling change_directory() explicitly switches to
manual mode and disables auto-management for subsequent operations.

STANDARDIZED TEMPLATE METHOD PATTERN
=====================================

All transport methods now follow a consistent pattern with clear separation
between public API (parent class) and protocol implementation (subclasses).

Public methods in Koha::File::Transport:
  - connect()          → Resets directory state, calls _connect()
  - disconnect()       → Resets directory state, calls _disconnect()
  - change_directory() → Sets manual mode flag, calls _change_directory()
  - upload_file()      → Manages connection/directory, calls _upload_file()
  - download_file()    → Manages connection/directory, calls _download_file()
  - list_files()       → Manages connection/directory, calls _list_files()
  - rename_file()      → Ensures connection, calls _rename_file()

Subclass responsibilities (SFTP/FTP/Local):
  - Implement _connect() - Protocol-specific connection logic
  - Implement _disconnect() - Protocol-specific disconnection logic
  - Implement _change_directory() - Protocol-specific directory change
  - Implement _upload_file() - Protocol-specific upload logic
  - Implement _download_file() - Protocol-specific download logic
  - Implement _list_files() - Protocol-specific listing logic
  - Implement _rename_file() - Protocol-specific rename logic
  - Implement _is_connected() - Protocol-specific connection check

Test plan:
  prove t/db_dependent/Koha/File/Transports.t

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-21 15:28:17 -07:00
Martin Renvoize 8acfc28441
Bug 38489: Add disconnect method and proper connection cleanup to file transport system
This patch adds proper connection cleanup functionality to the file
transport system, addressing a gap identified during EDI migration.

Changes:
- Add disconnect() method to base Transport class interface
- Implement SFTP disconnect using connection->disconnect()
- Implement FTP disconnect using connection->quit()
- Add DESTROY methods to ensure connections are cleaned up on object destruction
- Local transport includes no-op disconnect for API consistency

This ensures that FTP/SFTP connections are properly closed, preventing
connection leaks and potential resource exhaustion issues.

Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-14 14:51:47 -06:00
Martin Renvoize 51d5691e35
Bug 38489: Add rename_file method to transport system and Local transport class
This patch enhances the file transport system with file renaming
capabilities and adds a Local transport class for development/testing.

Changes:
- Add rename_file() method to base Transport class and SFTP/FTP subclasses
- Create Koha::File::Transport::Local for local directory operations
- Add 'local' transport type to polymorphic mapping
- Update transport instantiation to handle Local transport
- Add Local transport tests to verify polymorphic behavior
- Enhanced connect() method for Local transport validates directories

The rename_file functionality is needed for EDI file processing workflow
where files are renamed after download to mark them as processed.

Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-14 14:51:47 -06:00
Tomás Cohen Arazi 6b478e0da9
Bug 40163: (follow-up) Fix wrong GNU links and update test
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-24 09:25:04 -06:00
Tomas Cohen Arazi 67a7901663
Bug 39190: Rename API from sftp_servers to file_transports
This patch renames the entire API structure from SFTP-specific to generic
file transport terminology to reflect the polymorphic nature of the system.

API Changes:
- Endpoints: /config/sftp_servers -> /config/file_transports
- Parameters: sftp_server_id -> file_transport_id
- Definitions: sftp_server -> file_transport
- Controller: Config::SFTP::Servers -> Config::File::Transports

File Changes:
- api/v1/swagger/definitions/sftp_server.yaml -> file_transport.yaml
- api/v1/swagger/paths/config_sftp_servers.yaml -> config_file_transports.yaml
- Koha/REST/V1/Config/SFTP/Servers.pm -> Koha/REST/V1/Config/File/Transports.pm
- t/db_dependent/api/v1/sftp_servers.t -> file_transports.t

Updated References:
- swagger.yaml: paths, definitions, parameters, tags
- Template JavaScript: API URLs and variable names
- Controller: package name, variable names, error messages
- Tests: endpoints, parameters, descriptions
- API mapping: to_api_mapping field name correction

To test:
1. Apply this patch
2. Run: yarn api:bundle (to regenerate swagger bundle)
3. Run: prove t/db_dependent/api/v1/file_transports.t
4. All API tests should pass
5. Verify admin interface uses new API endpoints
6. Test CRUD operations work correctly

The API now consistently uses 'file_transports' terminology matching
the polymorphic file transport system that supports FTP, SFTP, and
future transport types.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-19 13:23:25 -06:00
Tomas Cohen Arazi ab18d5a989
Bug 39190: Fix circular dependency causing C3 MRO failures
Resolved circular dependency chain:
Transport.pm → TestTransport.pm → Transports.pm → SFTP.pm → Transport.pm

Changed 'use Koha::BackgroundJob::TestTransport' to runtime 'require'
in the _post_store_trigger method to break the dependency cycle.

This fixes the C3 method resolution order failures in
Koha::File::Transport::FTP and Koha::File::Transport::SFTP.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-19 13:23:23 -06:00
Tomas Cohen Arazi 1286c924b0
Bug 39190: Adjust schema to new structure
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-19 13:23:22 -06:00
Martin Renvoize 3fef445515
Bug 39190: Re-implement the connection testing
We now use background jobs triggered by Transport->store to update the
connection status.

Status is now a json blob with more details, so we update the api
definition to reflect that.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-19 13:23:17 -06:00
Martin Renvoize 277a02daeb
Bug 39190: Add key handling to SFTP Transports
We add triggers to store in the base class and then use them in the SFTP
transport class to write out the key file to the filesystem for later
use in connect.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-19 13:23:15 -06:00
Martin Renvoize 3354bed6ef
Bug 39190: Rework (S)FTP into File::Transport classes
This patch adds polymorphic transport classes to Koha for FTP and SFTP
file transport configurations.

We introduce a base class for interacting with the base configuration
storage and subclasses to impliment the protocol specific transport
handling.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-19 13:23:14 -06:00