Commit Graph

22 Commits

Author SHA1 Message Date
Jonathan Druart a8526d4419
Bug 41274: Remove exceptions from files listed in the result repo
IF we have pushed some files that should have been excluded from the
tests but have been processed previously and failed, we should skip it
the next build anyway

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-21 08:41:48 -07:00
Jonathan Druart a0f768f784
Bug 41274: Remove the exceptions from git range
When listing the files from a git range we want to remove the exception
files from that list

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-21 08:41:48 -07:00
Jonathan Druart 5253306187
Bug 41274: Remove range parameter from ls_perl_files
It's not used.

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-21 08:41:40 -07:00
Jonathan Druart 647b28e536
Bug 40292: Use Koha::Devel::Files to list dbrevs
For reusability and to limit files to the git index

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-31 15:26:20 -06:00
Jonathan Druart d5da194323
Bug 39877: Force commit id length to 10
The size of the commit id can vary, depending on the git repo.
For instance Jenkins had stored results using length of 9 (e9a07fe6a)
because it's a shallow clone.

But 9 is not enough for our repo:
% git rev-list --all | sort | uniq -w 9 -D
29dfcf49f28bb8e75ab4b71a2d68f194928d3de9
29dfcf49f6

We need to force the length for both commands we use, so commits matched
during the comparison.

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-17 06:51:18 -06:00
Jonathan Druart e9a07fe6ac
Bug 39877: Fix license
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 09:35:13 -06:00
Tomás Cohen Arazi bf6e766f0c
Bug 39877: Tidy POD
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:20 -06:00
Jonathan Druart b2970f874e
Bug 39877: Use OS codename to separate OSes
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:20 -06:00
Jonathan Druart 873b6638f0
Bug 39877: Do not clone if env var is not set
If we don't want the incremental run we should return early in ->new to
prevent the git clone.

Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:19 -06:00
Jonathan Druart 03e52ef3e7
Bug 39877: Prevent files to be processed twice
If the file had a failure and still has, we should not list it twice.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:19 -06:00
Jonathan Druart 7f93977cd7
Bug 39877: Fix confusion with URL prefix
git clone needs https://, but we need to remove it to build the push url

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:18 -06:00
Jonathan Druart cc58dabe3d
Bug 39877: Apply to other slow tests
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:18 -06:00
Jonathan Druart 4971ed8d68
Bug 39877: Remove files that have been deleted
Ignore deleted or renamed files

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:17 -06:00
Jonathan Druart 9f32eea89f
Bug 39877: Incremental test runs
The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab (keep the default readme)
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-10 06:20:16 -06:00
Tomás Cohen Arazi 406e2534ae
Bug 40872: List opac/unapi as a Perl file
This patch makes `tidy.pl` treat the file as a Perl one.

To test:

1. Run:
   $ ktd --shell
  k$ misc/devel/tidy.pl opac/unapi
=> FAIL: You get:

```
Cannot guess filetype for opac/unapi at /kohadevbox/koha/Koha/Devel/Files.pm line 241.
```
2. Apply this patch
3. Repeat 1
=> SUCCESS: The file is now tidy.
4. Sign off :-D

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-24 09:25:05 -06:00
Jonathan Druart cc72eabc5f
Bug 40858: Search merge conflict on files part of the git index
Same as bug 34303 and bug 33743, our test suite must parse/process files
that are part of the index, not all the other files.

It's currently failing on kohadev-db-request.txt that is owned by root.
But it's not part of the Koha codebase, this file should not be tested.

There were also node_modules and po files that were processed.

Test plan:
  prove t/00-merge-conflict-markers.t
must now return gren

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-24 09:25:03 -06:00
Jonathan Druart 36bd4bd7b4
Bug 40858: Rename context 'all' with 'core'
All was not returning all the files present in the git index, but all
Koha core files. Let's rename it with 'core' to remove ambiguity.

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-24 09:24:55 -06:00
Jonathan Druart e9eb20249e
Bug 40541: Add missing new line at the end of files when missing
We have insert_final_newline = true to ensure files will contain a new
line at the end. However we have seen commits with "No new line" added
in the git log.

We should have a xt test to prevent more occurrences.

We should also have a test at the QA script level:
https://gitlab.com/koha-community/qa-test-tools/-/issues/96

Test plan:
  prove xt/find-missing-new-lines-at-the-end-of-file.t
should return green.

Remove the new line at the end of a file, eg.
vim -b mainpage.pl
:set noeol
:wq

Now
  prove xt/find-missing-new-lines-at-the-end-of-file.t
is failing on mainpage.pl.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-31 12:42:09 -06:00
Tomas Cohen Arazi 85b65afd1d
Bug 39876: Add build-git-snapshot type resolution
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-08 15:42:01 -06:00
Jonathan Druart 48263039af
Bug 39876: Apply module to other tests
We are ready to reuse Koha::Devel::Files in other places where
`git ls-files` is used.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-08 15:41:59 -06:00
Jonathan Druart e7e42c5e88
Bug 39876: Add context for tidy
With this patch we adjust what's needed to make misc/devel/tidy.pl run
identically as before

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-08 15:41:59 -06:00
Jonathan Druart 1d4071298c
Bug 39876: Move `git ls-files` to Koha::Devel::Files
This patch introduces a new Koha::Devel namespace. It clearly shows
that its purpose is for development and will not contain any business
logic.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-08 15:41:58 -06:00