Commit Graph

85 Commits

Author SHA1 Message Date
Adriaan de Groot
7e4cb28c1c [unpackfs] Refactor mounting for an entry
- The entry knows where it should be mounted, and can remember that
- mount_entry() didn't use self, so made no sense as a method
  of the Operation class
2020-04-04 23:19:08 +02:00
Adriaan de Groot
49a584377e [unpackfs] Refactor counting an entry
- This does not have to live in the Operation
- Uses only the local imgmountdir (where the entry is mounted)
2020-04-04 23:10:22 +02:00
Adriaan de Groot
a78556b56a [unpackfs] Revert text mode
- Follow original patch from Gabriel Craciunescu: just drop
  the *bufsize* parameter and stick to binary reads.

Text mode was associated in my testing with multiple hangs,
which didn't show up during binary-reads.
2020-02-21 16:37:58 +01:00
Adriaan de Groot
248776144e [unpackfs] Now it's text, it's not encoded anymore 2020-02-21 12:39:26 +01:00
Adriaan de Groot
6d0f5e4111 Merge branch 'kpmcore-manager' 2020-02-20 22:03:45 +01:00
Adriaan de Groot
327c1c0e7b [unpackfs] Reading from the pipe is line-oriented
- Using bufsize=1 causes warnings, since we never bothered
  to explicitly set text mode.
2020-02-20 21:40:33 +01:00
Adriaan de Groot
7c7af28a0d [unpackfs] Make clear that the FS is unsupported by *this* kernel
Patch from Gabriel Craciunescu. FS support is determined be
the kernel, not by Calamares, so make that clearer in the message.
2020-02-20 21:17:33 +01:00
Adriaan de Groot
2bc296b468 [unpackfs] Fix typo's introduced in june 2020-02-13 16:23:49 +01:00
Adriaan de Groot
94083edf9c Merge branch 'master' into kpmcore-manager 2020-01-29 12:55:46 +01:00
Adriaan de Groot
10d1c4cf5b [unpackfs] Improve progress reporting
- don't rely on exactly 100 files being copied (thanks to Kevin Kofler)
2020-01-20 19:39:05 +01:00
Adriaan de Groot
f6d89354a1 Merge branch 'master' into kpmcore-manager 2020-01-10 14:54:13 +01:00
Adriaan de Groot
4370669fc3 [unpackfs] Put exclusion settings into the rsync command line 2019-10-11 17:46:01 +02:00
Adriaan de Groot
9d47716e5d [unpackfs] Add settings per-item for exclude and excludeFile
- support excluding specific files / lists when rsyncing, in case
   the squashfs has more in it than you really want to copy over.
2019-10-11 17:19:56 +02:00
Adriaan de Groot
4552631086 [unpackfs] Prep for more information in file_copy
- Give the whole entry to file_copy, not just the
   destination. This will allow file_copy to work
   with local excludes.
 - Pluck entry.destination out immediately, to keep
   code changes minimal.
 - Document the parameters.
2019-10-11 16:56:04 +02:00
Adriaan de Groot
311c65737f [unpackfs] Remove unused parameter
- list_excludes() turns the extra mounts from global storage
   into --exclude parameters for rsync; it doesn't do anything
   with the destination parameter.
 - while here rename to something more descriptive
2019-10-11 16:47:54 +02:00
Adriaan de Groot
fe1ddb8ead [unpackfs] Expand on the documentation 2019-10-11 16:32:39 +02:00
Adriaan de Groot
57c0b2088c [unpackfs] Allow naming a non-directory for single files
- this enables renaming files on copy
2019-10-11 15:48:17 +02:00
Adriaan de Groot
4acf63fd9a [unpackfs] Document fields of the UnpackEntry
- Document fields
 - While here, remove code for special-casing single-file copies.
   Those already work, based on rsync.
2019-10-08 17:31:51 +02:00
Adriaan de Groot
05c0398126 [unpackfs] Support multiple items better
- it's ok if item one creates directories where item two will write,
   so don't check for existence of all directories on start-up.

Reported by ArcoLinux.
2019-10-08 16:19:39 +02:00
Adriaan de Groot
d58fdaa2cd [unpackfs] Stub of special handling for a single file 2019-10-08 15:00:25 +02:00
Adriaan de Groot
95f725831f [unpackfs] Shuffle the documentation
- move all the examples to an EXAMPLES section
 - leave one single-file example enabled in the configuration
2019-09-30 17:35:31 +02:00
Adriaan de Groot
6fb2563c75 [unpackfs] Copy files when sourcefs = "file"
- Just use the existing rsync code, which can do both
   files and directory trees.
 - The existing code assumed we were always copying directories.
   Now double-check beforehand.
2019-09-30 17:14:23 +02:00
Adriaan de Groot
27cdaba8b2 [unpackfs] Special-case file sources 2019-09-30 17:14:23 +02:00
Adriaan de Groot
7856c1a6a8 [unpackfs] Do more checks before doing any work 2019-09-30 17:14:23 +02:00
Adriaan de Groot
784bc5b86e [unpackfs] Expand documentation
- comment out the squashfs example, since it's not readily
   available even on developer systems
 - add entries for the upcoming "file" type.
2019-09-30 17:14:23 +02:00
Arnaud Ferraris
0f6b2dbe2d [unpackfs] Enable to use "/" as a source
If we don't have/need an image for the rootfs, we might want to
configure the `/` directory as a source for unpackfs. Unfortunately,
this raises an error:
  - unpackfs first creates a temporary directory
  - it then creates a subdirectory for each source, using the source
path's basename
  - when the source is `/`, the basename is an empty string, therefore
the module tries to create an already existing directory

In order to prevent this error, we use the `os.makedirs` function with
parameter `exist_ok=True` instead of `os.mkdir`.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>
2019-08-23 17:26:45 +02:00
Arnaud Rebillout
2ab45cf4f0 Make sure that variable num_files_total_local is declared
This variable is declared in `if m:`. Of course if this codepath doesn't
run, the variable is not declared an Python doesn't like this kind of
surprise...

Signed-off-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>
2019-08-12 15:37:02 +02:00
Adriaan de Groot
b0bcdba867 [unpackfs] Needs mount before it
- The mount module must happen before unpackfs because that (mount)
   module sets up the root mount point (in /tmp) and some other
   variables needed later.
2019-08-01 12:37:03 +02:00
Adriaan de Groot
c14239ca30 [unpackfs] Still raise if mount fails 2019-06-20 16:58:27 +02:00
Adriaan de Groot
1b91f831bf [unpackfs] Use the Python mount-API 2019-06-20 16:14:30 +02:00
Kevin Kofler
71d991e2e4 [unpackfs] Do not use -o loop on a device
src/modules/unpackfs/main.py (UnpackOperation.mount_image): Check
whether entry.source is a regular file or a device and only use
`-o loop` on regular files, not devices.

At least on Fedora >= 29, `-o loop` fails on the read-only device
`/dev/mapper/live-base` (though `-o loop,ro` would be accepted).
2019-05-06 15:04:58 +02:00
Adriaan de Groot
fb412c177c [unpackfs] Improve human-readable name 2019-03-29 17:32:52 -04:00
Adriaan de Groot
5c4f2423f8 [unpackfs] Fix error in error-handling (warn -> warning) 2019-03-29 17:27:26 -04:00
Adriaan de Groot
d87badbf45 [unpackfs] Add test with too-small destination FS 2019-03-29 17:22:47 -04:00
Adriaan de Groot
2a6bf50621 [unpackfs] Don't let ir-phase reduce total number of files 2019-03-29 16:53:02 -04:00
Adriaan de Groot
fae0b8c2f8 [unpackfs] Re-jig progress reporting
- rsync reports its own progress, and reports on files that
   find -type f doesn't. This meant that the numbers didn't
   match what was stored in entry.total
 - The ir-phase adds files to be handled; to-phase happens once
   ir-phase is over and the remaining files are processed.
   By adding the to-phase files, percentages over 100% were
   reported (in part because the number of files doesn't match).
 - Update expected entries total from rsync output.
 - Re-jig computation of how done everything is: tally it
   up in integers, and do only one global progress percentage.
2019-03-29 16:48:46 -04:00
Adriaan de Groot
6d85fd3586 [unpackfs] One last progress call afterwards 2019-03-29 16:24:22 -04:00
Adriaan de Groot
9ce34782ee [unpackfs] Avoid double / at end 2019-03-29 16:09:11 -04:00
Adriaan de Groot
dc2fafe324 [unpackfs] to-chk is also progress information
- ir-chk happens first, and then there's a phase with to-chk
   messages; use those as well.
2019-03-29 12:51:00 -04:00
Adriaan de Groot
38821f47ba [unpackfs] Make comment match code
- The mismatch between "ir-chk" and the comment "to-check" led me
   to check (ha!) the output of rsync, and it outputs "to-chk"
   during small transfers; make sure the comment reflects what
   is actually being used to track progress (which is "ir-chk").
2019-03-29 12:39:45 -04:00
Adriaan de Groot
b3a7545217 [unpackfs] One more test case 2019-01-25 11:43:33 -05:00
Adriaan de Groot
d95193f646 [unpackfs] Translate and log other error messages 2019-01-25 08:02:56 -05:00
Adriaan de Groot
79ce2c2d0f [unpackfs] Translate pretty module name 2019-01-25 07:54:51 -05:00
Adriaan de Groot
28efe2b85a [unpackfs] Log messages in English, too
- For debugging purposes, need to log the message in English before
   returning the (user-visible) error description.
2019-01-25 07:53:39 -05:00
Adriaan de Groot
8ddf801bfb [unpackfs] Switch on translations for user-visible strings 2019-01-25 07:46:19 -05:00
Adriaan de Groot
18f172e80c [unpackfs] Add tests
- Add global- and job-configurations for test runs.
 - Add a driver script that sets up some assumptions on the host
   system so that the tests can complete.
 - The idea is that these tests together get a decent code-coverage
   for the module.
2019-01-25 07:20:16 -05:00
Adriaan de Groot
2d684bb385 [unpackfs] Be more resilient to globalStorage weirdness
- If no extraMount is defined, just don't mount anything
2019-01-25 07:14:23 -05:00
Adriaan de Groot
e96a29b1b9 [unpackfs] Improve error message wording 2019-01-25 06:08:56 -05:00
Adriaan de Groot
56d05d5834 [unpackfs] Reduce file-reading
- Only need to get the list of supported filesystems *once*,
   not for each and every filesystem that is going to be unpacked.
 - Be more Python-idiomatic.
2019-01-25 06:01:49 -05:00
Alf Gaida
784bbd3bc8 finally !must! never fail unhandled
but it does without help
2017-12-26 22:25:40 +01:00