add support for parallel downloads

[mmdbalkhi]

• Parallel downloads #57

The number of concurrent downloads can be configured with:

  -p <n> / --parallel-download <n>
  fetch_jobs=<n> in /etc/xbps.conf.d/

---

One open question is the preferred way to present download progress when multiple downloads happen concurrently.

Currently the fetch callback output is serialized with a mutex and
only package completion messages are printed. The detailed per-package
progress output used in the single-download case does not translate
well to parallel downloads.

Possible options could be:

• keep the current simple behavior (print when a package finishes), or
• switch to a single aggregated progress indicator for the wholetransaction.

For example:

...
linux6.6-6.6.127_1.x86_64.xbps: downloaded
Downloading packages: 41% (3/8 active) 512MB / 1.2GB

Feedback on the preferred output style would be appreciated before
finalizing the UI behavior.

---

Note: due to connectivity limitations in Iran, my responses or
updates to this PR may occasionally take a few days.

[Duncaen]

error handling in libfetch is not thread safe.

[mmdbalkhi]

@Duncaen do you mean that concurrent fetches themselves are unsafe or only the error reporting in libfetch?

[Duncaen]

Its using a bunch of global variables, like fetchLastErrCode, which is also uses to propagate the http status code to the caller.

xbps/lib/fetch/fetch.h

Lines 164 to 184 in 7553cb7

	extern auth_t fetchAuthMethod;
	/* Last error code */
	extern int fetchLastErrCode;
	#define MAXERRSTRING 256
	extern char fetchLastErrString[MAXERRSTRING];
	/* I/O timeout */
	extern int fetchTimeout;
	/* Connect timeout */
	extern int fetchConnTimeout;
	/* Connect attempt delay */
	extern int fetchConnDelay;
	/* Restart interrupted syscalls */
	extern volatile int fetchRestartCalls;
	/* Extra verbosity */
	extern int fetchDebug;

.

[Duncaen]

mmdbalkhi | hi there, [#682:xbps] duncaen: do you mean that concurrent fetches themselves are unsafe, or only the error reporting in libfetch, or my implementation?
duncaen | it has a global fetchLastErrCode which is also used to store the http status code
duncaen | when a download finishes you want to look at that code at least in a few cases its important to know the duncaen | so if you have multiple threads they potentially overwrite the status of eachother
duncaen | haven't checked much further whether there are more global variable issues
duncaen | xbps already does use it threaded, in xbps-query -Ro, but nobody should be using taht anyways
mmdbalkhi (~mmdbalkhi@user/mmdbalkhi) has quit (Quit: Client closed)

just in case this was lost to the disconnect.