Age | Commit message (Collapse) | Author |
|
|
|
|
|
|
|
Use 'rayon' to parallelise the repository processing. Each repository is
processed in a thread in the default 'rayon' pool.
In order to get thread-safe access to the database, I followed some
advice from a Stack Overflow answer by VasiliNovikov
(https://stackoverflow.com/users/1091436/vasilinovikov):
https://stackoverflow.com/questions/62560396/how-to-use-sqlite-via-rusqlite-from-multiple-threads/62560397#62560397
VasiliNovikov recommended creating a database connection pool using
'r2d2_sqlite'. This way we don't have to share a database connection
between threads, but each thread can have its own connection.
This also means we can remove mutable requirements in a bunch of places
involving our `database::Db` type since we're no longer managing the
database connections directly.
|
|
Remove all async from the project by switching from 'reqwest' to 'ureq'.
This should make the code simpler, and hopefully enable us to try out
multithreading.
|
|
Trying to get rid of async. This compiles, but fails with the following
runtime error:
Error code 21: Library used incorrectly
Need to investigate further.
|
|
Use the Tokio runtime we created to run the blocking async tasks.
Trying to set this up so I can get results back from the spawned tasks,
but I'm currently having trouble working out how to extract them from
the async task and return them from `run()`. I suppose I could just
print out the errors directly in that `while let` loop, but ideally I'd
like to return all errors from `run()` rather than printing in `run()`.
|
|
Allows a maximum repo size to be given as a command line argument.
Repos larger than this will not be mirrored. This gives us a way to save
server space by avoiding gigantic repositories.
|
|
I think, at least. Took a lot of research and trial and error to get
this to compile, working out how to set up the multi-threading for async
code. The idea here is to be able to process each repo in potentially
multiple threads and do that processing work in parallel.
|
|
Define the options we want to take. Not using them yet.
|
|
CGit reads the repository modification time from the following
locations, in order from top to bottom:
1. agefile
2. repo.git/refs/heads/{default_branch | "master"}
3. repo.git/packed-refs
(https://git.zx2c4.com/cgit/tree/ui-repolist.c?id=bd6f5683f6cde4212364354b3139c1d521f40f39#n35)
Update the `/refs/heads/{default_branch}` file mtime when cloning and
updating the repo to match the GitHub `updated_at` time.
This ensures that when mirroring old repositories, they don't appear at
the top of the CGit repository index list when sorting by age.
|
|
|
|
Get rid of the boxed errors to make matching on errors easier.
|
|
|
|
Based on the guide here:
https://libgit2.org/docs/guides/101-samples/#repositories_clone_mirror
References:
https://github.com/libgit2/libgit2sharp/issues/577
https://github.com/libgit2/libgit2.github.io/pull/31
|
|
|
|
|
|
|