| Age | Commit message (Collapse) | Author |
|---|
|
|
|
I artificially limited the number of repositories processed to two for
testing so that I wouldn't download an mirror all of my repositories
while testing the program. Now that things seem to be working, remove
this artificial limit.
|
|
|
|
Rejigger some types and signatures to allow us to get references to the
`base_cgitrc` path instead of copying it for each repository.
|
|
Remove the `Iterator` test implementations that didn't work out.
|
|
Prefix each error line with the text "error: " to make it clear that's
what it is, and that it's separate from errors printed on other lines.
Worked out how to set up an `Iterator` for `MultiError` based on a
comment by 'chris-morgan' (https://old.reddit.com/user/chris-morgan) on
Reddit /r/rust:
> 1. Implement your own iterator type which wraps existing iterator
> types (std::slice::Iter, and std::vec::IntoIter if you want a
> consuming iterator).
> Advantages: most flexible, ensures API stability if you
> need to change internal details.
> Disadvantages: a lot more effort, if you want to do it properly
> (which involves implementing about ten traits on your iterator
> wrapper type); and if slices or their iterators add something new,
> you don’t get it unless you implement a wrapper yourself.
>
> 2. Have your iter() functions and IntoIterator implementations use the
> standard iterator types directly.
> Advantages: easier, gets you all the other trait implementations on
> std::slice::Iter for free—AsRef, Clone, FusedIterator,
> ExactSizeIterator, Debug, Send, DoubleEndedIterator, TrustedLen,
> Sync).
> Disadvantages: if you need to restructure things so that this is no
> longer an option (e.g. store things in a different type of vector
> and thus need to map it before presenting it to the user) it’s a
> breaking change.
>
> 3. Implement Deref<Target = [(K, V)]> and just treat your Bucket<K, V>
> as a &[(K, V)]. (Read-only; implement DerefMut if you want to allow
> mutations of values.)
> Advantages: easy, and lets you simply treat the whole thing as a
> slice (this is what Vec<T> does).
> Disadvantages: there really aren’t any, if it matches your purpose.
> (If not, it’s useless.)
(https://old.reddit.com/r/rust/comments/7a0slp/questionimplementing_iterator_for_a_struct_with_a/)
|
|
|
|
|
|
|
|
Decided to use 'anyhow' errors instead of a generic boxed error.
|
|
So you know what the error referred to.
|
|
|
|
Make all chained methods indented.
|
|
Return all errors from repo processing. This allows us to provide
information on all errors that happened while processing, but continue
processing all the repos even if there's an error in one of them.
A new `MultiError` type wraps a list of errors to do this.
|
|
Not sure when or why I added this.
|
|
Remove my old tests now that we have a multi-threading setup that
actually works.
|
|
Use 'rayon' to parallelise the repository processing. Each repository is
processed in a thread in the default 'rayon' pool.
In order to get thread-safe access to the database, I followed some
advice from a Stack Overflow answer by VasiliNovikov
(https://stackoverflow.com/users/1091436/vasilinovikov):
https://stackoverflow.com/questions/62560396/how-to-use-sqlite-via-rusqlite-from-multiple-threads/62560397#62560397
VasiliNovikov recommended creating a database connection pool using
'r2d2_sqlite'. This way we don't have to share a database connection
between threads, but each thread can have its own connection.
This also means we can remove mutable requirements in a bunch of places
involving our `database::Db` type since we're no longer managing the
database connections directly.
|
|
Remove all async from the project by switching from 'reqwest' to 'ureq'.
This should make the code simpler, and hopefully enable us to try out
multithreading.
|
|
Turns out I need to specify all the flag I want in the open call,
including the one to open the database for reading and writing. This
fixes the "Error code 21: Library used incorrectly" error I was getting
earlier.
|
|
To allow us to work out where the error is coming from.
|
|
|
|
Remove all the async database calls and Tokio spawning. Still haven't
worked out the error code 21 database error from earlier, but this will
hopefully allow us to use normal threads directly.
|
|
Trying to get rid of async. This compiles, but fails with the following
runtime error:
Error code 21: Library used incorrectly
Need to investigate further.
|
|
Still isn't multi-threaded. Not sure what I'm doing wrong.
|
|
|
|
|
|
|
|
Looks like I didn't delete the file in
67d7632b900f7221c1a3fb1927cd97b7cb60c71e.
|
|
|
|
Looks like the work doesn't happen on multiple threads. All of the tasks
printed the same thread ID. Need to do some more work to get this
working properly, it seems.
|
|
Collect all errors into a list. I think I'm going to return them as a
list from this function.
The runtime appears a lot slower with this change. Need to figure out
what that's about.
|
|
Use the Tokio runtime we created to run the blocking async tasks.
Trying to set this up so I can get results back from the spawned tasks,
but I'm currently having trouble working out how to extract them from
the async task and return them from `run()`. I suppose I could just
print out the errors directly in that `while let` loop, but ideally I'd
like to return all errors from `run()` rather than printing in `run()`.
|
|
To separate the actions more.
|
|
|
|
Remove the hard-coded test repositories I was using and replace them
with real ones retrieved from the GitHub API.
Enable I/O and timers on the Tokio runtime in order to enable the async
GitHub API request.
|
|
Don't panic here so we can use our own error message template.
|
|
Print the error instead of unwrapping.
|
|
|
|
Don't see any reason to do this now.
|
|
|
|
|
|
Allows a maximum repo size to be given as a command line argument.
Repos larger than this will not be mirrored. This gives us a way to save
server space by avoiding gigantic repositories.
|
|
Remove this now that we have something that I think works.
|
|
Rather that relying on the Cargo features we've enabled to define this,
create a multi-threaded runtime in code.
|
|
I think, at least. Took a lot of research and trial and error to get
this to compile, working out how to set up the multi-threading for async
code. The idea here is to be able to process each repo in potentially
multiple threads and do that processing work in parallel.
|
|
|
|
|
|
|
|
|
|
This is no longer relevant.
|
