| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Explain why we need to skip and un-skip the callback.
Refs #6095
|
|
Since this method was run on `after_save` and in it
`RouteWayCostCalculator` calls `#update` on the given route, it caused
an infinite recursive call to the worker. To prevent this, wrap the
`#calculate!` call in methods that unset and reset the callback.
Refs #6095
|
|
Just running `Chouette::Route.find(id)` doesn't give us anything if we
don't have a `Referential` selected. In order to properly get the
correct route, pass the referential ID to the worker and switch to that
referential before trying to select the route.
Refs #6095
|
|
This new method will launch a worker that takes the route's `StopArea`s,
converts them to `WayCost`s, sends them to the TomTom API to calculate
distance and time costs, and saves those costs to the route's `costs`
field.
Refs #6095
|
|
Also added a *dumb* `WorkgroupExport` for testing purposes.
|
|
|
|
|
|
I was getting this error when I removed the `rescue nil`s:
Failures:
1) WorkbenchImportWorker foreign lines uploads the following entries: ["OFFRE_TRANSDEV_20170301122517", "OFFRE_TRANSDEV_20170301122519"]
Failure/Error: eg_file.unlink
NoMethodError:
undefined method `unlink' for #<File:0x007fef1adbace8>
# ./app/workers/workbench_import_worker.rb:81:in `upload_entry_group_stream'
# ./app/workers/workbench_import_worker.rb:52:in `upload_entry_group'
# ./app/services/zip_service.rb:61:in `<<'
# ./app/services/zip_service.rb:61:in `finish_current_output'
# ./app/services/zip_service.rb:35:in `add_entry'
# ./app/services/zip_service.rb:27:in `block in _subdirs'
# .../.gem/ruby/2.3.3/gems/rubyzip-1.2.1/lib/zip/entry_set.rb:38:in `block in each'
# .../.gem/ruby/2.3.3/gems/rubyzip-1.2.1/lib/zip/entry_set.rb:37:in `each'
# .../.gem/ruby/2.3.3/gems/rubyzip-1.2.1/lib/zip/entry_set.rb:37:in `each'
# .../.gem/ruby/2.3.3/gems/rubyzip-1.2.1/lib/zip/central_directory.rb:182:in `each'
# ./app/services/zip_service.rb:26:in `_subdirs'
# .../.gem/ruby/2.3.3/gems/rubyzip-1.2.1/lib/zip/file.rb:133:in `open_buffer'
# ./app/services/zip_service.rb:21:in `block in subdirs'
# ./app/workers/workbench_import_worker.rb:39:in `each'
# ./app/workers/workbench_import_worker.rb:39:in `each_with_index'
# ./app/workers/workbench_import_worker.rb:39:in `upload'
# ./app/workers/workbench_import_worker.rb:19:in `perform'
# ./spec/workers/workbench_import_worker_spec.rb:68:in `block (4 levels) in <top (required)>'
# ./spec/workers/workbench_import_worker_spec.rb:68:in `block (3 levels) in <top (required)>'
# ./spec/workers/workbench_import_worker_spec.rb:13:in `instance_eval'
# ./spec/workers/workbench_import_worker_spec.rb:13:in `block in expect_upload_with'
# .../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:268:in `load'
# .../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:268:in `block in load'
# .../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
# .../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:268:in `load'
# .../.gem/ruby/2.3.3/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'
# -e:1:in `<main>'
Johan said that in his tests on his local machine, the temp files
weren't getting deleted. When I try running the specs, the temp files do
get deleted, so not sure what was going on. Figure it may be because of
the `rescue nil`s, so I decided to remove those in the hopes of always
ensuring we close and unlink the temp files.
The trouble with doing that, however, was that it caused the error and
thus test failures like the example above. I managed to fix the problem
by using `File.unlink` instead of the instance method.
Hopefully this works to ensure that the temp files are always deleted.
Refs #4315
|
|
If the directory doesn't exist, we should create it to ensure we don't
get errors trying to write to a location that doesn't exist. I had
removed this because I assumed the dir would be created during
deployment, but I guess I shouldn't assume. Also, we want to ensure the
dir gets created on dev machines without errors.
This line is updated from last time where we'd create
`Rails.root.join('tmp', 'imports')`. Now we use our `#temp_directory`,
which can be variable.
Refs #4315
|
|
Instead of creating our own files for these subdirectory Zip files, use
temp files.
There are two advantages to doing this:
1. We can customise the location they get saved to. This means we can
use `Rails.root` as a default, but specify a custom location in a
Rails config variable.
2. The files will be automatically deleted on every garbage collection.
In actuality, #2 above was already handled by the `eg_file.unlink` line,
and #1 could be easily added to the existing code.
Using a `Tempfile`, though, makes the fact that these are temporary
files more explicit.
The reason why we want to be able to customise the location of these
tempfiles is that in our production environment set up by Puppet, the
`#{Rails.root}/tmp/imports` directory isn't writable by the application,
which causes this code to error. We thus want a way to change the path
in different environments.
We keep the `eg_file.unlink` call here because we want the temporary
file to be deleted as soon as possible. The Ruby documentation even
recommends it
(https://ruby-doc.org/stdlib-2.5.0/libdoc/tempfile/rdoc/Tempfile.html#class-Tempfile-label-Explicit+close):
> it's good practice to do so: not explicitly deleting unused Tempfiles
> can potentially leave behind large amounts of tempfiles on
> the filesystem until they're garbage collected.
I got rid of the `#upload_entry_group_tmpfile` method in this change,
moving its contents to `#upload_entry_group_stream`. This was easy
enough to do since it was only called in that one place. The reason I
did this is because it took an open `File` object as an argument and
closed the file at the end of the method. That's a major side effect. I
don't think the method should be closing a file it doesn't own. After
moving this work to `#upload_entry_group_stream`, we can close and
unlink the file in the same place where it was created.
Refs #4315
|
|
Now that Sidekiq retries are deactivated
(3487ce6f34ef187a2a8daf170440f51f659c46b7), we need to enable retry on
the workers that need it. We do want retry on the sync workers, so
enable these locally in the worker classes.
Refs #5495
|
|
As of 3487ce6f34ef187a2a8daf170440f51f659c46b7, we now default to
`retry: false` on all our Sidekiq workers. Setting it in individual
workers is now repetitious.
Refs #5495
|
|
usage. Refs #5283
|
|
|
|
af83/5281-workbench-import-structural-bug-reading-zip
5281 workbench import structural bug reading zip
|
|
5006 wb import filter refs with foreign lines
|
|
- CR change from #5006, severity of zip file messages :warning -> :error
- View specs for showing import_messages
|
|
|
|
|
|
|
|
Implementation of Filtering Out Referentials with foreign lines in `app/workers/workbench_import_worker.rb`
- Using ZipService with the set of allowed lines of the organisation
- Fixing Integration Specs of the worker
- Refactoring, Debugging and Rebasing
|
|
ReferentialCloningWorker#perform. Refs #5283
|
|
|
|
af83/4782-create-worker-for-compliance-check-copy--rb201710301748
4782 create worker for compliance check copy rb201710301748
|
|
|
|
|
|
param name
|
|
Previously I had been passing the `ComplianceControlSet` ID to the Java
API. The API actually expects the ID of the `ComplianceCheckSet`.
In order to get this ID, have the `ComplianceControlSetCopier` return
the `ComplianceCheckSet` that it creates, and use that to pass the the
ID to the Java API.
Refs #4782
|
|
Call the Java API URL to launch a validation after having copied the
`ComplianceControlSet` into a `ComplianceCheckSet`.
Copied the format with error logging from
`NetexImport#launch_java_import`.
In the test, make the ID variables global by putting them in `let`s.
Add `config.iev_url` to the Rails `test` environment so that we can
access this from our tests.
TODO: I just realised I'm sending the wrong ID. I should be sending the
check set ID.
Refs #4782
|
|
A new worker that launches the work to copy a `ComplianceControlSet` to
a `ComplianceCheckSet`.
Currently only handles a single control set and referential pair.
Probably want to extend this to handle multiple lists of these objects.
Also need to add in a call to the Java API to validate the check.
Refs #4782
|
|
|
|
|
|
|
|
spurious directories
- Speced and implemented
Missing:
- Exact message parameters (change key from xxx, how to add names of spurious directories)
|
|
- Discussing Spec
- Speced and implemented the corrupt zipfile part
Next:
- Spec incorrect directory structure part
- Implement incorrect directory structure part
|
|
|
|
|
|
files. Refs #4259
|
|
The new form is `#update` and the specs (broken as they are) mock
`#update`, so this call should do the same.
|
|
Refs #4293
|
|
|
|
|
|
|
|
|
|
This param isn't permitted by the controller and we're currently not
using it for anything. It produces a warning message in the logs, so I
figured we should remove it. We may want to change the architecture to
create referentials outside of the API, and thus would have to add this
param in, but until then it's not needed.
|
|
|
|
Retry removed (but not the deadcode yet) [amend me]
|
|
Debugging Java Integration (ZipService -> HTTPService -> Object Creation)
- Created an ErrorFormat concern to see what went wrong in the HTTP call
as so many things can go wrong due to:
* Timeout Issues
* Illegal database setup or cleaning before tests
|
|
- Replzed Lazy Enum over get_next_entry and group by with explicit looping and yielding to an instance level enumerator
that yields streams per subdir (wich represents a referential).
- Fixtures and Specs that use a REAL usecase.
Next: Get rid of metadata kludge (should fix failing specs)
|
