| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
And fix logs
|
|
6996 - Dockerfile for build
|
|
af83/6884-tomtom-matrix--handle-error-when-response-doesn,t-inclu
Handle API error(s) in Tomtom matrix. Fixes #6884
|
|
|
|
|
|
|
|
|
|
- Update the form
- Use policies and update the seeds accordingly
- Add a link in the navbar(s)
|
|
|
|
I previously tried to correct a circular dependency problem in
a057276129b1f62b811743db3b8f867a05241ed3, but that didn't fix it (it was
intermittent, and came back).
After some wrangling, I've now deduced with some confidence that the
problem comes from `RouteWayCostCalculator`, which used
`TomTom::Matrix::RemoteError`. From the way it looks, this seems to mess
up the Rails autoloader since `tom_tom.rb` will try to load the `Matrix`
class from the `TomTom.matrix` call above. Or something.
In an attempt to fix the circular dependency error for real this time,
move the error class to a completely separate module from `Matrix`, and
refer to this when we need to use the error class.
Refs #6884
|
|
We might not always get a nicely formatted JSON
`['error']['description']` response body. Sometimes, like for example
when you use an incorrect API key, even with an 'application/json'
content type, TomTom will respond with:
<h1>Developer Inactive</h1>
What?
In that case, the response has a 403 status. In addition to checking for
an error in the response, should also be checking for the HTTP status
code.
Log the status code in the exception to give us more information about
what went wrong.
Update our existing tests now that `#check_for_error_response` takes a
response object instead of a JSON string.
Refs #6884
|
|
I was getting the following circular dependency error in Sidekiq:
2018-05-03T14:47:17.974Z 19217 TID-owg7qzmqc WARN: RuntimeError: Circular dependency detected while autoloading constant TomTom::Matrix::Point
2018-05-03T14:47:17.974Z 19217 TID-owg7qzmqc WARN: .../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:492:in `load_missing_constant'
.../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:184:in `const_missing'
.../stif-boiv/lib/tom_tom/matrix.rb:44:in `block in points_from_way_costs'
.../stif-boiv/lib/tom_tom/matrix.rb:41:in `each'
.../stif-boiv/lib/tom_tom/matrix.rb:41:in `points_from_way_costs'
.../stif-boiv/lib/tom_tom/matrix.rb:12:in `matrix'
.../stif-boiv/lib/tom_tom.rb:24:in `matrix'
.../stif-boiv/app/services/route_way_cost_calculator.rb:8:in `calculate!'
.../stif-boiv/app/workers/route_way_cost_worker.rb:12:in `perform'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:167:in `execute_job'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:139:in `block (5 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:135:in `block (4 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
.../.gem/ruby/2.3.3/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/sidekiq.rb:33:in `block in call'
.../.gem/ruby/2.3.3/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:369:in `perform_action_with_newrelic_trace'
.../.gem/ruby/2.3.3/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/sidekiq.rb:29:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb:10:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:134:in `block (3 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb:32:in `with_context'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:183:in `stats'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:131:in `block in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:126:in `process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:82:in `process_one'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:70:in `run'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:17:in `watchdog'
../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:26:in `block in safe_thread'
Fix it by moving the error class to a new file.
Refs #6884
|
|
If there's an API error, we shouldn't update the route's `costs` field.
Let's say we've already calculated some costs for a route A. We then edit
and re-save A, which triggers a recalculation of the costs. Now the
TomTom API responds with an error. We don't want to overwrite our
existing costs with an empty array because they could still be useful.
In this case, we should instead keep the existing costs we already had.
To achieve this, move the `RemoteError` rescue into
`RouteWayCostCalculator`, leaving the error unhandled in
`TomTom.matrix`.
Refs #6884
|
|
Occasionally, the following error would appear in our logs:
NoMethodError
RouteWayCostWorker/perform
Error message
NoMethodError: undefined method `each_with_index' for nil:NilClass
Stack trace (show Rails)
/app/lib/tom_tom/matrix.rb: 83:in `extract_costs_to_way_costs!'
/app/lib/tom_tom/matrix.rb: 23:in `matrix'
/app/lib/tom_tom.rb: 24:in `matrix'
/app/app/services/route_way_cost_calculator.rb: 8:in `calculate!'
/app/app/workers/route_way_cost_worker.rb: 12:in `perform'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 167:in `execute_job'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 139:in `block (5 levels) in process'
/var/lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq.rb: 36:in `block in <module:Sidekiq>'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 135:in `block (4 levels) in process'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 128:in `block in invoke'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
…-4.2.10/lib/sidekiq/middleware/server/active_record.rb: 6:in `call'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
…idekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb: 10:in `call'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
…kiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb: 74:in `call'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 133:in `invoke'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 134:in `block (3 levels) in process'
…/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb: 32:in `with_context'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 132:in `block (2 levels) in process'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 183:in `stats'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 131:in `block in process'
/var/lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq.rb: 35:in `block in <module:Sidekiq>'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 126:in `process'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 82:in `process_one'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 70:in `run'
…lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb: 17:in `watchdog'
…lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb: 26:in `block in safe_thread'
My best guess is that this was caused by TomTom responding with an
error, which we weren't handling previously. In that case, the response
would be a JSON string, but include an `'error'` field instead of
`'matrix'` and `'summary'`. Thus, when we'd try to
`matrix_json['matrix']`, it would fail.
Add a new method that checks for errors before we try to parse
`WayCost`s. If a server error is detected, we log the message to the
Rails log and return an empty array.
Refs #6884
|
|
By default the TomTom API uses traffic data to calculate travel time.
This means that previously, getting costs for the exact same route twice
at different times could produce different results.
Johan made a good suggestion to turn off this functionality as it's
apparently enabled by default. This should get us correct times
regardless of time of day since the distances will be effectively the
same.
Refs #6661
|
|
Force-load all models beforehand
|
|
|
|
|
|
6360 Add checks on calendars during WorkbenchImport
|
|
GTFS import (first step)
|
|
|
|
af83/6404-way-costs--snap-distance-and-time-to-1-when-between-0-a
6404: RouteWayCostUnitConverter: Snap values between 0–1 to 1
|
|
|
|
|
|
used. Refs #6047
|
|
We use integer values for distance (kilometres) and time (minutes). If
the values aren't very big, like bus stops inside a city for example,
the integer conversion will put distance and time at `0`, which isn't
correct.
To continue to use our chosen measurement units while still displaying
something that makes sense to users, snap any values >0 and <=1 to `1`.
Refs #6404
|
|
|
|
|
|
Make it clearer why we have to use a custom serialiser here.
Refs #6222
|
|
Rails serialises `BigDecimal`s as JSON strings to prevent loss of
precision. The `latitude` and `longitude` columns in `StopArea` are
stored as `BigDecimal`s. The trouble is that TomTom's API requires the
latitude & longitude values to be JSON floats, not strings.
Make a new JSON serialiser that converts the `BigDecimal` coordinates to
float to allow the values to be correctly interpreted by the API.
Refs #6222
|
|
Already handled this a few commits ago.
Refs #6222
|
|
Using a set ended up not working out. I needed to be able to index into
the list in `#extract_costs_to_way_costs!`, and sets aren't indexable.
This is because they're supposed to be unordered, though modern Ruby
implements `Set` with `Hash` under the hood, which is ordered in Ruby.
I like the idea of having a data structure that automatically eliminates
duplicates, but it wasn't meant to be, because for the extraction to
`WayCost`s, I need an ordered list. Rather than create a new
`OrderedSet` type, I just went the simple route and used an Array,
eliminating the duplicates manually because I know when duplicates are
supposed to occur due to the nature of the data set.
Remove the `#eql?` and `#hash` methods from `TomTom::Matrix::Point`.
Because we're not longer using `Set`, these methods don't need to be
implemented.
Refs #6222
|
|
Change this code to get stop IDs based on the change in
be3e1effcdea87909a181c7e9b12cf6867b1839d. It should use these IDs to
construct new `WayCost`s with the combination of stop IDs from departure
and arrival stops.
This doesn't work currently because the method has code that tries to
index the `points` collection, but it's a `Set`, and sets don't support
indexing and aren't ordered, so this code errors. Need to change the
`Set` to something else that will work here.
Refs #6222
|
|
Rewrite this method to accept `TomTom::Matrix::Point`s instead of plain
`Geokit::LatLng` coordinates. Do this because this method needs to take
the result of `#points_from_way_costs` as input, and that method now
returns a `Set` of `Point`s.
Refs #6222
|
|
Instead of a generic `ids` array, use clearer variable names for these
values.
Refs #6222
|
|
We need to persist stop IDs in order to properly construct `WayCost`
objects from the costs returned from the TomTom matrix API.
In order to persist stop IDs, my idea here is to group together a point
and its corresponding ID into a bucket. When we later
`#extract_costs_to_way_costs!`, we'll be able to grab the correct ID for
a given coordinate to create a `WayCost` from it.
Here, we create a new `TomTom::Matrix::Point` class that encapsulates a
coordinate and an ID, and build a `Set` of those. I needed an `#eql?`
and `#hash` method on `Point` as described in the `Set` documentation
(https://ruby-doc.org/stdlib-1.9.3/libdoc/set/rdoc/Set.html) in order to
properly maintain a unique set.
Refs #6222
|
|
A new component to the `TomTom` module that asks TomTom's Matrix API
endpoint
(https://developer.tomtom.com/online-routing/online-routing-documentation/matrix-routing)
to compute `WayCost`s. The matrix API will give us all costs between
each pair of coordinates. This will enable us to provide costs for any
combination of points in a journey pattern.
Given a list of `WayCost`s, it will send all points from those costs to
the matrix API and return a list of all non-zero `WayCost`s between all
pairs of coordinates.
`points_from_way_costs()` extracts unique coordinates from the
`WayCost`s.
`points_as_params()` builds a list of points in the format expected by
the matrix API.
The response from the matrix API is formatted as a two-dimensional array
consisting of rows and columns that pair each "origin" point with each
"destination" point. We loop through this matrix and construct new
`WayCost` objects for each pair of coordinates.
At the moment, I haven't figured out how I want to save `WayCost` IDs
when creating the new pairs. Leaving that for later.
Refs #6222
|
|
|
|
|
|
|
|
|
|
confirmed_at. Refs #6141
|
|
|
|
The JavaScript validation doesn't pass if we have more than two decimal
places (I think because it uses a number field step value of 0.01).
Round the values to allow them to pass frontend validation.
Refs #6203
|
|
Because we need to pass a Ruby hash to Rabl instead of a JSON string,
get rid of our serialiser and instead turn it into a function that just
converts the distance & time units.
Fix a bug in the test that had the `'1-2'` key as a symbol instead of a
string which was caused by a copy-paste from JSON and not being thorough
enough in search-and-replace.
Refs #6203
|
|
This serialiser will take `Route#costs` and convert the distance and
time fields from meters to kilometres and seconds to minutes
respectively. We need this because the frontend uses kilometre and
minute units while the TomTom API gives us the others (and we store the
data we get from TomTom without treatment).
Unfortunately, due to the way that Rabl works, this doesn't quite work
just yet. The serializer returns a string, and Rabl just puts this
string into the JSON output instead of a real JSON hash. Looks like I'm
going to have to convert my serializer into a generic converter.
Refs #6203
|
|
af83/6095-route--calculate-distance-and-time-cost-between-stops
Calculate distance and time cost between Route stops. Refs #6095
|
