| Age | Commit message (Collapse) | Author |
|---|
|
af83/6884-tomtom-matrix--handle-error-when-response-doesn,t-inclu
Handle API error(s) in Tomtom matrix. Fixes #6884
|
|
I previously tried to correct a circular dependency problem in
a057276129b1f62b811743db3b8f867a05241ed3, but that didn't fix it (it was
intermittent, and came back).
After some wrangling, I've now deduced with some confidence that the
problem comes from `RouteWayCostCalculator`, which used
`TomTom::Matrix::RemoteError`. From the way it looks, this seems to mess
up the Rails autoloader since `tom_tom.rb` will try to load the `Matrix`
class from the `TomTom.matrix` call above. Or something.
In an attempt to fix the circular dependency error for real this time,
move the error class to a completely separate module from `Matrix`, and
refer to this when we need to use the error class.
Refs #6884
|
|
We might not always get a nicely formatted JSON
`['error']['description']` response body. Sometimes, like for example
when you use an incorrect API key, even with an 'application/json'
content type, TomTom will respond with:
<h1>Developer Inactive</h1>
What?
In that case, the response has a 403 status. In addition to checking for
an error in the response, should also be checking for the HTTP status
code.
Log the status code in the exception to give us more information about
what went wrong.
Update our existing tests now that `#check_for_error_response` takes a
response object instead of a JSON string.
Refs #6884
|
|
I was getting the following circular dependency error in Sidekiq:
2018-05-03T14:47:17.974Z 19217 TID-owg7qzmqc WARN: RuntimeError: Circular dependency detected while autoloading constant TomTom::Matrix::Point
2018-05-03T14:47:17.974Z 19217 TID-owg7qzmqc WARN: .../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:492:in `load_missing_constant'
.../.gem/ruby/2.3.3/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:184:in `const_missing'
.../stif-boiv/lib/tom_tom/matrix.rb:44:in `block in points_from_way_costs'
.../stif-boiv/lib/tom_tom/matrix.rb:41:in `each'
.../stif-boiv/lib/tom_tom/matrix.rb:41:in `points_from_way_costs'
.../stif-boiv/lib/tom_tom/matrix.rb:12:in `matrix'
.../stif-boiv/lib/tom_tom.rb:24:in `matrix'
.../stif-boiv/app/services/route_way_cost_calculator.rb:8:in `calculate!'
.../stif-boiv/app/workers/route_way_cost_worker.rb:12:in `perform'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:167:in `execute_job'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:139:in `block (5 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:135:in `block (4 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
.../.gem/ruby/2.3.3/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/sidekiq.rb:33:in `block in call'
.../.gem/ruby/2.3.3/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:369:in `perform_action_with_newrelic_trace'
.../.gem/ruby/2.3.3/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/sidekiq.rb:29:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb:10:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:134:in `block (3 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb:32:in `with_context'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:183:in `stats'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:131:in `block in process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:126:in `process'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:82:in `process_one'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:70:in `run'
.../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:17:in `watchdog'
../.gem/ruby/2.3.3/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:26:in `block in safe_thread'
Fix it by moving the error class to a new file.
Refs #6884
|
|
If there's an API error, we shouldn't update the route's `costs` field.
Let's say we've already calculated some costs for a route A. We then edit
and re-save A, which triggers a recalculation of the costs. Now the
TomTom API responds with an error. We don't want to overwrite our
existing costs with an empty array because they could still be useful.
In this case, we should instead keep the existing costs we already had.
To achieve this, move the `RemoteError` rescue into
`RouteWayCostCalculator`, leaving the error unhandled in
`TomTom.matrix`.
Refs #6884
|
|
Occasionally, the following error would appear in our logs:
NoMethodError
RouteWayCostWorker/perform
Error message
NoMethodError: undefined method `each_with_index' for nil:NilClass
Stack trace (show Rails)
/app/lib/tom_tom/matrix.rb: 83:in `extract_costs_to_way_costs!'
/app/lib/tom_tom/matrix.rb: 23:in `matrix'
/app/lib/tom_tom.rb: 24:in `matrix'
/app/app/services/route_way_cost_calculator.rb: 8:in `calculate!'
/app/app/workers/route_way_cost_worker.rb: 12:in `perform'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 167:in `execute_job'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 139:in `block (5 levels) in process'
/var/lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq.rb: 36:in `block in <module:Sidekiq>'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 135:in `block (4 levels) in process'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 128:in `block in invoke'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
…-4.2.10/lib/sidekiq/middleware/server/active_record.rb: 6:in `call'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
…idekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb: 10:in `call'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
…kiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb: 74:in `call'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 130:in `block in invoke'
….0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb: 133:in `invoke'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 134:in `block (3 levels) in process'
…/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb: 32:in `with_context'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 132:in `block (2 levels) in process'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 183:in `stats'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 131:in `block in process'
/var/lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq.rb: 35:in `block in <module:Sidekiq>'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 126:in `process'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 82:in `process_one'
…ems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb: 70:in `run'
…lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb: 17:in `watchdog'
…lib/gems/2.3.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb: 26:in `block in safe_thread'
My best guess is that this was caused by TomTom responding with an
error, which we weren't handling previously. In that case, the response
would be a JSON string, but include an `'error'` field instead of
`'matrix'` and `'summary'`. Thus, when we'd try to
`matrix_json['matrix']`, it would fail.
Add a new method that checks for errors before we try to parse
`WayCost`s. If a server error is detected, we log the message to the
Rails log and return an empty array.
Refs #6884
|
|
By default the TomTom API uses traffic data to calculate travel time.
This means that previously, getting costs for the exact same route twice
at different times could produce different results.
Johan made a good suggestion to turn off this functionality as it's
apparently enabled by default. This should get us correct times
regardless of time of day since the distances will be effectively the
same.
Refs #6661
|
|
|
|
Make it clearer why we have to use a custom serialiser here.
Refs #6222
|
|
Rails serialises `BigDecimal`s as JSON strings to prevent loss of
precision. The `latitude` and `longitude` columns in `StopArea` are
stored as `BigDecimal`s. The trouble is that TomTom's API requires the
latitude & longitude values to be JSON floats, not strings.
Make a new JSON serialiser that converts the `BigDecimal` coordinates to
float to allow the values to be correctly interpreted by the API.
Refs #6222
|
|
Already handled this a few commits ago.
Refs #6222
|
|
Using a set ended up not working out. I needed to be able to index into
the list in `#extract_costs_to_way_costs!`, and sets aren't indexable.
This is because they're supposed to be unordered, though modern Ruby
implements `Set` with `Hash` under the hood, which is ordered in Ruby.
I like the idea of having a data structure that automatically eliminates
duplicates, but it wasn't meant to be, because for the extraction to
`WayCost`s, I need an ordered list. Rather than create a new
`OrderedSet` type, I just went the simple route and used an Array,
eliminating the duplicates manually because I know when duplicates are
supposed to occur due to the nature of the data set.
Remove the `#eql?` and `#hash` methods from `TomTom::Matrix::Point`.
Because we're not longer using `Set`, these methods don't need to be
implemented.
Refs #6222
|
|
Change this code to get stop IDs based on the change in
be3e1effcdea87909a181c7e9b12cf6867b1839d. It should use these IDs to
construct new `WayCost`s with the combination of stop IDs from departure
and arrival stops.
This doesn't work currently because the method has code that tries to
index the `points` collection, but it's a `Set`, and sets don't support
indexing and aren't ordered, so this code errors. Need to change the
`Set` to something else that will work here.
Refs #6222
|
|
Rewrite this method to accept `TomTom::Matrix::Point`s instead of plain
`Geokit::LatLng` coordinates. Do this because this method needs to take
the result of `#points_from_way_costs` as input, and that method now
returns a `Set` of `Point`s.
Refs #6222
|
|
Instead of a generic `ids` array, use clearer variable names for these
values.
Refs #6222
|
|
We need to persist stop IDs in order to properly construct `WayCost`
objects from the costs returned from the TomTom matrix API.
In order to persist stop IDs, my idea here is to group together a point
and its corresponding ID into a bucket. When we later
`#extract_costs_to_way_costs!`, we'll be able to grab the correct ID for
a given coordinate to create a `WayCost` from it.
Here, we create a new `TomTom::Matrix::Point` class that encapsulates a
coordinate and an ID, and build a `Set` of those. I needed an `#eql?`
and `#hash` method on `Point` as described in the `Set` documentation
(https://ruby-doc.org/stdlib-1.9.3/libdoc/set/rdoc/Set.html) in order to
properly maintain a unique set.
Refs #6222
|
|
A new component to the `TomTom` module that asks TomTom's Matrix API
endpoint
(https://developer.tomtom.com/online-routing/online-routing-documentation/matrix-routing)
to compute `WayCost`s. The matrix API will give us all costs between
each pair of coordinates. This will enable us to provide costs for any
combination of points in a journey pattern.
Given a list of `WayCost`s, it will send all points from those costs to
the matrix API and return a list of all non-zero `WayCost`s between all
pairs of coordinates.
`points_from_way_costs()` extracts unique coordinates from the
`WayCost`s.
`points_as_params()` builds a list of points in the format expected by
the matrix API.
The response from the matrix API is formatted as a two-dimensional array
consisting of rows and columns that pair each "origin" point with each
"destination" point. We loop through this matrix and construct new
`WayCost` objects for each pair of coordinates.
At the moment, I haven't figured out how I want to save `WayCost` IDs
when creating the new pairs. Leaving that for later.
Refs #6222
|
