I was integrating an external API. The datetime it stamped on every response never agreed with my local clock — and not by a clean, constant amount. Sometimes it was 4 minutes behind. Later, 8. Then more. The gap kept drifting.

  our clock          their timestamp       gap
  ─────────────────────────────────────────────
  10:14              10:10                 −4 min
  11:02              10:54                 −8 min
  13:40              13:29                 −11 min
                                            ↑
                            not a constant — it grows

The Wrong Theory#

My first instinct was that we were sending the wrong value, so I checked the payload. The date was correct. That check cost me time: a timezone shift usually leaves the date component intact and only moves the clock, so a date-only comparison passes while the instant is wrong. The date only flips when the shift crosses midnight — which is why “the date is right” feels like evidence and isn’t.

My second theory was timezone, full stop. That was closer, but it could not explain the numbers. A timezone error is a fixed offset. It does not grow by 4 minutes over an afternoon.

A constant error is one bug. An error that changes size is usually two bugs wearing the same coat.

The Root Cause#

Two separate faults stacked on top of each other:

total error = fixed offset      → the API is not timezone aware.
                                  It takes and returns naive timestamps —
                                  a wall clock with no offset, no Z, no zone —
                                  and reads them in whatever zone its own
                                  servers happen to run in.

            + drifting offset   → their server clock itself was not
                                  synchronised. It slipped further behind
                                  real time as the day went on.

The first fault made the timestamps wrong. The second made them inconsistently wrong, which is what hid the first one. A naive timestamp is not a point in time — it is a point in time plus an unstated assumption, and the assumption belongs to whoever parses it last.

What Actually Fixed It#

Three things, in this order:

  • Find out what timezone the external API’s servers are in. Not what the docs imply — what they actually do. Send a known value, read it back, measure the delta.
  • Find out the exact timezone of our own pods. Container timezones are not a given. The host, the base image, and the runtime can each disagree, and a pod rescheduled onto a different node can quietly change the answer.
  • Align them, or normalize at the boundary. Pick one representation — UTC with an explicit offset — and convert on the way in and on the way out. Never let a naive timestamp cross a network boundary in either direction.

And one thing that came out of the drift specifically: stop treating a remote timestamp as truth. Record your own timestamp next to theirs on every call. The pair is what let me see the drift; theirs alone just looked like noise.

The Lesson#

You do not get to choose the quality of the APIs you integrate with. An API that takes naive timestamps has pushed a correctness burden onto every one of its callers, and being right about that will not fix your output. Assume nothing about the other side’s clock, measure the delta instead of reasoning about it, and normalize at your own edge — that is the only part of the system you control.