Blog / Tags

#latency

2 articles

  1. 900 KB of one repeated letter took almost six minutes to countPerformancetiktoken-go is quadratic on one long piece of text: how we measured it, two fixes that did not hold, the 512-byte bounded count, and what it does to billed tokens.
  2. Our gateway spent its latency budget before calling the modelPerformanceSerial database and Redis round trips on our relay hot path, three changes that removed many of them, and what the measurements do and do not show.