jayhemnani
say hi →

/oss · 6 merged

tashfeenahmed/freellmapi

A server that puts many free AI model providers behind one OpenAI-style endpoint, with routing and automatic failover.

the project on GitHub (opens in a new tab)

  1. 30 Oct 2026

    Cached prompt tokens are reported instead of zero

    What was broken
    The responses endpoint always said zero prompt tokens came from the cache, even when the provider said many did.
    What I changed
    The provider's cached token count is now passed through in the usage numbers.
    Why it matters
    People can see how much prompt caching is saving them.

    see the code, #1721 (opens in a new tab)

  2. 30 Oct 2026

    Tool calls stay in the same turn as the text before them

    What was broken
    In two kinds of replayed conversations, tool calls were split into a second assistant turn, which some providers reject.
    What I changed
    Tool calls are now joined to the assistant turn just before them when that turn has none yet.
    Why it matters
    Conversations that mix search or reasoning with tool calls work with more providers.

    see the code, #1720 (opens in a new tab)

  3. 30 Oct 2026

    The one-tool-at-a-time setting is passed on to the model

    What was broken
    A request asking the model to call at most one tool per turn lost that setting, so the model could still call several.
    What I changed
    The setting is now read and sent to the provider as its own one-call-at-a-time option.
    Why it matters
    Apps that need tools run one by one get the behavior they asked for.

    see the code, #1714 (opens in a new tab)

  4. 30 Oct 2026

    Streamed replies report the real token counts

    What was broken
    When a reply was streamed, the provider's real token counts were thrown away and a rough guess was reported instead.
    What I changed
    The real counts from the provider are now kept and passed on at the end of the stream.
    Why it matters
    Usage logs, dashboards and per-key limits are accurate for streaming tools, which is most of the traffic.

    see the code, #1713 (opens in a new tab)

  5. 24 Oct 2026

    A cut-off answer is marked incomplete

    What was broken
    When a model stopped because it hit its length limit or a content filter, the Responses endpoint still called the answer completed.
    What I changed
    Such a turn is now marked incomplete with the reason, the way the Responses API defines it, both streamed and not.
    Why it matters
    Coding tools can tell a cut-off answer from a finished one instead of taking half an answer as the whole.

    see the code, #1624 (opens in a new tab)

  6. 24 Oct 2026

    Two tool calls at once from Gemini stay two calls

    What was broken
    When Gemini asked to run two tools at once in a streamed reply, the server glued them into one broken call. Most clients then got an error, and the key was put on a cooldown.
    What I changed
    Each streamed tool call now gets its own number, like the other providers already did, and calls without an id no longer share one.
    Why it matters
    Agents that run tools in parallel through Gemini work instead of failing.

    see the code, #1623 (opens in a new tab)

← every project