From LLM Service to LLM Platform: How BizReach Standardized Production AI

Across the industry, production LLM development has moved through a familiar arc. The first useful prototype is often an external model call wrapped in an API. The first production service is where the real system appears: asynchronous execution, model guardrails, user experience, observability, and the question of who owns each of those concerns.

BizReach reached the same conclusion through its own product work. BizReach is a direct-recruiting platform where companies and headhunters send scout messages to candidates based on their published resumes. In July 2023, BizReach shipped a GPT-powered resume auto-generation feature, and our evaluation at the time showed an average 40% increase in scout reception for resumes created with the tool. Starting as our first production PoC shortly after ChatGPT’s release, this was a carefully controlled effort, with multiple stakeholders evaluating safety, product value, and operational risk. That process led to what we internally call the SAR (Sender-Agent-Receiver) architecture. Later, similar product requests appeared on the company side through job-posting generation.

The architectural lesson, however, was not simply that LLM features were valuable. The lesson was that once one service succeeds, other teams will ask for similar capabilities, and the hidden runtime work around the model becomes too important to copy service by service.

This article is about that transition: how we moved from one hardened LLM service to a shared LLM platform, what design decisions led to an ownership model that allowed product teams to ship independently, and why the first platform shape was intentionally limited to one request and one generation lifecycle.

BizReach's LLM journey accelerated once shared runtime capability became a platform.

1. The First Service: Hardening the API Wrapper

In 2023, many developers were learning the same first pattern: wrap the OpenAI API, engineer a prompt, expose an endpoint, and iterate. Useful for prototyping, but incomplete for production.

The first BizReach LLM service could not be only a synchronous wrapper. LLM calls are long-running operations, as compared with ordinary web request expectations. The output is also different from many traditional ML API responses: for long-form text, the user experience benefits from progressive retrieval to keep the user engaged, instead of waiting silently for one final response. At the time, red-teaming practices for generative AI were still maturing in the industry, and prompt injection or jailbreak-style misuse was an active concern. Input and output needed guardrails that preserved the level of professionalism expected in BizReach product experiences. Client-side APIs also needed to know whether generation was still progressing, finished, timed out, or had ended because of a guardrail path or external failure.

So the first service was a system design challenge. It was already SAR-shaped, even before we turned it into a platform. SAR is short for Sender-Agent-Receiver: in generic terms, a web-queue-worker pattern with LLM-specific lifecycle behavior wrapped around it. Each role is a distinct stage, so a slow model call never has to hold a synchronous connection open.

1. Sender accepts and validates an incoming request, hands the work to a queue, and immediately returns a request_id.
2. Agent picks that work up asynchronously, runs the generation, and writes output as it is produced.
3. Receiver answers the client’s polling, returning generated output as it becomes available.

This design hardened the API-wrapper mental model. The public API could still be simple for clients, but the system behind it separated request acceptance, generation execution, and result retrieval. The three component services could also be scaled independently based on different operational signals, which made capacity planning and debugging easier.

SAR kept the client interface simple by making the generation lifecycle explicit behind it.

The design worked in production. That point matters. The later platform was not a rescue from a failed service. It was a generalization of a service that had proved product value and taught us which runtime concerns were too expensive to rediscover.

Production also made the cracks visible. Some issues were immediate: timeout values that looked safe in controlled testing needed adjustment under real workloads, concurrent workers needed tuning, and moderation behavior had false-positive cases that only appeared with real inputs. Monitoring and alerts reduced TTD (time-to-detect), while logging improvements reduced TTM (time-to-mitigate).

Others were more subtle. During later platform work, APM trace inspection surfaced a data and status write ordering issue that violated ACID consistency. It was invisible while it lived inside one product-specific implementation and was enough to delay chunk retrieval for the client.

That is why our view changed. From the outside, the first LLM feature felt primarily like a prompt problem, especially given the risks associated with LLMs and the focus on prompt engineering at the time. Over time, we understood that it was a complete product-surface and failure-mode design problem. Prompt quality matters, but production trust depends on what happens when the model is slow, partial, off-format, blocked by moderation, or affected by upstream behavior outside the product team’s control.

2. Shifting Runtime Risk Down

If the first service had remained the only LLM feature, the dedicated implementation might have been acceptable. It was close to the product, and the team that operated it understood its history.

The second service changed the question. On paper, this looked like a straightforward reuse problem: copy the proven SAR architecture, carry over the hard-won production fixes from the first launch, and layer in new product-specific prompts and schemas.

That assumption is the trap.

Successful system architectures look reusable, but their real value often sits below the surface — the decisions that made the first service reliable in production. By creating copies instead of formally turning those decisions into a shared execution model, we risk new services drifting from the intended behavior. A product team can and should own its domain intent, but once services from multiple domains depend on the same execution model, enforcing that structure becomes a platform responsibility.

We had three realistic options.

We chose the shared platform. With forked codebases, each service adds similar fixes and features at different times and in slightly different forms. That makes the execution models hard to keep consistent, and the accumulating drift adds new points of failure. Each service also runs under different load and different user behavior, so a bug usually surfaces in one service first. With a shared runtime, the fix reaches every other service before the same problem occurs there; without one, it stays in the service that found it, and there is no guarantee the other teams apply the same fix. In code, these fixes and features look like ordinary helper functions. In practice, they are operational contracts: guarantees about how every service on the platform executes. Instacart described a similar pressure in its Maple platform: once multiple teams need LLM processing, fragmented operational handling is no longer feasible. Our workload was different, but the pattern was familiar.

Google’s platform engineering writing uses the phrase “shift down” for moving recurring responsibilities into underlying platform layers, thereby reducing the operational burden on developers. This framing matched our situation closely and helped us describe the next step: which production responsibilities were too important to leave inside every product surface?

Answering that required identifying “quality attributes” and separating product intent from runtime risk.

• Reliability depended on queue behavior, timeout handling, status ordering, chunk retrieval and lifecycle consistency.
• Security depended on guardrail design and strict execution paths.
• Efficiency depended on independent scaling behavior of SAR components.
• Observability depended on shared logs, traces, metrics, and a unified status vocabulary.

Product intent, on the other hand, belonged in endpoints, schemas, prompts, output expectations, and user experience.

Recurring runtime risk shifts down into a shared execution model; product intent stays on top.

In other words, the platform decision was not “move common code into a shared repository”. It was “move recurring runtime risk into a shared execution model”. Every onboarded service should benefit through the shared platform instead of re-learning the same production lessons on its own schedule.

To do that cleanly, we had to look back at SAR as the platform’s execution model, and understand what each element was doing in production.

3. SAR as the Harness

SAR is easiest to describe as a web-queue-worker pattern, but that is only the starting point. Its production value was making a long-running LLM call behave like a controlled request lifecycle: validate, enqueue, execute, moderate, persist, retrieve, observe, and terminate work, without forcing the client to hold one fragile synchronous connection.

Looking back from 2026 with more recent industry vocabulary, SAR was our early take on a “harness”: the infrastructure around an LLM that lets it act, observe, remember, recover, and stay within guardrails. Using the SAR harness, our product features could manage latency, state, partial outputs, failure modes, and guardrails consistently.

SAR as the Harness around a raw LLM call.

The client-facing shape remained simple: submit a generation request, receive a request_id, then poll for generated chunks until the lifecycle ended. The execution model behind it had several features that we identified as platform responsibilities.

Asynchronous Lifecycle Separation

The first feature was asynchronous lifecycle separation. In a conventional synchronous API wrapper, request acceptance, model execution, and response delivery happen inside one fragile path. SAR deliberately decoupled the request path from the generation path by splitting the lifecycle across Sender, Agent, and Receiver.

The main payoff was that each component could be scaled on its own signal: Sender and Receiver are latency-bound and autoscale on request rate (RPM) and p99 latency; Agent pool is throughput-bound and autoscales on queue depth and in-flight generation concurrency. Because LLM generation takes orders of magnitude longer than accepting a web request, coupling them in one service would have forced us to overprovision web servers just to get more worker capacity. The queue absorbs this impedance mismatch: it buffers bursts, smooths the arrival distribution the workers see, and makes backpressure explicit. Capacity planning becomes precise and efficient as a result.

The client did not need to understand queue timing, worker state, or internal persistence details. It only needed to know whether to consume a chunk, keep polling, or stop. This distinction answered the platformization question: product teams should be able to change what they generate without changing how generation moves through the system.

Decoupled Persistence: Request, Status, Data

The second feature was managing requests, lifecycle status, and generated outputs in separate tables. That separation gave each SAR component a stable minimum surface to manage artifacts produced in the lifecycle.

Separating request, status, and data made failures easier to localize.

This made failures easier to localize. Because each artifact is owned by a specific stage, a missing or stalled record points at the responsible component rather than the system as a whole. The debugging signals in the figure above trace each symptom back to its producer stage.

The platform element was the persistence model: what gets written, when it gets written, and which SAR service reads it. The product elements were the request schemas and response formats. Products owned the data shape, while the platform owned how that data moved through the lifecycle. Outside the request path, archival jobs could merge lifecycle records and export them to object storage, keeping the operational key-value stores focused on active retrieval and debugging.

Status as a Finite-State Machine

The third feature was controlling the execution “status” with a deterministic finite-state machine. In a tightly connected async system, unclear middle states are expensive. Sender, Agent, and Receiver run as separate services, but they still need one shared lifecycle vocabulary.

Platform lifecycle governed by a deterministic state machine; Client interpretation driven by status and chunk availability.

WIP is the only non-terminal state. FINISHED and terminal error states end the lifecycle. Receiver then interprets execution state together with chunk availability, because the client is not only asking whether generation is done; it is asking what to do next for a specific requested chunk.

The useful takeaway is that asynchronous LLM services need deterministic lifecycle language instead of vague middle states. The platform element was deterministic status behavior: state transitions, terminal conditions, and Receiver interpretation. This prevented clients from inventing their own interpretations of slow output, missing chunks, moderation exits, or timeout paths. The product element was deciding how each client experience should respond to those standardized statuses.

Progressive Retrieval for Text and JSON

The fourth feature was progressive chunked retrieval. We realized the need to serve different output formats for different generation tasks. Resume generation produced long-form text that users could read progressively. Later generation tasks needed structured JSON outputs that product UIs could place into specific sections.

One lifecycle, two retrieval units; progressive delivery improves perceived latency.

For text, waiting for one final response would make the feature feel slower than it actually was. Chunking streamed responses let Receiver deliver partial output as soon as Agent had written it. This separated time-to-first-chunk from total generation time, which matters for UX as users can see progress.

For json, the platform used the same lifecycle but treated top-level key-value pairs as retrieval units. That made structured generation usable in the same polling model, but it came with a trade-off: streaming improves perceived latency, but it limits how strongly the platform can validate the full response schema before returning partial output. We accepted that trade-off, and production has supported our hypothesis: most product workflows benefit more from progressive delivery than from waiting for one fully-validated API response.

The platform element was chunking logic, per-chunk status updates, and Receiver interpretation for text and json modes. The product elements were output mode selection, expected response format, and UI behavior. This let different product surfaces use the same lifecycle while still receiving output in the shape their users expected.

Guardrails in the Execution Path

The fifth feature was moderation guardrails in the execution path. Instead of being an optional client-side responsibility, content moderation was made part of Sender and Agent behavior, supported by stricter system prompts.

On the input side, product teams identified which request fields needed moderation before enqueueing generation work. This gave the system a chance to intercept unsafe or policy-violating requests before they reached the model, and it also gave operators a clearer signal when bad actors appeared.

On the output side, Agent checked generated chunks before writing them to the data store. Initially, this was centered on a rule-based pattern-matching algorithm for streamed text. As the platform evolved, the guardrail strategy expanded into layered checks: cloud-provider content filters, model refusal detection, and blocklist detection.

Moderation is built-in: only approved input reaches the model, only approved output reaches retrieval.

The platform element was the configurable moderation mechanism, failure status, and enforcement path. The product elements were moderated fields, system prompt design, and moderation resources. This gave product teams control over domain-specific moderation resources while keeping guardrail behavior consistent across services.

This was the architectural insight that made platformization possible. SAR was not just the shape of one service. It was a harness around generation. Once we could name platform concerns and product intent separately, we could start decoupling them.

4. Configuration-as-Code

Once a shared runtime is established, the next question is how much variability the platform should actually expose. “Shift down” talks about moving risk into a shared model, but that process can easily leave too many moving parts in the wrong place. Of course, there is no shortage of things a platform could expose to its consumers. But how much should they actually control? For platform teams, defining this boundary is the difference between building true golden paths and inheriting bespoke technical debt.

If a product team can accidentally tweak the internal state machine or override critical retry mechanics during integration, incident response becomes a guessing game, and the organization hasn’t actually shifted risk down. We need a stricter path where product teams are exposed to only the smallest useful surface to declare product intent, while the platform guarantees how that intent is executed.

Declarative infrastructure platforms are powerful precisely because they enforce this boundary through a strict separation of concerns. Kubernetes establishes the “record of intent” as the foundational mental model for modern infrastructure: users declare the desired state in a YAML manifest, and the control plane continuously works to match it. Engineering teams at John Lewis further narrowed the Kubernetes surface area into a custom Microservice resource after finding that only a fraction of what developers wrote was truly application-specific. Airbnb added the safety lesson: exposing variability is risky unless it is strictly validated before it ever reaches the runtime.

Our scope was much smaller than that of those infrastructure giants, but the lesson mapped well. The goal was not to make everything configurable. It was to expose only the smallest useful product-facing surface, and lock the risky operational mechanics away from the config boundary.

A config pack, therefore, declared only what varied by product: endpoint routing, schema definitions, generation mode, moderation fields, prompt entry points, and agreed-upon SLA tuning values.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

endpoints:
  - name: "example-generation"
    route: "/example-generation"
    mode: "text"
    request_schema: "/config/schemas/openapi.yaml"
    prompt_script: "/config/prompts/generator.py"
    moderation_fields:
      - "user_input"

generation:
  chunk_update_timeout: 20
  chunk_size: 100

moderation:
  input_bl_path: "/config/data/input_blocklist.txt"
  output_bl_path: "/config/data/output_blocklist.txt"

The more important detail is what the config pack did not contain. It did not decide how to initialize status, when a chunk was safe to return, how output moderation terminated generation, or how a stale generation became a client response. Those behaviors stayed invariant in the platform runtime.

The config pack is the contract: products declare what varies; the platform guarantees how it runs.

The Exception: Prompt Construction

The one place we deliberately allowed product-specific code was prompt construction. Prompts are deeply tied to domain intent and complex input data, making them too nuanced for flattened YAMLs. However, to prevent this from breaking the platform boundary, we constrained it behind a small, strict Python plugin interface:

1
2
3
4
5

class PromptGenerationPlugin(Protocol):
    prompt_version: str

    def create_prompts(self, input_data: dict) -> list[Prompt]:
        ...

This interface gives product teams the full flexibility required for advanced prompt engineering while keeping the platform completely ignorant of domain-specific logic.

The Deployment Surface

The deployment surface made the boundary operational, not just architectural. A product service extended the platform image and copied its config pack into a standard location.

1
2
3

FROM llm-platform/<sender|agent|receiver>:1.x
COPY ./config_pack /config
ENV CONFIG_PATH=/config/config.yml

The platform image builds the runtime. The service image pins a platform version and overlays the product configuration. Configuration is validated against the platform at startup, failing fast in development or staging environments if the contract is violated.

This was the real payoff. Configuration-as-code did not make the LLM service reliable by itself. The value came from reducing the product-facing surface until SAR could own the failure modes. If the declaration is wrong, the problem should be caught at the config boundary; any operational failures should belong to the platform. That ownership boundary is what defined the team topologies in the next section.

5. Platform Ownership and Team Topology

Conway’s Law suggests that systems eventually mirror the communication structures of the organizations that build them. By drawing a strict architectural boundary between runtime execution and product intent, we effectively reverse-engineered that law, creating a system that allowed our teams to work with complete independence. The technical boundary established by the config pack did more than protect the system. It solidified cross-team interactions.

Using the vocabulary of Team Topologies, this declarative approach explicitly divided our responsibilities. Product teams act as stream-aligned owners, focusing entirely on domain value, prompt quality, and user experience. The platform team, meanwhile, absorbed the enabling and complicated-subsystem work: managing async queues, optimizing latency, scaling the SAR components, and maintaining guardrail infrastructure. While the platform team took on the heavier operational burden, that work was finally distinct, predictable, and cleanly isolated from business logic.

One platform team enables many stream-aligned product teams behind a versioned contract.

This separation naturally decoupled our release cycles. In our shared monorepo, the platform team publishes versioned base images at their own pace through standard CI/CD pipelines. Product teams simply watch their specific service directories; when they need to tweak a prompt, their pipeline overlays their config pack onto a pinned platform image and deploys. A product team can ship prompt iterations multiple times a day without waiting for a platform release train, and platform engineers no longer act as gatekeepers for business logic.

When things do break, the clarity of the architecture makes incident response straightforward. When an alert hits our central triage channel, it is immediately obvious whether it is a product issue like request schema validation errors or a platform issue like upstream model latency.

By drawing a hard line between what the LLM feature should do and how the system executes it, we created the operational freedom to scale. We can now support a growing number of distinct AI product features across BizReach without requiring a linear increase in platform headcount.

6. A Deliberately Small Platform

It is worth being honest about what this initial platform is not. The first platform shape supported exactly one contract: a single generation request, moving through a single lifecycle, retrieved as chunks until it terminates. There was no multi-step orchestration, no tool use, no conversation memory, no agentic loop. In 2026, that can sound conservative.

That limitation was the point. Every behavior described in this article — the status state machine, chunked retrieval, moderation paths, the config boundary — was reliable precisely because the lifecycle it governed was small enough to be fully specified. A platform earns the right to grow by making its current contract boring: deterministic to operate and predictable to debug. Generalizing before that point would have meant scaling our open questions instead of our answers.

Looking back, I would not tell teams to build a platform before their first LLM feature, but to design that first service as if a second one may someday exist. The first successful service proves product value. The second tests whether the architecture can scale an organization. The arc that began with one hardened API wrapper ends with a new default inside BizReach. When a team proposes a new generative feature, they no longer begin by rediscovering system semantics; they dive directly into product intent — endpoints, prompts, and schemas.

What comes next will test this boundary rather than discard it. In fact, it already has. To support richer multi-step execution, our architecture has evolved. Today, while the outer SAR shape manages the asynchronous lifecycle, SAR’s internal Agent service now executes deterministic LLM workflows. Looking further ahead, agentic workloads introduce a much larger guardrail surface and pressure the assumption that one request maps to one lifecycle. But the core lesson remains: those capabilities must arrive as an explicit execution model with named states and owned failure modes, not as clever code copied service by service.

The first platform standardized one request, one lifecycle, one result; multi-step graph workflows are what comes next.

If you’re interested in learning about the transition from single-generation calls to graph-based LLM workflows, keep an eye on the Visional Engineering Blog so you don’t miss Part 2. And if tackling these kinds of distributed systems and LLMOps challenges sounds like your kind of engineering, we are hiring — come help us build the future of recruitment.

• AI
• Platform
• Backend

Kunal Jain

AI Platform Lead at BizReach's AI Platform Group. I keep our LLMs reliable, observable, and safely employed, so AI can advance careers instead of ending them.