LLMs and Concepts of Applied AI

Core Building Blocks

Inference is the compute layer of AI text generation. It's what's operating the model. Some inference providers own their models (OpenAI, Anthropic, Google) and some host licensed models from other labs (Bedrock, Vertex AI, DigitalOcean).

Harness is the application layer leveraging inference — it manages conversation state and actually executes anything the model asks for.

Between these blocks sit shared conventions — Agents, Tools/Functions, MCPs — that neither block strictly requires, but that most harnesses and inference APIs have converged on to talk to each other.

Inference and Tokens

What are tokens? Essentially words, but not exactly. They're the units of text completion that have relationships expressed mathematically, but for the sake of this discussion let's just think of them as words.

The LLMs we use when we refer to "chat" or "AI" are generally just predicting tokens based on the tokens they receive as input.

Inference and Tokens

So we can think of our LLM completions as input tokens and output tokens. When an LLM is generating output tokens, it's processing all the prior tokens as input — including the tokens it's already output.

This is why context size matters so much — every token you get back in a chat includes all the tokens before it as input. As the transcript grows in length, so does the input for each token.

What's a "turn"? A turn is one request/response cycle with the inference layer: you send input (system prompt + conversation history + any tool results) and the model streams back output until it stops for that request — either with a final answer or a tool-call request. Tokens are billed per turn: once for the full input, once for the total output generated in that turn. A single user message can trigger several turns if the harness loops through tool calls before returning a final answer (we'll see this in the coding harness example).

Even though, under the hood, generating each output token technically means reprocessing all prior tokens as input, you're not billed per-token-of-context — just once per turn for input and output.

There are strategies for making this more efficient, like KV caching, but that's out of scope for this talk.

Harness

Harnesses are generally the application or client interfacing with the inference layer. The inference layer is only responsible for taking input and producing output. When we use AI to "do things," the doing of things is happening in the harness layer.

For instance, if you want an AI to give you the weather, you might have a harness that has defined a tool get_weather that accepts a location as an argument. The LLM isn't fetching weather — it's simply outputting a request to call get_weather with some arguments, and the harness is what actually executes that.

Shared Conventions

Wait, what are these tools and agents we keep mentioning? Right! Let's go over the concepts shared between harness and inference.

System Prompt

Most inference APIs provide an OpenAI-like interface for chat/completions. Included in that structure is the concept of a system prompt.

When creating a chat payload to send to an inference layer, you can also supply a system prompt. This is a set of instructions for the LLM to guide its response with. This is where you specify instructions or give the LLM a "persona."

Agents

"AI Agent" is an overloaded term, but in the context of a harness, an Agent is essentially a system prompt with some additional conventions — most notably a declared list of tools it's allowed to use.

Most harnesses define their Agent files with frontmatter — a markdown convention of triple-dash (---) delimiters with YAML in between, placed at the top of the file.

This allows for storing metadata, but most importantly it can communicate which tools are available.

Sub Agents and Skills

Sub Agents: an Agent that another Agent can invoke like a tool — useful for delegating a scoped task (e.g., research, code search) without cluttering the parent conversation's context.

Skills: a packaged, on-demand set of instructions (and sometimes scripts/resources) that an Agent loads only when the task matches — keeping specialized knowledge out of the system prompt until it's actually needed.

Tools/Functions

Tools/functions are similar to system prompts in that they're a standard interface for chat/completion inference layers.

You define tools in your chat payload and specify what each one does and the input parameters (if any) it expects.

LLMs can then determine if they have enough context to execute a tool you've defined, and if so, will return a tool execution request.

The harness is responsible for executing the tool, not the inference layer. The LLM is simply predicting tokens — that's it.

Agents and Tools

So in the frontmatter, agents can specify tools available to them — but this is merely a convention. The inference layer does not have enough information on its own to know what or how to call the tool.

The harness needs to parse the tools list and provide its tool definitions accordingly.

Some tools are local, like searching the local file system (coding harnesses do this), and some are bespoke to the harness. Others are more generic, like creating a GitHub issue.

It would be unmanageable to teach every harness how to create a GitHub issue and distribute that tool definition around. This is what led to the Model Context Protocol (MCP).

MCPs

MCPs are a way to publish and make tools (and their execution) accessible remotely.

There are all sorts of details around MCPs that are out of scope here, like transport layers and OAuth (and MCP also covers things beyond tools, like resources and prompt templates) — for now just think of MCP as a standard API for AI tools.

Coding Harness

Let's look at how all these components work together in a coding harness. Say we have a project of code that has a bug in it.

As users, we'll want to ask it to find and fix the bug — so we'll start with a simple description of the bug and ask it to fix it.

Coding Harness

Included in this request is a system prompt and the harness's tool definitions.

One of the tools is likely a grep-like tool.

So the inference layer will likely output some acknowledgment text plus a tool-execute request — this is turn one.

Coding Harness

The harness executes the tool and appends the results to the conversation as a tool-result message — this isn't something the human user typed, it's the harness relaying data back to the model — then sends everything back to the inference layer for the next turn.

Based on these results, the LLM will likely decide to read a portion of a specific file.

Coding Harness

The LLM then concludes the source of the problem — because it was given the right information, which it figured out how to obtain on its own across the last couple of turns.

It'll output a brief explanation of what it thinks is wrong, then propose a solution in the form of a patch tool call.

Tokens Add Up

In this simple example, you can see how tokens add up and aggregate per turn — and how one user message turned into several billed turns before the final answer came back.

What's Out of Scope (For Now)

We've covered the core mental model: inference, harness, and the shared concepts connecting them.

A couple of things we intentionally skipped: some models have native web search built in (functioning like a hosted tool the provider runs for you), and some are "reasoning" models with extended chain-of-thought before answering. Both are worth their own talk, but aren't needed to understand the core concepts here.