AI Stack Layer · 3 of 8

AI Frameworks

The orchestration layer above raw model APIs — chains, agents, RAG pipelines, prompt management, evaluation. The plumbing that turns "call an LLM" into a real product.

OrchestrationRAGAgentsTool UseLayer 3
← Back to AI Landscape
Quick Facts

At a Glance

Basic Concepts

  • Chain: a sequence of LLM + tool steps where each output feeds the next.
  • Agent: an LLM that picks tools and takes actions in a loop until done.
  • RAG (Retrieval-Augmented Generation): fetch relevant docs → stuff into the prompt → ask the model.
  • Tool calling: the LLM decides which function to invoke; the framework runs it and feeds results back.
  • Memory: conversation history (short-term) and stored facts (long-term).
Landscape

The Major Frameworks

FrameworkLanguageBest for
LangChainPython, TSMost popular; chains, agents, integrations to ~everything.
LangGraphPython, TSStateful, graph-based agent workflows (LangChain's successor for agents).
LlamaIndexPython, TSRAG-first — ingestion, indexing, query engines.
HaystackPythonProduction RAG & search pipelines (deepset).
Semantic KernelC#, Python, JavaMicrosoft's enterprise orchestrator with planners.
DSPyPythonProgrammatic prompts that auto-optimize against metrics.
Anthropic Agent SDKPython, TSClaude-native agents with built-in tool use & subagents.
OpenAI Agents SDKPythonOpenAI-native agent framework with handoffs & guardrails.
Vercel AI SDKTypeScriptStreaming UI helpers for Next.js / React.
CrewAI / AutoGenPythonMulti-agent collaboration patterns.
Mechanics

Core Patterns

Retrieval-Augmented Generation (RAG)

The standard pattern for "ask my docs":

  1. Ingest: chunk documents, generate embeddings, store in a vector DB.
  2. Retrieve: embed the user's question, find top-K similar chunks.
  3. Augment: stuff those chunks into the prompt as context.
  4. Generate: ask the LLM to answer using only that context.

Modern RAG adds re-ranking, hybrid (keyword + vector) search, query rewriting, and metadata filters.

Tool Use & Function Calling

You describe a function (name, JSON schema). The model decides when to call it and produces a JSON argument; the framework runs it, feeds the result back, and the model continues. Used for:

  • Real-time data (weather, stock prices).
  • Database queries.
  • Math & computation (the LLM is bad at it; Python isn't).
  • External APIs (Stripe, Slack, GitHub).
Agents & Loops

An agent is an LLM in a loop: think → act → observe → repeat. The framework handles:

  • Planning — break the task into steps.
  • Tool selection — pick the right action.
  • State management — what's been tried, what worked.
  • Termination — when to stop (max steps, completion check).

Newer frameworks (LangGraph, Anthropic Agent SDK) model agents as explicit state machines instead of opaque loops.

Prompt Management & Evaluation
  • Prompt templates — versioned, parameterized, often kept outside code.
  • Evals — golden datasets + scoring (LLM-as-judge, human, exact match).
  • Tracing — Langfuse, LangSmith, Weave, Phoenix capture every prompt & response.
  • A/B testing prompts & models in production.
Pick a Framework
If you want…Reach for
Maximum integrations & ecosystemLangChain
Graph-based stateful agentsLangGraph
RAG done well, fastLlamaIndex
Microsoft / .NET shopSemantic Kernel
Provider-native agentAnthropic / OpenAI Agent SDK
Streaming UI on the webVercel AI SDK
Auto-optimized promptsDSPy
Continue

Other AI Stack Layers

Foundation ModelsModel ProvidersVector DBsDev AgentsMLOpsClassic MLData Prep ↑ Back to AI Landscape