Full demo available here:

Checkout this repo and try it out on a repo of your choice (or this one!) -

https://github.com/runloopai/demo-code-understanding-mcp

The current state of affairs

AI has evolved from a basic coding assistant to a full development partner; developers are using the magic of AI to instantly generate new UI components, tests, and even full features!

When working on your small side project, the magic feels like it never ends but when you get to work on Monday and ask for help within your company’s large monorepo that magic is replaced by chat loops of pure frustration.

Vibe Coding Complications

As the overall repo context gets larger and your code becomes filled with more and more company specific patterns, AI agents struggle to select the relevant context. This makes sense - AI agents are really ‘day 1’ employees and teaching an intern in English how to do a task can often be more work than simply doing it yourself. However, unlike our favorite interns, the AI agent will remain a day 1 employee no matter how much work it gets through.

So how can we help our favorite AI intern grow over time or make our interactions with them better despite growing context sizes? We have some options:

1. Wait for better models with large context windows…but we want solutions we can apply today and not in 6 months!
2. Make it easier to save previous context and specify new context. Tools like Cursor made it easier than ever to save context in special rules files and add new context to a chat with intuitive in-chat search.

Both these options solve the problem, but neither of them capture the feeling of true magic. You’re still picking a model and giving it a set of instructions to follow, in essence problem-solving for the AI.

Humans also have a limited context and we are able to use various heuristics to learn how to look through large or even unfamiliar repos efficiently. Surely, we can apply the same strategies human’s use to our AI Agents?

We can start by examining the “indexing strategies” human developers use to navigate and understand large repos that we could emulate:

• Common Knowledge Indexing: We use common patterns associated with coding and specific frameworks to infer where relevant code might be stored in a repo. For example, we often look for JS tests using the same file name suffixed by .test.js
• Code Structure Knowledge: Powered by our IDEs, we often use the static relationships between code symbols to quickly jump around the codebase and understand its relationships. Ex: VS Code's Go to definition feature!
• Code Runtime Indexing: We often run tests and observe the runtime behavior of the code associated with key tests to understand the relevant code paths for specific features. An example is following an e2e test trace through your codebase, noting where interaction and issues occur. LLM’s traditionally can’t do this without humans running debuggers locally and then adding print-outs to the context.
• Historical Knowledge Indexing: Beyond the current state of the code, we also check design docs, historical Pull Requests, previous issues and evolution of the code to understand intent. Checking PRs for last changed files is often how developers begin debugging bugs caused from multiple people pushing to a shared codebase.

AI is great at the first (better than most people!) but often misses the mark on the last three, creating a significant gap in capability. We set out to test how much those strategies could supercharge our AI agents.

Demo: Code Indexing MCP with Runloop

To test whether giving AI agents access to these “indexing strategies” would help, we created an MCP server that enables you to load repos in Runloop’s isolated sandboxes and create various indices to try and inform repo navigation. Drop it in Claude and give it a shot!

Common Knowledge Indexing Tools

Terminal access for AI models to access file structure and content on their own

1. Ask the LLM about your repository in plain text and let it pick the path to explore using the terminal tool execute_command_on_devbox
2. Use our prompt basic_code_understanding and see which tools the AI can compose together

Repository mapping for high-level folder based overview using Aider’s excellent Repo Map

1. Ask the LLM to load in a high level summary of the folder structure via read_repo_map tool and see how the additional context supercharges your queries

Code Structure Knowledge Tools

AST and Semantic search using the excellent cased/kit for intelligent code indexing and retrieval

1. Ask the LLM to fetch specific information about the repo via the tools run_kit_cli_get_file_tree, run_kit_cli_extract_symbols, and semantic_code_search all of which will return static information about the directory and the relationship between files
2. Use our prompt static_code_understanding and see which tools the AI will compose together

Code Runtime Indexing Tools

Test execution traces to reveal runtime code flow patterns associated with specific tests

1. Ask the LLM to run a specific test via run_pytest_call_trace which will follow the test’s trace and return information
2. Use our prompt dynamic_code_understanding and allow the AI to iterate over one or many tests

Historical Knowledge Indexing

PR history analysis to understand historical design decisions and understand which files and code symbols tend to get edited together

1. Ask the LLM a question via the tool github_history_semantic_search that uses semantic search on the body and titles of past PRs to find relevant files
2. Use our prompt  historical_code_understanding and see how different tools can be used to infer insight from the historical context

Benefits of the MCP Approach

1. Quick context-gathering: From hours of manual file selection to seconds of automated retrieval
2. More accurate and complete answers: AI has access to the right context needed to provide correct solutions
3. Deeper code understanding: Runtime information provides insights impossible with static analysis alone
4. Code preservation: Historical context helps AI understand why code is structured as it is
5. Continuous learning: The system builds a knowledge base about your codebase that improves over time

Try it yourself

Want to understand how it works in detail? Follow the quick-start below and let us know how it goes!

Setup repo

Set up MCP connection

Run runloop_mcp_config_setup.py to add this mcp to Claude or Cursor.

You can also do it manually by making edits to ~/Library/Application Support/Claude/claude_desktop_config.json or ~/.cursor/mcp.json


{
 "mcpServers": {
   "code-understanding": {
     "command": "/PATH_TO_YOUR_VENV/bin/uv",
     "args": ["--directory", "/REPO_PATH", "run", "runloop_mcp.py"],
     "env": {
       "RUNLOOP_API_KEY": "YOUR_RUNLOOP_API_KEY",
       "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY",
       "GH_TOKEN": "YOUR_GH_TOKEN"
     }
   }
 }
}



Setup a snapshot to iterate from

We’ve bundled a runloop_setup.py which initializes all relevant libraries and creates a snapshot so you can reduce repeated setup and test multiple repos at your own pace. The MCP server is also configured to dynamically created devboxes and download the needed libraries when there isn’t a snapshot present!

uv run runloop_setup.py


Open Claude Desktop/Cursor and run a few prompts

Tell me about <https://github.com/runloopai/runloop-example-code-understanding>?

Which functions in <https://github.com/runloopai/runloop-example-code-understanding> were recently changed in the git history?

What are the foundational functions in <https://github.com/runloopai/runloop-example-code-understanding>?