Members-Only
Recent Talks & Demos are for members only
You must be an AI Tinkerers active member to view these talks and demos.
February 26, 2026
·
Bogotá
From Benchmark to Agent: A2A‑Driven Werewolf Arena for Evaluating LLM Agents
Explore Werewolf Arena, a benchmark evaluating LLM agents' reasoning, persuasion, and deception in a social deduction game. See how agents perform under partial information.
Overview
Werewolf Arena is a multi‑agent social‑deduction benchmark built on the AgentX–AgentBeats stack, developed as part of the second cohort of TribuPapers and the 2025 Berkeley Agentic AI MOOC, and submitted to the AgentX–AgentBeats competition. It instantiates the Werewolf game as an 8‑player environment where a Green Agent orchestrator manages role assignment, day/night phases, interaction flow, and ELO‑style ratings, while multiple LLM‑driven Purple Agents connect via the A2A protocol to debate, vote, and execute actions under partial information.
The project provides a research‑oriented evaluation pipeline that stress‑tests core agentic capabilities such as probabilistic reasoning, persuasion, deception and detection, role‑conditioned behavior, and narrative consistency in noisy multi‑agent settings. Each game produces both quantitative metrics (win‑rate by role, survival, vote accuracy, ELO adjusted by opponent strength) and qualitative scores via an LLM‑as‑a‑Judge component inspired by G‑Eval, which rates agents along multiple dimensions (reasoning quality, persuasive power, deceptive skill, adaptability, and consistency) and generates textual justifications. Technically, the system is implemented in Python with FastAPI for Green/Purple A2A endpoints, uses OpenAI LLMs (configurable models such as gpt‑4o‑mini) for both policies and judge, exposes a JavaScript/HTML/CSS leaderboard UI, and is fully containerized with Docker/Docker Compose and automated via GitHub Actions CI to run tournaments and update metrics reproducibly.
The project was developed collaboratively by Daniel Santiago Sandoval Higuera, Sadid Alexis Romero Mahecha, Julian Anibal Henao Garcia, and Andres Felipe Garcia Sanchez.
Video
Links
FastAPI-based multi-agent social deduction framework benchmarks AI reasoning via A2A.
AgentBeats ELO leaderboard for Werewolf AI agents using A2A protocol.
Tech stack
- GPT-4oGPT-4o (omni) is OpenAI's flagship multimodal model: it delivers GPT-4 intelligence with native, real-time processing across text, audio, and vision.This is GPT-4o, OpenAI’s 'omni' model: a single neural network natively handling text, audio, and image inputs and outputs. It matches GPT-4 performance on English text and code, but surpasses it on non-English language, vision, and audio benchmarks. The speed is a major upgrade: it achieves human-level responsiveness in voice, with an average response time of 0.32 seconds (a significant jump from GPT-4’s 5.4 seconds). Developers get a 128K token context window and a model that is more cost-efficient than its predecessor, making high-intelligence, real-time applications viable.
- PythonPython: The high-level, general-purpose language built for readability, powering everything from web backends to advanced machine learning models.Python is the high-level, general-purpose language prioritizing clear, readable syntax (via significant indentation), ensuring rapid development for any team . Its ecosystem is massive: use it for robust web development with frameworks like Django and Flask, or leverage its power in data science with libraries such as Pandas and NumPy . The Python Package Index (PyPI) provides thousands of community-contributed modules, offering immediate solutions for tasks from network programming to GUI creation . The language is actively maintained by the Python Software Foundation (PSF), with the stable release currently at Python 3.14.0 (as of November 2025) .
- FastAPIFastAPI is a modern, high-performance Python web framework for building APIs with automatic OpenAPI documentation.FastAPI is a robust, high-speed Python web framework: it is built on Starlette (for async capabilities) and Pydantic (for data validation and serialization). Leveraging standard Python 3.8+ type hints, the framework automatically generates interactive API documentation (Swagger UI/ReDoc) and enforces data validation, effectively reducing developer-induced errors by an estimated 40%. This architecture delivers performance on par with Node.js and Go, significantly increasing feature development speed (up to 300% faster). It is production-ready, fully supporting OpenAPI and JSON Schema standards for all API specifications.
- DockerDocker is the open-source platform that packages applications and dependencies into standardized, portable containers for consistent execution across any environment.Docker is the industry-standard containerization platform, enabling developers to build, ship, and run applications efficiently. It uses the Docker Engine (the core runtime) to create lightweight, isolated environments called containers: these units bundle an application’s code, libraries, and configuration. This self-contained approach guarantees consistency, eliminating the 'it works on my machine' problem across development, testing, and production environments (local workstations, cloud, or on-premises). Docker debuted in 2013 and now serves over 20 million developers monthly, simplifying complex workflows like CI/CD and microservices architecture by leveraging tools like Docker Hub for image sharing and Docker Compose for multi-container applications.
- OpenAI APIOpenAI API: Your direct gateway to cutting-edge AI models (GPT-4o, DALL-E 3, Whisper), enabling scalable, multimodal intelligence integration into any application.The OpenAI API provides authenticated, programmatic access to a powerful suite of generative AI models. Developers leverage REST endpoints and official libraries (Python, Node.js) to integrate capabilities like advanced text generation (GPT-4o), image creation (DALL-E 3), and speech-to-text transcription (Whisper). This platform is engineered for scale, supporting millions of daily requests for tasks from complex reasoning to real-time customer support agents, ensuring your application gets reliable, state-of-the-art intelligence.
- AgentAn Agent is an autonomous software system (often LLM-driven) that perceives its environment, makes goal-directed decisions, and executes multi-step actions without continuous human oversight.This is the next-generation automation layer: a software entity capable of independent operation. Agents utilize a core perception-action loop, integrating memory and planning systems to break down complex objectives (e.g., 'book a full business trip') into actionable subtasks. They are distinguished by their autonomy, moving beyond simple chatbots or rule-based scripts. Modern examples like AutoGPT and Devin AI demonstrate their capacity to use external tools (APIs, web browsers) and adapt their strategy in real-time, delivering a step-change in efficiency for enterprise workflows and complex data analysis.
- A2AA2A (Agent2Agent) is the open communication protocol enabling secure, cross-platform collaboration between specialized AI agents.A2A, or Agent2Agent Protocol, is the open standard for AI agent interoperability, launched in April 2025 by Google and over 50 technology partners (e.g., Atlassian, SAP, MongoDB). It defines a standardized language for autonomous agents to discover capabilities, delegate tasks, and securely exchange information. The protocol leverages established web standards like HTTPS and JSON-RPC 2.0 to ensure enterprise-grade security and reliability. This framework breaks down silos, allowing a specialized fraud-detection agent, for example, to seamlessly coordinate with a customer-service agent, automating complex, multi-step workflows across diverse vendor ecosystems.
- HTTPHTTP (Hypertext Transfer Protocol) is the application-layer protocol driving all web data exchange: It dictates the client-server request-response cycle using specific methods (e.g., GET, POST) and status codes (e.g., 200, 404).HTTP (Hypertext Transfer Protocol) operates as the foundational application-layer protocol for the World Wide Web, managing the client-server request-response cycle. A client (browser) sends a request (e.g., a GET or POST method), and the server replies with a response, including a three-digit status code (like 200 OK for success) and the resource body. Though inherently stateless, the protocol leverages cookies (RFC 6265) to maintain session state across requests. We've seen major evolution: HTTP/1.1 introduced persistent connections, while HTTP/2 and the emerging HTTP/3 (based on QUIC) focus on stream multiplexing and efficiency for faster, modern web performance. This structure of methods, headers, and status codes is the universal language for web data transfer.
- OpenAI LLMsThe premier suite of Large Language Models (LLMs), including the foundational GPT-4 and the advanced reasoning-focused 'o' series, setting the global standard for generative AI applications.OpenAI LLMs represent the cutting edge of generative AI, headlined by the powerful GPT-4 and the specialized 'o' series models (e.g., o1, o4-mini). These models handle complex tasks: text generation, code synthesis, and advanced reasoning, often exceeding human-level performance on benchmarks like the AIME math exam. Developers access this technology via the OpenAI API, leveraging models like GPT-3.5 for cost-optimized solutions and GPT-4 for high-fidelity, multimodal applications. The continuous introduction of new architectures, such as the chain-of-thought training in the 'o' models, consistently pushes the boundary on AI capability and efficiency for enterprise and consumer products.
- eErlang is a functional programming language designed for building massive, real-time systems with 99.9999999% availability.Engineered by Ericsson in 1986 to power telecommunications infrastructure, Erlang excels at managing millions of concurrent processes without breaking a sweat. It uses a lightweight actor model and the BEAM virtual machine to ensure that if one process fails, the rest of the system stays upright (a philosophy known as let it crash). Today, it is the backbone for high-stakes platforms like WhatsApp, which handles billions of messages daily, and Goldman Sachs’ high-frequency trading systems. If your project demands hot-swappable code and zero downtime, Erlang is the industry standard for reliability.
- gpt‑4o‑mini) for both player policies and the judgeGPT-4o-mini serves as a high-speed, cost-effective engine for simultaneous agent policy execution and automated judging.OpenAI's GPT-4o-mini delivers 128k context windows and a 15-M-token-per-minute rate limit, making it the optimal choice for multi-agent simulations. By deploying this model for both player policies and the judge, developers achieve sub-second latency and a 99% cost reduction compared to GPT-4o. This unified architecture ensures consistent reasoning patterns across the game loop: the model generates strategic moves for players while simultaneously providing structured evaluations of those actions with high reliability (scoring 82% on the MMLU benchmark).
Finding related talks...
Compose Email
Sending...
Email preview
Loading recent emails...