Building CrewFactory: My Experience Building a Multi-Agent System for the Qwen Cloud Hackathon
CrewFactory was born out of a specific challenge: the Qwen Cloud hackathon, Agent Society track. The brief asked for a system where multiple agents with different capabilities would collaborate through task division, dialogue, and negotiation to solve complex problems – and demonstrate, with data, that this teamwork outperformed a single agent doing everything on its own.
That phrase – “measurable efficiency gain over a single agent” – ended up being the compass for the entire project. Every architectural decision, from how agents delegate tasks to how they resolve disagreements, had to answer that question.
System Architecture
CrewFactory
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────┐ ┌───────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │ Chat │ │ Teams │ │ Projects │ │ Laboratory │ │
│ │ Sessions │ │ (Negot.+ │ │ (Git │ │ (A/B Experiment │ │
│ │ (WS/HTTP)│ │ Orchest.)│ │ repos) │ │ Runner + Judge) │ │
│ └────┬─────┘ └─────┬─────┘ └────┬─────┘ └────────┬─────────┘ │
│ │ │ │ │ │
│ ┌────┴────────────────┴───────────────┴───────────────────┴──────┐ │
│ │ Session Manager (Singleton) │ │
│ │ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ Metadata│ │ Session │ │ Prompt │ │ Tool │ │ │
│ │ │ Store │ │ Lister │ │ Builder │ │ Factory │ │ │
│ │ └─────────┘ └─────────┘ └──────────┘ └──────────────┘ │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────┼────────────────────────────────┐ │
│ │ Agent Runtime (Vendored) │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ Agent │ │ Tools │ │ Memory │ │ Compaction │ │ │
│ │ │ (ReAct) │ │ (bash, │ │ (notes, │ │ (zap/sum- │ │ │
│ │ │ │ │ vision, │ │ sessions)│ │ mary) │ │ │
│ │ │ │ │ MCP...) │ │ │ │ │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Permission Engine │ Approval Manager │ Security (AES-256) │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
I started with the bare minimum: a chat with real-time streaming connected to an agent runtime, with authentication and sessions. The boring-but-necessary foundation to iterate on. Weeks later, that foundation had grown into a full multi-agent orchestration platform.
The tech stack: Bun + Hono on the backend, React 19 + Vite + TypeScript strict + Tailwind CSS v4 on the frontend, with WebSocket as the backbone for all real-time communication. No database – localStorage on the client, filesystem on the server.
Task Division: From a Single Prompt to a Team With Roles
The first thing I had to solve for the track was literally: how does a group of agents decide who does what? The first version was naive – one agent received the full task and solved it alone. That’s not collaboration, it’s just delegation in disguise.
The real solution came with task decomposition. The agent uses a tool called decompose_tasks that takes an objective and breaks it into a dependency graph. It’s not magic prompt engineering: it’s a tool with a strict Zod schema that the agent must respect:
// packages/shared/src/schemas.ts -- Task schema validated with Zod
const TaskSchema = z.object({
id: z.string(), // "t1", "t2", ...
title: z.string(), // "Build authentication API"
prompt: z.string(), // Self-contained instructions
status: z.enum(["pending", "running", "done", "failed"]),
log: z.string(),
depends_on: z.array(z.string()).default([]), // DAG: ["t1", "t2"]
estimated_steps: z.number().optional(),
});
const TaskRunnerStateSchema = z.object({
tasks: z.array(TaskSchema),
currentTaskId: z.string().nullable(),
status: z.enum(["idle", "decomposing", "running", "paused", "completed", "failed"]),
});
The tool builds a deterministic prompt that asks the LLM for only the task structure – no conversation, no unnecessary context:
// apps/server/src/core/tools/decompose-tool.ts
function buildDecomposePrompt(objective: string, mode: string): string {
const modeInstruction =
mode === "dag"
? "Where possible, express parallelism by specifying which tasks each step depends on."
: "Tasks must be strictly sequential.";
return [
`Objective: "${objective}"`,
`Decompose this into at most ${maxTasks} tasks. ${modeInstruction}`,
`CRITICAL: Your response must end with ONLY a valid JSON array. No prose after.`,
"Format:",
`[{"id":"t1","title":"...","prompt":"...","depends_on":[],"estimated_steps":3}]`,
`Rules: IDs must be "t1","t2"..., prompt must be self-contained. Only JSON.`,
].join("\n");
}
The detail that took the longest to tune: at first, planning went through a full agent session – an LLM call with all the conversation context, skills, history, memories. It was slow and expensive in tokens. I replaced it with streamSimple(): a direct call that only runs “divide this” without the weight of a full session. Same result, a fraction of the cost and latency.
For execution, the delegation pattern is explicit and typed:
// apps/server/src/core/tools/delegate-tool.ts
export function createDelegateTaskTool(opts: DelegateTaskOptions) {
return {
name: "delegate_task",
description: `Delegate a task to another agent, project, team, or session.
Returns a structured summary instead of the full conversation log.`,
parameters: {
type: "object",
properties: {
targetType: { type: "string", enum: ["agent", "project", "team", "session"] },
targetId: { type: "string", description: "Agent ID, project UUID, or team ID" },
task: { type: "string", description: "The prompt to send to the target" },
model: { type: "string", description: "Optional explicit model override" },
autonomyMode: {
type: "string",
enum: ["read-only", "standard", "autonomous"],
},
},
required: ["targetType", "targetId", "task"],
},
// ... execution creates isolated session, forwards events via WS,
// returns structured envelope: { status, executive_summary, artifacts, risks }
};
}
Each agent executes its part in its own isolated session, with its own history and context. No shared memory that could contaminate another agent’s work. When an agent needs another’s result, it explicitly awaits it and resumes execution as soon as it arrives – instead of constantly polling.
Task Decomposition Flow
[Objective] ──> decompose_tasks() ──> [t1: setup] ──> [t2: backend, depends_on: t1]
└──> [t3: frontend, depends_on: t1]
└──> [t4: deploy, depends_on: t2, t3]
Dialogue and Negotiation: When Agents Disagree
This is the part that connects most with the spirit of the track. It’s not enough for agents to divide work among themselves; they need to be able to discuss things when there’s disagreement.
For this I built “negotiation teams.” The concept is simple: multiple agents debate a proposal over rounds. But the implementation has to be rigorous – not a pretty prompt, but an engine running against a protocol with explicit rules:
// packages/shared/src/schemas.ts -- Negotiation protocol
const NegotiationProtocolSchema = z.object({
agreementPattern: z.string(), // regex: /ACUERDO\s+ALCANZADO/
counterPattern: z.string().optional(), // regex: /CONTRAPROPUESTA/
rejectPattern: z.string().optional(), // regex: /RECHAZO/
maxRounds: z.number().int().min(1).max(20).default(3),
quorumThreshold: z.number().min(0).max(1).default(0.51),
arbiterAgentId: z.string().optional(), // tiebreaker
});
The NegotiationRunner is not an LLM talking to itself – it’s a real execution loop:
// apps/server/src/teams/negotiation/negotiation-runner.ts
export class NegotiationRunner {
async dispatch(username: string, teamId: string, userMsg: TeamMessage): Promise<void> {
const team = teamStore.getTeam(username, teamId);
if (!team) throw new Error("Team not found");
// Gate: ignores trivial messages ("hola", "ok", "si") to avoid triggering debate
if (!isSubstantiveMessage(userMsg.content)) {
teamStore.appendMessage(username, teamId, { /* guidance message */ });
return;
}
const controller = new AbortController(); // Cancels the entire chain if aborted
// ... executes rounds, evaluates consensus, escalates to arbiter if needed
}
}
A typical round loop:
Negotiation Rounds Flow
[User Prompt] ──> Round 1: Agent A (proposes), Agent B (objects), Agent C (supports)
└──> Consensus? NO (quorum < 0.51) ──> Round 2
└──> Round 2: Agent B (adjusts), Agent A (accepts), Agent C (accepts)
└──> Consensus? YES ──> [Final result]
If maxRounds = 3 and no consensus ──> [Arbiter resolves]
I had to put hard limits on this quickly: without a cap on rounds and delegation chain depth, a disagreement between agents can turn into an infinite loop where they keep citing each other without reaching any conclusion. The solution was an explicit circuit breaker with maxRounds and an isSubstantiveMessage() rule that filters empty responses – an agent doesn’t reply out of politeness if it has nothing technical to contribute.
Measuring Efficiency: The Requirement Almost Everyone Ignores
It’s tempting to build the multi-agent system and assume it’s better because it sounds more sophisticated. To avoid that, I built an experiment laboratory: the same task runs in three variants and an LLM judge compares them blindly:
Experiment: "Build a REST authentication API with rate limiting"
┌─────────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ Variant A (Baseline) │ │ Variant B (Debate │ │ Variant C (Debate │
│ Single Agent │ │ without Leader) │ │ with Leader) │
│ │ │ │ │ │
│ ┌───────────────────┐ │ │ ┌───┐ ┌───┐ ┌───┐ │ │ ┌───┐ (Leader) │
│ │ Dev Agent │ │ │ │ A │ │ B │ │ C │ │ │ │ L │---> member A │
│ │ (does everything │ │ │ └───┘ └───┘ └───┘ │ │ └───┘---> member B │
│ │ alone) │ │ │ Rounds of debate │ │ Leader arbitrates │
│ └───────────────────┘ │ │ without hierarchy │ │ the debate │
│ │ │ │ │ │
└───────────┬─────────────┘ └─────────┬────────────┘ └──────────┬───────────┘
│ │ │
└──────────────────────────┼──────────────────────────┘
│
┌────────────────┴────────────────┐
│ LLM Judge (blind evaluation │
│ Alpha vs Beta vs Gamma) │
│ │
│ Criteria: quality, efficiency, │
│ negotiation │
└────────────────┬─────────────────┘
│
┌──────────┴──────────┐
│ Composite Score │
│ GlobalScore = │
│ 50% Quality + │
│ 30% Efficiency + │
│ 20% Negotiation │
└──────────────────────┘
The schema for a variant result captures everything that matters:
// packages/shared/src/schemas.ts -- Experiment variant result
const VariantRunResultSchema = z.object({
status: z.enum(["completed", "failed"]),
durationMs: z.number(),
tokensIn: z.number(),
tokensOut: z.number(),
negotiationRounds: z.number().optional(),
escalationsToLeader: z.number().optional(),
agreementReached: z.boolean(),
finalOutput: z.string(),
divergenceEventsCount: z.number().optional(),
arbitrationRoundsCount: z.number().optional(),
protocolActivationRate: z.number().optional(),
scores: z.object({
taskQuality: z.number(),
efficiencyScore: z.number(),
negotiationScore: z.number().optional(),
globalScore: z.number(),
judgeReasoning: z.string().optional(),
criteriaScores: z.record(z.number()).optional(), // per-criterion scores
efficiencyDetail: z.object({
numAgents: z.number(),
effectiveRounds: z.number(),
adjustedDuration: z.number(),
adjustedTokens: z.number(),
}).optional(),
}),
});
The composite scoring formula (commented in the engine code):
// apps/server/src/laboratory/scoring.ts -- global score composition
// GlobalScore = (Quality * 0.5) + (Efficiency * 0.3) + (Negotiation * 0.2)
//
// Where Efficiency penalizes multi-agent overhead relative to baseline:
// Efficiency = 100 - (overheadPenalty)
// overheadPenalty adjusted by log(numAgents) to avoid unfairly
// penalizing teams with more members
The LLM judge runs in an isolated namespace and streams its reasoning live:
// apps/server/src/laboratory/judge.ts
const JudgeResponseSchema = z.object({
Alpha: OutputEvaluationSchema, // Single Agent variant
Beta: OutputEvaluationSchema, // Multi-No-Leader variant
Gamma: OutputEvaluationSchema, // Multi-With-Leader variant
});
// The evaluation is streamed in real-time via WebSocket to the frontend
// so the user can see the judge's reasoning as it happens
This laboratory ended up being indispensable for my own design judgment: every time I added a coordination layer, I could see whether it added value or not. Several times the answer was “more overhead” and I had to simplify.
Observability: Seeing What the Agents Are Doing
Once you have multiple agents in parallel, the urgent question is: what is each one doing? The data model behind the session kanban:
// apps/server/src/core/session/session-lister.ts
type SessionListItem = {
id: string;
name: string;
status?: "active" | "streaming" | "task-running" | "sleeping";
projectName?: string;
agentId?: string;
teamId?: string;
experimentId?: string;
isExecution?: boolean;
totalTokens?: number;
toolCallCount?: number;
durationMs?: number;
errorCount?: number;
turnCount?: number;
archived?: boolean;
// ... 20+ more fields
};
// Server-side filtered query:
interface SessionListQuery {
search?: string;
agentId?: string;
projectName?: string;
status?: string;
from?: string; // date range
to?: string; // date range
isExecution?: boolean;
sortBy?: string;
sortDir?: string;
}
This powers the 3-column board (Idle / Working / Done), the status dots in the sidebar (green = active, gray = sleeping), and the vertical timeline inside each chat showing thoughts, tool calls, and responses with their duration.
Layered Prompt System
Every agent in the system doesn’t receive a flat prompt – it receives a conditional composition of 4 independent layers, resolved at runtime based on context:
// apps/server/src/core/prompts/prompt-assembly.ts
type PromptAssemblyMode =
| "standard-session" // Global or project chat
| "channel-member" // Agent inside a channel
| "team-orchestration" // Orchestration team leader
| "debate-stateless" // Negotiation debate -- no memory or tools
| "agent-startup" // Standalone programmatic agent bootstrap
| "subagent-spawn" // Executor subagent
| "experiment-member"; // Agent inside a laboratory experiment
// Each mode injects different fragments:
// Identity -> Role -> Instance -> Protocol
// And appends conditional instructions:
// Environment, HTML Preview, AG-UI, Persistent Memory,
// Subagent Delegation, Task Delegation
This allows the same agent to behave differently whether it’s in a 1:1 chat, a negotiation team, or as an executor subagent – without rewriting its base prompt.
Factory Tool: One Tool to Rule Them All
Instead of having separate skills (factory-projects, factory-agents, etc.) that the agent calls via bash + curl, I created a unified meta-tool. The agent does everything with a single typed, runtime-validated call:
// apps/server/src/core/tools/factory-tool.ts
// Supported entities and actions:
// manage_factory(entity, action, id?, params?)
//
// Entities: agents | projects | teams | channels | skills | sessions | env | experiments
// Actions: list | get | upsert | delete | send | ...
// Runtime validation via self-documenting contracts:
function validateParams(entity: string, action: string, id: string, params: any) {
const contract = FACTORY_CONTRACTS[entity];
const actionContract = contract.actions[action];
for (const [paramName, paramDef] of Object.entries(actionContract.params)) {
if (paramDef.required && !params[paramName]) {
return `Parameter "${paramName}" is required for "${action}" on "${entity}".`;
}
// type checking: string, number, boolean, enum
}
}
// Contracts are exposed via GET /api/factory/contracts
// The agent can auto-discover schemas without hardcoding anything
This eliminated ~500 lines of factory-* skills that were doing curl to the API with fragile, formatting-error-prone commands.
Permissions and Security
The permission model separates three execution levels per session:
┌──────────────┐ ┌───────────────┐ ┌──────────────┐
│ Read-Only │ │ Standard │ │ Autonomous │
│ │ │ │ │ │
│ write: NO │ │ write: ASK │ │ write: YES │
│ edit: NO │ │ edit: ASK │ │ edit: YES │
│ bash: NO │ │ bash: ASK │ │ bash: YES │
│ │ │ │ │ │
│ Explorer │ │ Builder │ │ Autonomous │
│ subagents │ │ subagents │ │ subagents │
└──────────────┘ └───────────────┘ └──────────────┘
Subagents inherit their parent’s restrictions, and interactive approvals have a global overlay with a 60-second countdown (auto-deny if no response). The permission engine follows a deny-first policy: it first blocks destructive patterns (fork bombs, rm -rf on critical directories, curl | bash), then evaluates user rules, and finally decides whether to ask or execute directly based on the mode.
AES-256-GCM encryption at rest protects sensitive config files (auth.json, env.json), deriving the key from JWT_SECRET. And a bash output filter sanitizes stdout and stderr by masking user secrets with ***hidden*** before they reach the UI.
WebSocket Infrastructure
A single WebSocket shared across the entire application, not one per hook or view:
// apps/server/src/ws/factory.ts
// Factory pattern: each connection receives a unique wsId via crypto.randomUUID()
// Registry pattern: Maps of userSockets, sessionSockets, channelSockets, teamSockets
// with explicit cleanup on view change, no WeakMap
// Auto-subscribe on prompt send: avoids race condition
// where the message arrives before the socket is subscribed to the session
// Cookie-based auth in onOpen: validates httpOnly cookie
// with fallback to synchronous SQLite lookup for programmatic tokens
The client uses exponential backoff with jitter, an offline queue limited to 50 messages, and automatic reconnection with streaming state recovery.
Key Takeaways
The hackathon requirement – real collaboration, with measurable improvement – ended up being the best design filter I could have had. Every time a feature didn’t help divide tasks better, resolve a disagreement, or demonstrate a real gain, I dropped it.
If you’re building something similar:
-
Don’t start with the full multi-agent system. Build the minimum coordination needed to resolve a real disagreement.
-
Measure against a baseline. The experiment laboratory isn’t a luxury – it’s the only way to know if you’re adding value or just complexity.
-
Hard limits are not optional. Without
maxRounds,maxDepth, and circuit breakers, a disagreement between agents is a silent infinite loop. -
One typed tool > ten skills with curl.
manage_factoryreplaced ~500 lines of fragile bash with a single runtime-validated call. -
Observability is not a nice-to-have. Without the session kanban and status dots in the sidebar, debugging a multi-agent bug is like finding a needle in a dark haystack.
The full code is at github.com/themikehage/crewfactory.
