2026-01-24

Today I slept in until 11 o’clock and felt completely relaxed. It’s been a long time since I slept so freely.

1. What I did today:

   1. Deployed a new version of node-unit. I felt confident pushing it because I had thorough end-to-end tests. Specifically, I spun up a TimescaleDB (PostgreSQL 17) using Docker, then launched two node-unit instances, and inserted 21 @yuants/portal entries into the database to test. The final result converged to a state where each node-unit took half of the deployments.

      This test basically verifies that when a bunch of unowned deployments appear, and two node-units come online, you can observe them taking turns claiming deployments. If anything is missing, it’s one that actually occupies CPU/memory workload, and another scenario where a node-unit goes offline for some reason.

   2. Used the new multi-agent version of legionmind in Yuan to solve the issue of vendor-gate earn account outputting account flows. I had the agent first create documentation using legion, producing the following documents:

      .legion/tasks/vendor-gate
      ├── context.md
      ├── docs
      │   ├── api-doc.md
      │   ├── pr-body.md
      │   ├── report-walkthrough.md
      │   ├── rfc.md
      │   ├── spec-bench.md
      │   ├── spec-dev.md
      │   ├── spec-obs.md
      │   └── spec-test.md
      ├── plan.md
      └── tasks.md
      

      Feels like a decent workflow. However, there is some conflict between my new multi-agent system and the original legionmind’s documentation writing. I should carefully consider the boundaries of each thing—for example, the standards for how each document type should be written should be placed into separate skills, and legionmind should be a description of the workflow. Each agent should be able to load a few smaller skills to assist them in their work.

      Another issue is that during its first run, it made a mistake: it output the account flows into =account-actions-with-credential.ts=. This happened because I asked it to reference vendor-okx to complete the earn account integration—I did that because currently only OKX’s earn account is also connected as an account. But the AI also picked up some outdated practices from that. The current exchange integration standard is to publish all accounts via =provideExchangeServices=, not using =provideAccountActionsWithCredential= to connect accounts.

      This knowledge is not available to a brand-new AI agent. How should such knowledge be modeled? How can I provide this project context as an external brain for the AI agent? This is a question worth pondering deeply; I’ll need to think it over tomorrow.

   3. In the afternoon, I cooked to entertain sy’s friends—it exhausted me. So tomorrow I’ll continue working.

2. Thoughts:

   • As mentioned above, I need to carefully consider how to compactly design an external brain for an AI agent. The simplest approach could start with a set of AGENT.md files. I’ve tried this before, but maintaining those documents itself has a fairly high overhead. Distinguishing valuable experiences from noise is a difficult problem. Currently, memory is similar to other prompts, except the agent might have its own loop to update memory. The most important thing is still how to measure the results of the AI agent’s work.

   • Regarding the previous point, I came across an interesting article. Let me summarize it in my own words: First, evaluation of an agent’s single-step work can be categorized into several types:

     1. Static tool eval: compilers, linters, unit tests, e2e tests
     2. Model eval: using another LLM to judge based on our defined prompts
     3. Human eval: I judge

     Then, systematic evaluation of an agent has two types:

     1. Capability-oriented: Answers what the agent can do? And the pass rate might be low, e.g., using legion to gradually execute larger, harder tasks—like exploring a new frontier.
     2. Regression-oriented: Does it still retain previously gained abilities? For example, repeatedly testing certain tasks to ensure stable performance.

     Once a new capability is introduced, it should transition from capability-oriented to regression-oriented.

     The article also mentions two important metrics: pass@K and pass^K

     • pass@k: At least one success in k attempts. More attempts → higher probability of at least one success. Use case: You only care about “finding at least one viable solution.”

     • pass^k: All k attempts must succeed. More attempts → harder to maintain consistency. Use case: Users expect a reliable production agent every time.

     FYI: Reference article

   • My energy is still a bit low. I worked for a while in the afternoon and then cooked dinner, and felt quite tired. When will I become like CZ and not need sleep?

3. Plans for tomorrow:

   1. Think about the eval agent model, and continue iterating on the multi-agent system.
   2. Cluster security issues—must be tackled.
   3. Legion-github-bridge.