Rationale

Mission-critical bot infrastructure and analytics dashboards use the same data, but have very different hard requirements.

For dashboard use cases,

• Low-latency queries are not a hard requirement; a dashboard taking several seconds to load is usually tolerable.
• Returning stale or cached data or data from slightly different block heights is tolerable. A user can refresh the page, or the difference is not visually noticeable.
• Failure to sync on-chain data (getting “stuck” at a block) is merely a user inconvenience rather than a possible loss of funds.

TheGraph, GoldSky, Dune and other query solutions tend to be satisfactory for analytics and dashboard purposes but not bot infrastructure. They do not provide low-latency, self-hosted, always-up-to-date query solutions where failure to sync is unacceptable. Opportunities to deploy on upcoming L1 or L2 networks are limited to what an external query solution supports, or what they are willing to add. This is frustrating when an RPC node is always available on any given network – an EVM network must have at least one RPC provider for any dApp to function.

Not all query solutions have full application-level SQL support, or any SQL support at all. This forces anyone using such a query solution to discard decades of tooling, knowledge, production deployment experience, and flexibility built around the existing, battle-tested SQL databases that all Web2 applications use today. We see little reason to abandon this wealth. Although EVM data is coming from a different source in a different format, the business and technical problems remain the same as Web2: “I want my application to reliably act on queries to provide business value”.

Most teams avoid the issues of commercial query systems by instead painstakingly fetching and decoding on-chain events directly within the bot using the same programming language without the help of any database or query solution. Some implementations simply store all state in memory which causes a complete resync if the bot restarts. This is unacceptable in production settings. These approaches limit future scalability since each bot has its own ad-hoc query logic.