Benchmark: Apollo Federation Gateway v1 vs v2 vs WunderGraph vs mercurius-js

Jens Neuse, CEO & Founder of WunderGraph

Apollo just released their newest version of the Apollo Federation Gateway (v2), so I was curious how it performs against v1 and the WunderGraph implementation.

Both Apollo Gateway v1 and v2 are implemented using NodeJS, the WunderGraph Gateway is written in Go.

So far, WunderGraph is the only Federation implementation aside from Apollo. It was brought to my attention that there's also mercurius-js that implements the Federation specification. It's built on top of Fastify (NodeJS). Thanks to @matteocollina for pointing out! If you know any other implementation, please let me know!

TLDR#

WunderGraph achieves up to 271x (132x) more requests per second vs. Apollo Gateway v1 (v2), 99th percentile latency is 292x (54x) lower. Apollo Gateway v2 achieves 2x more rps than v1, 99th percentile latency is 5.6x slower than v1. While Apollo Gateway v1 had problems with timeout errors, v2 solved this problem.

Apollo Federation with Subscriptions#

In contrast to Apollo Gateway, WunderGraph supports Subscriptions. This is possible because Go has green threads (goroutines) which enable services to scale easily across all cores of a computer. Each Subscription can run in its own goroutine, which only takes up a few kilobytes of memory in stack size, so scaling this solution is quite efficient.

That said, getting the architecture right for federated subscriptions is a complex problem. Most if not all GraphQL server implementations "interpret" GraphQL Operations at runtime. This means, they parse the Operation into an AST at runtime and work through this AST to resolve the Operations.

WunderGraph takes a different approach. We've built a Query Compiler that divides resolving a GraphQL Operation into multiple stages. On a high level, we differentiate between the planning, and the execution phase. During planning, we evaluate the AST and build an optimized execution Plan, hence the "Query Compiler". This execution Plan can be cached, which makes this approach very efficient. But efficinecy is not everything. More importantly, this approach allows us to solve complex problems like resolving federated GraphQL Operations with a multi step compiler, combined with an optimized execution engine.

Btw. this Query Compiler and Execution engine is open source under a MIT license. It's used by more and more companies in production. We're very proud that the developers of Khan Academy joined the ranks of maintainers recently.

One last word on open source, graphql-go-tools, the library we're building WunderGraph upon, has some amazing contributors. Amongst them is Vasyl Domanchuk from the Ukraine, he contributed the DataLoader implementation that plays an important role in making the engine so fast. This implementation solves the N+1 problem when resolving nested federated GraphQL Operations.

Benchmarking methodology#

I've setup the basic Federation demo, more info at the end of the post. For benchmarking, I've used the cli "hey" with a concurrency of 50 over 10 seconds.

Results - Apollo Federation Gateway vs WunderGraph#

Requests per second (small Query)#

Requests per second (large Query)#

Latency (small Query)#

Latency (large Query)#

Observations#

Apollo Gateway v1 always has timeout errors under high load. The newer version (v2) fixed this problem. However, v2 seems to be not yet mature as requests per second ranged from 10 to 400 in some test runs.

I've also found that Apollo now configures their gateway to use Apollo Studio by default. As an alternative, they provide you a code snipped to use curl. Additionally, there's a link to the docs to enable the Playground again, running on your local machine:

import { ApolloServer } from "apollo-server";
import {
  ApolloServerPluginLandingPageGraphQLPlayground
} from "apollo-server-core";

const server = new ApolloServer({
  typeDefs,
  resolvers,
  plugins: [
    ApolloServerPluginLandingPageGraphQLPlayground(),
  ],
});

mercurius-js is written in NodeJS, similarly to Apollos gateway. For the server, it's using the Fastify framework, which is visible from the results. On small payloads, it comes out on top of Apollo by almost 5x in terms of rps. It only seems that it struggles with the large Query. This could be either processing more data in general or due to the higher amount of network requests, the gateway has to make. Something must be going that here, which makes mercurius fall behind Apollo on the large Query.

Conclusion#

NodeJS is still not comparable in terms of performance vs. Golang. While the new version of the Apollo Gateway doesn't throw timeout errors anymore, it's visible that it doesn't scale well when GraphQL Operations become deeply nested.

Comparing the latencies of Apollo v2 for the small and large payload, it's observable that the numbers skyrocket when Operations become more nested.

WunderGraph on the other hand is not yet saturated with the workload. we could probably increase the nesting further until it has to give up.

If you want a fast Federation compatible Gateway solution, WunderGraph can save you a lot of money for hosting while increasing the security of your API.

What makes the difference?#

It's mainly two things. For one, WunderGraph is written in Go, a language that is much more capable when it comes to concurrent workloads like implementing an HTTP server. The second aspect is the architecture of WunderGraph. Instead of "interpreting" Operations, WunderGraph works with a Query Compiler that prepares the execution of an Operation at deployment time, removing all the complexity of working with the GraphQL AST at runtime.

If you want to learn more on this topic, have a look at the overview on the Query Compiler.

Demo Setup#

WunderGraph: https://github.com/wundergraph/wundergraph-demo

Apollo: https://github.com/StevenACoffman/federation-demo

In both cases, I was using the upstreams implemented using gqlgen to eliminate perfomance issues on the upstreams.

If you want to reproduce the results, just clone the repos and use hey or similar tools to benchmark yourself.

Test Query Small#

query {
    topProducts {
        upc
        name
        price
        reviews {
            id
            body
            author {
                id
                name
                username
            }
        }
    }
}

Test Query Large#

query {
    topProducts {
        upc
        name
        price
        reviews {
            id
            body
            author {
                id
                name
                username
                reviews {
                    id
                    body
                    author {
                        id
                        name
                        username
                        reviews {
                            id
                            body
                            product {
                                inStock
                                name
                                price
                                shippingEstimate
                                upc
                                weight
                                reviews {
                                    body
                                    id
                                    author {
                                        id
                                        name
                                        reviews {
                                            body
                                            author {
                                                id
                                                name
                                                username
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

What to read next

This is a curated list of articles that I think you'll find interesting.

• In the WunderHub Announcement, I talk about how WunderHub will change the way we share and collaborate on APIs. It allows you to share APIs like npm packages.
• How automating API integrations benefits your business is dedicated to C-level executives who want to learn more about the business benefits of automating API integrations.
• Another interesting topic is to JOIN APIs without Schema Stitching or Federation, just by using a single GraphQL Operation
• For those interested in the most common GraphQL Security vulnerabilities, I suggest to read about them and how WunderGraph helps you to avoid them.
• A classic post but still relevant is I believe that GraphQL is not meant to be exposed over the Internet. It's a controversial topic and many misunderstand it. But think about it, why is HTTP not mentioned a single time in the GraphQL specification?
• One very common problem of using GraphQL is the Double Declaration Problem, the problem of declaring your types over and over again. This post explains that it's even more complicated than just double declaration and how we can solve it.
• The Fusion of GraphQL REST and HTTP/2 is a very long post, probably too long for a blog post. But if you're interested in a deep dive on the motivations behind creating WunderGraph, this is the post for you.

About the Author

Jens Neuse, CEO & Founder of WunderGraph

Jens has experience in building native apps for iOS and Android, built hybrid apps with Xamarin, React Native and Flutter, worked on backends using PHP, Java and Go. He's been in roles ranging from development to architecture and led smaller and larger engineering teams.

Throughout his whole career he realized that working with APIs is way too complicated, repetitive and needs a lot more standardization and automation. That's why he started WunderGraph, to make usage of APIs and collaboration through APIs easier.

He believes that businesses of the future will be built on top of collaborative systems that are connected through APIs. Making usage, exploration, sharing and collaboration with and through APIs easier is key to achieve this goal.

Follow and connect with Jens to exchange ideas or simply participate in his feed of thoughts.

Follow Jens on TwitterConnect with Jens on Linkedin

Comments

Discuss on TwitterDiscuss on HackerNewsDiscuss on RedditDiscuss on LinkedInDiscuss on Discord