Arrow Turns 10

Neal Richardson

Arrow Summit 2025

October 2, 2025

Arrow was established as a top-level ASF project in January 2016. Here’s it’s brilliant first commit, from February.

That means it is approaching its 10th birthday.

I’ve been reflective about this milestone, and what it means for the project and the community. 2 reasons in particular

The first is that I see lots of new projects and startups in the data space, and when they talk about their tech stack or show architecture diagrams, Arrow is everywhere. It’s not just everywhere, it’s taken for granted that if you’re building a data system, you’re using Arrow.

For example, I was at Data Council in San Francisco back in the spring, and I kept seeing slides that just had Arrow everywhere.

Here’s Julien Le Dem, from Datadog, talking about the new data stack, the “Deconstructed Database” and the “Open Data Lake”

And then Ciro Greco from Bauplan showed this diagram (and I hope to see something similar a bit later today as well)

And it struck me how different of a world this is from when Arrow started, or even when I got involved in the project back in 2019. It was definitely the aspiration, but it was not a guarantee.

Seeing all of these presentations talking about Arrow as if it is part of the air we breathe, taking it for granted, I found it overwhelming–in a good way.

The second reason I’ve been reflecting on this is more personal:

This is my son, James. He turned 10 this year. That also got me reflecting on the passage of time, and what it means to support and nurture something over a long period of time, watching it grow and change, and really develop its own identity and trajectory.

arrowrbook.com

(and as you can see, he has great taste in books too)

Arrow is different

Arrow is different from most open source projects, in scope and in ambition. Trying to create a ubiquitous standard is not something most projects aspire to. There’s a different time scale, a different level of patience and persistence required, than building a tool to solve a specific problem.

And as I was reflecting on that, I realized that there are parallels with raising a child–also a long-term investment, with a lot of uncertainty about what the future will look like, and what they will become. And ultimately, you’re trying to nurture something that will become self-sustaining, and go beyond the limits of my imagination for them.

I’ve been fortunate to be a part of Arrow’s journey for the majority of its life. So today I want to share some thoughts about the project and the community, and in particular some of the non-technical factors that have contributed to its success. Understanding how Arrow got to where it is today can help us understand how to ensure its future.

About me

before we get any further, now that you’ve already met my son, I should probably introduce myself as well.

Actually, he saw me working on my slides and wanted to help, and this was his suggestion:

About me

• Started working on Arrow in 2019 (Ursa Labs)

• PMC member since 2020

• PMC chair 2024-25

• Currently working at Posit

I first got involved with Arrow in 2019, when I joined Ursa Labs to help coordinate and fund development of Arrow and related projects. Arrow sounded like a good idea, and I wanted to help make it happen.

…

Before I got into software, I was a political scientist. Usually, that’s not relevant. But today, some of that may come through in this talk.

it should be clear that the views and perspective I’ll be sharing today are colored by my own experiences. and one of the things I love about the Arrow project is that is so expansive, and there are subprojects that I haven’t been involved in much. I want to try to draw some general lessons from my experience, and I hope that others see themselves in them, but if I fail to mention your part of the story, please forgive me, I’m not trying to suggest that only the parts I’ve been involved in matter.

Non-technical factors?

I say “non-technical” for a couple of reasons. First, I don’t need to tell you how Arrow speeds up data workflows and improves interoperability. You’re going to hear some great talks today that will cover that, and chances are that if you’re here, you already have a sense of the technical merits of Arrow.

Second, because I think that the technical aspects are necessary but not sufficient for a project like Arrow to succeed in becoming ubiquitous.

Turns out JD Long and I were mind-melding this past weekend while I was working on this talk. He posted this on Bluesky:

The challenge is to change behavior

Fundamental challenge for a project like Arrow: get other developers to choose to be a part of the ecosystem, to build on it, to help it grow and evolve over time. Rather than going their own way. Yes, the tech matters, but this is a social challenge. We were trying to change behavior, trying to change people’s expectations.

All software on some level is a social experiment, an attempt to change your behavior–to get you to use my library, buy my product, etc.

But for Arrow, the social challenge is front and center. It’s core to the vision. If you don’t get people to build on it, then it won’t become a standard, and then it won’t be useful.

Interoperability

From the Arrow website:

The universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

Arrow is about interoperability, and that means that you really get the most value out of it when everything is built with it.

Interoperability

This was the point from the beginning. There’s this graphic that’s been on the website for as long as I can remember.

Interoperability

The idea is that all of these projects would only need to implement one export and import format, not N different ones. Sounds great! But that means you still have N projects you need to get to adopt your one true format, and the value of that format increases with the number of projects that adopt it. So you have a chicken-and-egg problem, and you need to convince people to change their behavior.

But the ambition of Arrow is actually bigger than just writing adapters. In fact:

Arrow is a radical social project

There is something more radical in the vision there about decoupling everything, so that you can mix and match the best components for your use case. If storage and compute are decoupled, you can choose the best compute engine for your use cases, and have different specialized layers work together seamlessly.

So it’s not just: use my library because it’s faster. Or even: adopt this standard because it’s more efficient. It’s: rethink how you build data systems.

That was an ambitious vision: a technical one, yes, but also a social one. Getting people to buy into that vision, and to build on it, is a challenge, just as much as it is to design the right specification and implement it in fast libraries across multiple languages.

Maybe you don’t think of yourself as being part of a radical agenda (maybe you do!), but that’s what you have been doing.

But in order for the revolution to succeed, we had to overcome some forces that worked against us, that created incentives for developers not to join us.

Collective action problem

• We would all benefit if we use the same format

• But building it is expensive

• Free rider problem: any individual has an incentive to let someone else to do the work

• If everyone does this, no one does the work, and we don’t have a standard

Coordination problem: how do you get enough to invest in the common good?

In other words, the challenge for Arrow in the beginning was a coordination problem. How do you get enough of the developers working on these various data projects, as well as their corporate overlords, to invest in the common good?

There were several forward-looking, strategic actions that some of the initial Arrow developers took to address this.

First is governance.

Governance

Arrow is part of the ASF:

• Vendor neutral: everyone contributes as individuals, not as representatives of their companies

• Self-governed by project management committee (PMC)

• Earned authority: PMC invites new committers and PMC members based on sustained engagement

• Apache 2.0 license: free-as-in-freedom

• Values: “community over code”, “doers decide”

Arrow is an Apache Software Foundation project. (read the slide)

In principle, this makes it safe for companies that are otherwise competitors to collaborate on it. And in practice, we’ve seen that happen.

Open governance doesn’t solve the coordination problem by itself, but it is a form of commitment device that removes some of the risk of cooperating with others. It signals that we are safe to depend on. (Contrast with a scenario where Arrow was something being developed by one big company/vendor.)

Second was fundraising and paying people to build out the technical foundations for Arrow.

Fundraising

Volunteer work is great and is essential for open source and Arrow in particular. But volunteer work tends to underfund foundational maintenance work, because it’s not as exciting as building a new feature or a new tool. Most people don’t choose to spend their free time fixing build systems, or reviewing PRs from other people who got to do the fun stuff.

Dremio, with Jacques Nadeau and others, made a big bet on Arrow early, and hired several people to work on it full time, particularly in Java. Most companies did not jump on board so much.

Wes was intially supported by Two Sigma, but he wanted to go bigger, so he partnered with RStudio (now Posit) to establish Ursa Labs in 2018 to help coordinate and fund development of Arrow and related projects. That’s where I joined in 2019, and we worked to source contributions from a number of companies to fund development, and hired a team of engineers to work on it full time.

Of course, we weren’t alone in contributing to Arrow, but at the time a high volume of contributions came from this team, in particular in these foundational areas, and we also made a point of investing heavily in code review and other maintainer activities to enable those who maybe didn’t have the ability to dedicate as much time to Arrow to be able to contribute effectively.

I saw a lot of this work as about trying to solve the collective action problem, by providing a way for companies to pool their resources to fund development of Arrow.

Fundamentals

• Build and packaging tooling (even s390x, solaris, etc.)

• CI everywhere

• Integration testing

• 1.0 release (2020)

all of these are deeply technical, of course!

but at least for the work I was involved in, there were always two goals in mind. the immediate technical goal, of course, but also keeping in mind the social aspect, changing behavior, of getting people to believe that it was safe and wise to build on Arrow, that we said this was going to be the standard, and we were grinding in order to make that a reality. Solving the collective action problem.

Early proof points

Early proof points

feather (2016)

Basic library for reading a simple subset of Arrow in R and Python

Showed that you could get significant speedups over CSV and other formats

Language independent –> better interoperability

Early proof points

pyspark (2017)

Orders of magnitude speedups in moving data between Spark and Python

Leveraged distinct implementations of the Arrow format (Java and C++)

Draw the rest of the owl

• DataFusion (2018), Ballista (2021)

• Acero (C++ execution engine, 2021)

If arrow is so fast reading from Spark, why not make the whole thing out of arrow?

Andy wanted to show that you could use Rust for data science and computing, so he built on arrow-rs, and eventually donated the project to Arrow, and then it later spun back out

All great technical achievements on their own, but had the dual purpose of demonstrating the radical vision of an Arrow-native world.

---

Great, so at this point a few years in, we’ve demonstrated the potential, and we’ve been grinding to build out the foundational libraries and infrastructure to make it possible for people to build on Arrow.

We’ve removed a lot of the risk of building on Arrow, by having a solid governance structure, and by building out the technical foundations and getting to a 1.0 release.

And because it’s liberally licensed, you’re not locked in. If you find issues with the libraries, or have a use case that you want to see supported in it, you can fork it and do your own thing if you need to. That’s good, right?

right?

Forking is bad for standards

Unfortunately, for a project like Arrow, forking is harmful for our goals. The core value of Arrow is that it is a standard.

If everyone forks it and makes their own “Arrow-ish” flavor, then you don’t have a standard anymore. So you don’t get the seamless interoperability, and your “deconstructed database” accumulates a bunch of friction.

So we as Arrow maintainers have to work harder to encourage developers to contribute to the core Arrow libraries, and engage with the community about the format. Because it is possible to fork, we have to work extra hard–we have to earn it.

ASF governance–everything in public, self-governing, etc.–sets us up to be able to draw people in. But it’s not sufficient.

Governance is not enough

• Earned authority and “doers decide”, but who is able to “do”?

• “Community over code”, but who feels welcome to be a part of the community?

Governance is not enough

• Building relationships with other projects and communities

• Being welcoming, actively encouraging new contributors

so we as maintainers of Arrow need to work hard to encourage new contributors, and to make it easy for them to contribute: to pull people in, to keep them from feeling like they only way they can make progress is to fork it and fragment the ecosystem.

This means: building relationships, …

• Always directing communication to public channels (Jira/GitHub, mailing lists): actively avoiding any impression that decisions were being made in private

… I remember doing this many times,

and realizing that I could have just made the fix myself and it would have been faster. Even though the technical end result would be the same, the social end result would be very different. We would have the bug fix and someone new who feels like they had a hand in making this project. (like a cake mix.) Maybe that would make the more likely to use our tools. Maybe if they had a positive experience engaging with us once, they would come back and do it again. you never know whether that person might end up as a core maintainer one day.

• Guides and other documentation to help new contributors get started

• Cultivating repeat contributors, looking to invite committers

So I think it is a combination of these factors–yes, technical excellence, but these non-technical factors as well–that have helped Arrow get to where it is today. Other open source projects may not need to work as hard to be open and inclusive in order to achieve their goals. But for Arrow and its radical agenda, these extra layers are needed to maintain the center of gravity, to keep pulling projects in.

I think it is safe to say that the original collective action problem has been solved. Based on what I see, Arrow has reached a critical mass, and because of the network effects, it is now the default choice for new data systems. It’s worth it for new projects to build on Arrow because so many other projects are already doing so, no need to solve a coordination problem to get them there.

“New projects” is an important phrase here. This gets to the radical vision of Arrow, and of changing the behavior of developers who are building new systems, to build them along these decoupled lines, with Arrow as the common foundation.

connect with cassandra/drill etc. graphic: that’s the previous generation, arrow built credibility by being there, and the next generation built on it from the beginning.

Actually what happened was the next generation:

A generational change

What about the next next generation?

great, so now we’re here, 10 years in. Arrow is everywhere. It’s taken for granted that if you’re building a data system, you’re using Arrow.

but what about the next generation? how do we make sure that arrow continues to be the foundation for the next next generation of data systems? there will always be innovation in data systems, query engines, hardware, programming languages, etc. And there will always be incentives to try to make something just work for your product, to re-bundle what is being deconstructed now.

how do we make sure that the developers of those systems choose to build on arrow, rather than going their own way?

in other words: when someone prepares the “Arrow Turns 20” talk, what will they say?

Our culture will make the difference

this is where I believe the culture of the community really matters. If we remain open to new ideas and new people, and if we actively welcome them and support them in becoming repeat contributors, then we will be well-positioned to continue to grow and evolve the Arrow ecosystem. this culture is essential to the long-term sustainability of the project.

the challenges will be very technical, of course! finding the best technical solutions to navigate these trade-offs will be essential. but our openness and inclusivity is our superpower–that’s what will make sure that the best technical minds keep coming to work with Arrow and engaging with the community.

So, to conclude, I feel like there’s some cliche I should land on.

(insert cliché here)

If you want to go fast, go alone.
If you want to go far, go together.

It takes a village to raise a child.

Choose kindness

There are lots of ways to run an open source project, or really to exist in the world. Maybe it’s possible to brute-force your way to ubiquity with just technical merits, by aggressively pushing your technical superiority and proving you’re right. But ultimately, I skeptical that that’s sustainable. And it’s not something I would want to be a part of anyway.

A healthy culture is important for sustainability

I believe we have a solid cultural foundation now. And in many ways, culture is self-reinforcing: it attracts people who share those values. And I don’t see anything that makes me concerned that we’re losing this welcoming culture.

But I wanted to take some time to highlight the importance of this culture, and to encourage all of us to continue to nurture it, because sometimes we don’t appreciate the importance of values and norms until they are gone. Not to drag my country’s current political situation into this, but it is a good object lesson in how the same formal institutions can lead to wildly different outcomes when there is a shift in norms around what is acceptable.

Thank you 🙏

if you’re already an arrow contributor or maintainer, thank you for your contributions and dedication to this radical social project. the community and values I described here, to say nothing of the technical achievements, would not be possible without you. it’s been such a privilege to get to work alongside many of you over these years, through some ups and downs, and to get to learn so much from you.

if you haven’t gotten involved with the Arrow community before, welcome! we’re glad you’re here today. and I hope you’ll stop by the project on github, join the mailing lists if you like, and please don’t hesitate to share any feedback or ideas for how we can improve things. (of course, as I mentioned, you may get invited to submit a PR to address something you reported, and who knows where that might take you…)

and finally, thank you again for coming today and listening.

Image credits

• From Flickr, under Creative Commons licenses:

  • Prairie dog: Tim Blair (https://www.flickr.com/photos/timblair/445110345)
  • Fragment from a mural by Roberto Rodríguez Navarro at Zapata’s old home in Anenecuilco, Morelos. (Arrow logo overlays not in the original) (https://www.flickr.com/photos/25222005@N08/4308675698)
  • Money: https://www.flickr.com/photos/59937401@N07/5856660723

• Forks: Sternsteiger Stahlwaren (https://www.pexels.com/photo/close-up-of-two-forks-joined-together-16550737/)

• Pretty green scenery: https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/Ageo_Green_Scenery_with_Fujinami_and_Nakabun_201907.jpg/2560px-Ageo_Green_Scenery_with_Fujinami_and_Nakabun_201907.jpg

Memes aside, the other photos are mine.