Daniel Compton

Articles - Snippets - About - Work With Me - Projects

Building Stable Foundations - Heart of Clojure 2019 talk

This was a talk I gave at the first Heart of Clojure in 2019. I wrote more about why I enjoyed it so much at the time.

You can also see the talk on Youtube.

Good morning. I was going to ask if everyone’s feeling awake, but I think after that karaoke, I think everyone’s pretty well awake by now.

This talk is called Building Stable Foundations, and it’s about building stable long lasting foundations for the Clojure community.

[00:00:30] Bozhidar has already introduced me slightly, but I’ll go back over it. My name’s Daniel Compton, I’m an open-source project maintainer. I am an administrator at Clojars. I write and record a Clojure podcast and newsletter at therepl.net, and I am working at a startup called Falcon.

Falcon is hiring for Clojure developers, remote Clojure developers. If you’re sort of in the U.S.-ish region, or you can work [00:01:00] on a similar kind of time zone, check out falconproj.com [Ed: Falcon shut down 😢] or come and talk to me, and I can tell you more about what we’re working on.

The other thing that you may know me from is an organization called Clojurists Together. With the help of our members, we fund and support open-source Clojure projects. The question is, well, why is that important? Firstly, who has heard of Clojurists Together?

Great. All right. Just about everyone, and who [00:01:30] here is a member or your company is a member of Clojurists Together. Great. The reason why Clojurists Together is important, or I think it’s important, is that Clojure is foundational, and open-source Clojure is foundational to what we work on. Of course, the Clojure language is open-source, but the vast majority of tools, and libraries, and services, that we all use as day-to-day Clojure programmers come from open-source software.

It’s important, not just for [00:02:00] us as a community, but open-source Clojure is really foundational for businesses too. Businesses are investing lots and lots of money into the Clojure ecosystem, and they’re relying on these investments lasting a long amount of time. I think it’s really important to sort of remember that context of the money being invested into Clojure and making sure that that is well-spent and it’s going to be [00:02:30] a long lasting investment.

Who here is lucky enough to get paid to work with Clojure? All right. Of those of you who had your hands up, who uses open-source Clojure code in your day job? It’s a bit of a silly question, isn’t it? That the idea that as a working Clojure programmer you wouldn’t be using a significant amount of open-source code as part of your work. Maybe there are some people. I suspect there are a couple, but for most of us a lot of the work we do is built on top of these really important open-source foundations. Open-source creates an incredible amount of value for everybody that gets to use it. It’s the shared resource that doesn’t run out when somebody uses it, that doesn’t mean that anyone else can use less of it. This is a really amazing property of software and open-source software.

But a few years ago, I started to notice [00:03:30] developers talking about burning out. In the Clojure community and also outside of the Clojure community, and this was kind of worrying to me, because a lot of these people were really important to the Clojure community. They were doing really important work.

I started to see these kinds of things being said. People saying things like, “Any tips for a post burnout return to open-source? I’m exhausted, but want to keep going.”

This [00:04:00] really broke my heart to see these people who I cared about, many friends and that they were not feeling supported. They were thinking of leaving, and I was really worried about this. We have this problem that the foundations of our community are perhaps not as strong as they could be, or they’re not being as supported as they could be.

Before we talk more about our open-source foundations, I think we can learn some things from building [00:04:30] foundations. This is what a building site looks like before you start building on it itself. The first thing you have to do when you’re building is to prepare these foundations, everything else goes on top of them. So you have to get this right first.

When you’re building foundations, there’s a few things you need to do. The first thing you need to do is survey the land. You need to find out, do you have the rights to work on this land that you think you own and is the structure that you’re going to [00:05:00] be building allowed to be built on this land. This is really important to get right, because if you get it wrong, you might invest all of this money into building something which you then need to take back down again.

Next thing you need to do is check under the ground. What is going on underneath the surface? You can see what’s on top of the ground, but it’s really important to find out what’s going on underneath there before you start building, so that you don’t end up being surprised by things later on. The last thing, and this is the most important thing and [00:05:30] what we most often think of when we’re thinking of building foundations is reinforcing the ground. A fully built house can weigh 50 to a hundred tons or more, and that’s a lot of weight to be putting into the ground, and so you really want to make sure that those foundations are going to be able to hold up to that kind of weight.

This is quite similar to what we do when we’re picking dependencies. The first thing we do is we’ve got to check the license. I hope we check the license and [00:06:00] we also will usually want to review the codebase. Is it well-written Clojure? Does it have tests? Is it actively maintained? These are all important things to be aware of before you pick a library, and the last thing, if this is going to be used in the critical path of your production software, you probably want to test it, find out how fast is it. Can it stand up to the full weight of production? Are there memory leaks, that sort of thing?

Foundations are really important [00:06:30] to building a house, but it turns out that they’re actually pretty cheap. In the U.S. on a $400,000 house build. They cost about $10,000, two and a half percent, so not really that much of the cost of the build. This is because it’s a very well understood process. It’s been done many times every day, and for the most part, it goes pretty smoothly.

Has anyone ever seen a house like this before? [00:07:00] What’s happening here is that the foundation on this house wasn’t built correctly the first time. It’s a little bit hard to tell, but this house has been lifted right up off the ground, quite a few meters up. Stacked up on these Jenga blocks so they could fit machinery underneath to repair the foundations, and then they’re going to slowly lower that house back down.

If you look closely, I’m not sure if it’s going to come through very well here, but there are some people standing underneath this house, [00:07:30] very brave people. Does anyone relate to this picture? Has anyone had to fix the foundational dependency in your codebase and feel like there’s a lot of things bearing down on top of you? As you can imagine, this is not a cheap process. It can cost $20,000 to $100,000 to do this kind of repair, which is two to 10 times more than the cost of building the foundations correctly in the first place

[00:08:00] Yesterday, Tiago talked to us about resilience, and I’m glad he really dug into this, because I don’t need to go too deeply, but we’ll remember resilience as the ability to recover from a stress or a change. This is what you’re going for when you’re building foundations, you want the foundations to be able to stand up to the elements and all of the things that a house is going to face over a hundred year lifetime. What we can see here is there’s two houses, on the left… If you look at them, superficially they kind of look the same. They’re [00:08:30] both held up by wooden poles. They’ve both got thatch kind of roofs. They’re both orangy. They’re both roughly the same size, but the one on the left is not very resilient at all, and the one on the right is incredibly resilient, even though it’s over the water.

This house on the right is, well, it’s a community. Firstly, it’s a community of houses. It’s not just a single house, and it’s also held up by many stilts, so that even if you lost one, or two or even a dozen of these stilts, [00:09:00] the whole structure would still be sound. Whereas on the left, you’ve got just two poles holding this house up. And if one of those two goes down, the whole thing is coming down. If you agree with me that open-source software is the foundation of the Clojure community, then we need to be asking this question, how resilient are our foundations? A few years ago, as I was looking around the Clojure community, it seemed to me that we were perhaps closer to the left-hand side than [00:09:30] the right-hand side.

We had a few people doing a tremendous amount of work for the Clojure community. People were depending on their work every day, they were using it. It was really important, but they were feeling unappreciated, perhaps burnt out, and there was this refrain of, “I’m exhausted, but want to keep going.” I thought they were at severe risk of burning out. If we lost these people, this would be somewhat equivalent to that house we saw, having to be lifted up on the blocks, [00:10:00] and there would be a huge investment. We would lose a huge amount if we lost some of these people, and it would cost us a lot of time and effort to regain that knowledge and hard won wisdom.

This led me and others to create this organization called Clojurists Together. We have a very simple mission. We want to fund and support open-source software, infrastructure, and documentation that’s important to the Clojure and Clojure script community.

We [00:10:30] fund three kinds of projects. The first kind of project is maintenance projects, and these are projects that are often not very interesting to fund. Maybe they’re very stable, used by lots of people, but over time, software needs maintenance, and if people aren’t being paid, or aren’t getting dedicated time to work on it these maintenance things can pile up over time. So this is a place where I think Clojurists Together is really well suited to help these projects, [00:11:00] because it’s work that sometimes wouldn’t get done otherwise, or it would take a lot longer. We really liked being able to fund these kinds of low-level boring kind of work, but these are really important improvements to make.

The next kind of work we like to fund is new development. This might be a project that is already well understood and used by many people, and they have an idea for a new feature, or a new release, or something that they would like to do, and they need some dedicated [00:11:30] time to work on it. We also take and fund projects who do this kind of work.

The last kind of project is new fledgling of projects, and we fund a few of these kinds of projects where someone comes to us with a seed of an idea, and perhaps they’ve already created proved out that this is an idea, but they would like some funding to go to the next level with it, and we’ve been able to fund some of these projects, and they have [00:12:00] really surpassed my expectations. You’ve done amazing work, so this is another kind of project we like to fund.

We launched in October 2017, and in 2018 and 2019, we’ve funded 11 projects.

Who in the audience has used one of these projects? All right. What about three projects. Who’s used three or more of these projects? Wow. Okay. Who’s used five or more of these projects? Great. All right. Is there anybody here who has not [00:15:00] used a single one of these projects?

All right. Well, that’s really good to see. [Nobody put their hand up]. I’m glad that this has benefited people.

I’m really excited to be able to announce today, this Heart of Clojure conference has been timed really well to be able to announce on stage the next funding round. We’re going to be funding four projects, $9,000 over the next three months.

The first project we are funding [00:15:30] is Calva. Calva is a VS Code extension for Clojure. VS Code has been gaining in popularity hugely over the last few years. I think it’s one of the sort of surprise sleeper hits of the programming community, so we’re really excited to be able to fund Calva.

We’re also funding Thomas Heller to work on Shadow CLJS again. He’s been doing incredible work for the Clojure and ClojureScript community on Shadow CLJS. I hear lots of people talking about it and using [00:16:00] it, and it’s a really great tool, and those improvements are benefiting lots of people.

Another project we are funding is Meander. Meander is a really interesting Clojure, ClojureScript, data transformation library. I probably can’t do it justice here, but Michiel Borkent [Ed: this was Timothy Pratley, sorry Michiel!] wrote a really great blog post about this, so if you find him here at the conference, ask him to tell you more about Meander, because it’s a really cool project, transforming data and maps, which [00:16:30] we do every day.

The last project is CIDER. I would say CIDER needs no introduction as well, because many, many people use CIDER and many more people also use Bozhidar’s work on the Orchard and in REPL and all of these other foundational things that sit beneath CIDER that we as a community get to take advantage of. I’m really excited to be able to fund these four projects, thanks to the [00:17:00] support of our members.

The way Clojurists Together works is that we have members sign up throughout the year, and then every quarter we will go to them and say, “Hey, what do you think we should be working on? What’s interesting to you, what do you think is useful? What areas do you think need support?” Then we take that information and we create a call for proposals. We say to the wider Clojure community, “Hey, here’s what our members would like us to fund. Please [00:17:30] give us some proposals,” and every quarter we get an amazing amount of proposals, more than we could pick. The quality of these submissions is really high. Then the committee members vote on the projects that we want to fund, and then we’ll fund those projects for three months. At the end of those three months, we turn around and do it all again, so it’s pretty straightforward.

Clojurists Together has been getting noticed and has influenced beyond [00:18:00] just the Clojure community. This is my friend, Devon, who works at GitHub, and she’s talking at GitHub Satellite earlier this year about GitHub Sponsors. Clojurists Together was mentioned as an influence on GitHub Sponsors. GitHub Sponsors is a project for where you can fund directly the projects that you use on GitHub. I’m really excited to see this. I think GitHub has a really wide reach, clearly, in the development community, and they’re helping to [00:18:30] change attitudes around open-source funding for both, in the Clojure community, but also, in the wider community. This is something I’m really looking forward to.

Devon has this quote, “Our goal is for open-source to be a serious career path people can set up.”

This is kind of the, I think, the next frontier for the Clojure community too. That we have a bunch of projects that are really important, that people are using, and I would really like to see [00:19:00] people be able to be funded to work full-time or nearly full-time on these projects for the good of the community. Clojurists Together, can’t do it all. We’re set up to do a particular kind of funding, and so what I’d like to see is that the community as a whole starts to build this open-source middle class of single or small teams being funded in a meaningful way, and being appreciated by the community so that they can do this [00:19:30] work that benefits everybody.

Part of this is that we’ve just launched a page on our site called Beyond Clojurists Together, where we are collecting Clojure projects and Clojure programmers who are accepting money from Patreon, Open Collective, GitHub Sponsors, or whatever other tools they have, to be funded directly. If you’re interested in funding projects and you don’t really know where to start, this could be a really good place for you to go start looking and finding [00:20:00] some of the dependencies that maybe you or your company is using to fund them directly.

How do we get there? How do we get to this glorious future where all of our dependencies are supported and maintained? It needs two things, time and money. Sometimes people and companies have capabilities for one or the other. We have a lot of people investing, people in companies too, investing in open-source [00:20:30] projects, open-sourcing internal things or maintaining projects. Other times companies don’t really have the time or the capacity to be directly working on open-source projects, but they have some money available that they could put into maintaining them instead.

The other side of the equation is money, and companies spend money on lots of things. They spend it on our salaries, software licenses, cloud hosting, hopefully [00:21:00] green cloud hosting after that talk yesterday, and offices and snacks.

Companies are used to spending money on lots of things, but I think open-source projects often aren’t shaped very well for companies to give money to, and companies need these two things. They need value. They need to be able to say to the decision-makers, to the finance department, “We are getting something of value when we are giving money to this project.” That doesn’t mean [00:21:30] that you need to sell out to the man. There’s lots of different ways that you can provide value to projects, to companies that are both valuable to the companies, that fit with the skills and capacity of what you have, but most importantly, still meet the values that you hold as a person and the values of your project.

This could be things like consulting, feature development, long-term support, maintenance, [00:22:00] and these are things that oftentimes you may be doing anyway, but if you’re able to sell it, and sell this to companies in a way that they are able to understand, and sell onto their decision holders, this can be a really powerful thing.

The other thing that companies need is invoices, and not just invoices, but all of the other financial infrastructure that goes along with things. Companies are not used to using PayPal [00:22:30] donation links to pay for their office rent. They pay money to a bank account, and so this is something where open-source projects, again, aren’t often shaped very well to accept this money.

This is where I’m really excited to see platforms like Open Collective, Patreon, GitHub Sponsors who are providing the shared common infrastructure for projects to be able to work, to be able to provide the kinds of things that companies [00:23:00] need to be able to give money to.

The question now is, are we investing in stable foundations? Are we building stable, resilient foundations that are going to last us the test of time? To answer that question, we need to sort of answer, well, compared to what? The main input to open-source projects is labor, people spending their time on it. I thought perhaps a good comparison to [00:23:30] this will be, how much are we spending on Clojure developer salaries and businesses. To answer that we need two numbers. We need how many Clojures developers there are, and how much they’re getting paid.

Estimates vary from perhaps 20,000 to 50,000 working Clojure developers. We want to be conservative here, so we will just take 20,000, and then we need to figure out how much they’re getting paid. The Stack Overflow Survey for 2019, put the average Clojure developer salary, worldwide, [00:24:00] at 90,000 U.S. dollars. Clojure’s been at the top for the last three years, which is pretty good, but I’ve heard people quibble with this number, say that maybe it’s a bit inflated or unrepresentative in some way, so let’s just take it back a bit more to still be really conservative, and we would just say $80,000 a year.

If you multiply these two numbers together, you end up with $1.6 billion a year being spent on Clojure developers. This doesn’t even include all [00:24:30] of the other expenses that go around Clojure that you need to run a business, and it doesn’t even include the full cost of hiring an employee, but I think this is a really good number to compare to and to think about.

If we were to spend a relatively small percentage of what we spend on Clojure developers, on supporting the foundations, that could be a meaningful amount of money, say two and a half percent of 1.6 billion would be $ [00:25:00] 40 million a year. I think, I can’t even really imagine quite what that would look like. It’s just such a long way away from where we are currently that it would… Yeah, I can’t imagine quite that what that would look like, but what I’ve seen from funding projects, even at a very small level far below this, is that the return on investment there has been incredible.

I think we would see some really exciting things start to happen, even more than they are currently, [00:25:30] if we were to be able to even approach this level of funding, which itself is, I think, very small compared to the value that we as a community get from these open-source foundations.

We come back to this image of the houses on the water, and this image really appeals to me, because I see some similarities here with the Clojure community. These are not always the most pretty houses, some of [00:26:00] those stilts down below look a little bit rickety, but together as a whole, these come together to form a really solid stable foundation. This is what I’m hoping that we can build together for the Clojure community, and to be clear, this is already happening, well before Clojurists Together. This is still a really strong community before this, and I’m excited to see kind of where we can go in the future.

If this idea appeals to you, here’s some things that you could [00:26:30] do.

I want to acknowledge many people here. Firstly, Heart of Clojure, Ana and Martin, they’ve put on a really great conference here, so I really appreciate them bringing me along to speak.

[00:28:30] I’d also like to thank Ruby Together, and André Arko in particular. Ruby Together is the organization that we modeled ourselves afterwards. They’ve done a lot of trailblazing work here and André Arko, in particular, has given us a lot of time so that we can learn from some of the things that they’ve done.

I’d also like to thank the Software Freedom Conservancy. I saw someone here had a [00:29:00] Software Freedom Conservancy shirt on yesterday. The Software Freedom Conservancy, if you’re not aware is our non-profit parent. They handle all of the accounting, and payments, and financial, and legal things that go on that companies need, and that we need to have happen, but that we can rely on them to provide for us. They’ve done an amazing amount of work for us, and so I really appreciate them.

Next group of people I’d like to think is [00:29:30] the projects that have applied. We have more projects apply every quarter than we could fund, and as I read over the applications, I’m really excited, because there’s so many good projects out there, and so I’d like to thank everyone who has applied.

I’d also like to thank the projects that we funded. Sometimes today in the talk, I might’ve slipped and said that we did this work and I wasn’t programming on CIDER, Bozhidar was, so I’d like to thank the projects that we funded, that have done this work. I know that often they’ve taken [00:30:00] time off paid work, or taken holiday time. While we’ve been able to fund them to a certain level, I know often that still means a pay cut for them to be able to do this work, so I really appreciate that.

Also, I’d like to thank the rest of the Clojurists Together team. We have these board members.

On the left we have Maria Geller, Daniel Solano Gómez, Larry Staton Jr., Nola Stowe, Fumiko Hanreich, and Laurens Van Houtven. These are all current board members. We also have [00:30:30] board members from the past, Bridget Hillyer, Toby Crawley, Devin Walters, and Rachel Magruder. [inaudible 00:30:36]is our admin assistant. If you’re a Clojurists Together member, you might’ve got some stickers from us recently, and if you’re wondering, why did this come from Spain? It’s because Rachel, our admin assistant lives in Spain.

I’d also like to thank our members. We’ve seen many of these names around the conference already. Pitch, Nubank, JUXT, Metosin, Adgoji, and Funding Circle. [00:31:00] These companies have all been a huge help to Clojurists Together, and have funded us in a really big way that have let us do all of the work that we’ve been able to do.

Also, I’d like to think these other company members, I don’t have time to go over all of their names, but many of these names are going to be familiar to you, and some of them are at the conference as well.

This is probably my favorite slide at the talk, because this is the names of the 200 members, developer members in the Clojure, [00:31:30] Clojurists Together that are members of Clojurists Together. Yeah, there’s far too many here to name them all. I tried to make the text bigger, so that you could even read the names, but then it just went on for slides, and slides, and slides.

I can’t name them all, but I’d really like to say thanks to the 204 developer members, the 34 company members, and also to the rest of the Clojure community.

The Clojure community is a [00:32:00] really special place. It’s warm. It’s very small, and I think we punch above our weight in terms of the impact that we can have, in terms of the tooling and libraries that we create. That’s in large part due to the contributions of all of the Clojure community coming together to build things for everybody. I think a good chunk of the Clojurists Together members are here at this conference, and so again, I especially want to say thanks to you for your support.

•