Almost two years ago, there was a Github issue on reagent (a ClojureScript React wrapper), suggesting that Preact be added as a substitute for React. I wrote up a fairly long comment about why I didn’t think this was a great idea (at least not immediately). React’s recent announcement of the new hooks feature made me think about it again. I’ve republished it here with a few edits for context and time.

Introduction

In principle, I’m not opposed to the idea of Reagent using Preact. It has some cool features and I like that it is small (although in comparison to the total compiled size of a normal CLJS app it’s probably a wash). If Preact worked 100% out of the box with Reagent with no code changes required then I would have no issues with someone swapping out the React dependency for a Preact one and calling it a day. If there are only a few minor tweaks to Reagent required to pass the tests, then again I don’t really have any issues with that. I suspect that even if you have 100% of the tests passing, there will still be issues, as Reagent was built around React, and may not have tests that would cover the difference in behaviour between React and Preact.

Abstraction

It looks like it may not be a pure lift and shift to support Preact. If that’s the case then we run into a bigger issue: abstraction. Reagent was written and built around the ideas and in the context of React. There are assumptions (probably tens or hundreds) built around React’s API and possibly implementation details too.

Adding abstraction adds a large cost because you can no longer program against a concrete API and implementation, you now have to consider two. There are three numbers in computer science, 0, 1, and many. We would be moving from 1 to many, and that takes work.

An aside: recently at work, we were looking at moving a legacy system from only supporting dates in the past to also be able to support dates in the future. This should be straightforward right? We talked to the programmers responsible for it and they couldn’t guarantee that it would work, nor whether supporting future dates would be easy or hard. In the building of that (or any) system, hundreds of simplifying assumptions are made around the context that the system is going to be built in.

It is a very common pattern to have different backends and I don’t see any downsides to it.

I can’t think of a single example of a system with multiple backends that didn’t have any downsides to it, e.g. ORMs, HTML/CSS/JS, Java. There may be some, but they would be the exceptions that prove the rule. Everything has a cost, the question is whether there is a benefit that outweighs the cost. It is much harder to remove something from software than to add it, which is why we should be certain that the benefits outweigh the costs.

While Preact strives to be API-compatible with React, portions of the interface are intentionally not included. The most noteworthy of these is createClass() … https://preactjs.com/guide/switching-to-preact#3-update-any-legacy-code

Reagent currently uses createClass. There are workaround options provided, but this is an example of some of the API differences between React and Preact which you need extra compatibility layers to support. Do we know if the compatibility layer works 100% correctly?

A possible future if Preact support is merged now

As a thought experiment, let’s assume that Preact is in Reagent with some kind of compatibility shim. Preact already has several performance optimisations that people can take advantage of:

customizable update batching, optional async rendering, DOM recycling and optimized event handling via Linked State. - (from Preact homepage)

Wouldn’t you want to be able to take advantage of those in your application? I certainly would. Now to do so, you may run into issues because the compatibility shim layer that was written was encoded around default assumptions of React, and they may not apply to Preact. Do we have to rework the shim layer, or lower level Reagent API stuff? Who is going to do that work? Who is going to review it and merge it?

Let’s consider the reverse. Perhaps in some new React version, Facebook comes out with a new API which is faster or better suited to Reagent’s style of rendering, so we want to switch to that [since writing this they came out with Hooks. These may be added to Preact also, but this isn’t certain and neither is the time-frame]. However that new model may not work with Preact. Again, we’re in a bit of a pickle: Preact users want to be carried along with Reagent and get the benefits of new Reagent work, but it may not be easy or possible to support the new API for them. Now what?

Consider everyday development on Reagent. Reagent’s source code is built around a very detailed understanding of React and is highly optimised. If Preact was supported too, then developers would probably need to gain an understanding of Preact too.

At the moment, Preact has one main contributor, it has been around for 1.5 years. React has many contributors. I’d estimate there are 100+ people with very deep knowledge of React. It’s been around (in public form) for 3.5 years. In general, the JavaScript community does not have a reputation for long-term support of projects. What happens if development slows/stops on Preact and the compatibility layer isn’t kept up to date? It is much harder to remove something than it is to add it. Who decides when/if to remove Preact from Reagent at a future date?

These are all hypotheticals, but I hope this demonstrates that the extra abstraction provided by supporting two VDOM layers doesn’t come for free. At the very least, it consumes extra brain cycles when testing and developing Reagent, extra support and documentation costs from users wanting to use one or the other, as well as extra indirection when running and debugging apps using Reagent.

The Innovators Dilemma

If you haven’t already, I highly recommend reading “The Innovators Dilemma” by Clayton Christensen. One of the key points he makes in that book is the difference between integrated and modular products and when to develop each kind.

CHRISTENSEN: When the functionality of a product or service overshoots what customers can use, it changes the way companies have to compete. When the product isn’t yet good enough, the way you compete is by making better products. In order to make better products, the architecture of the product has to be interdependent and proprietary in character.

In the early years of the mainframe computer, for example, you could not have existed as an independent contract manufacturer of mainframe computers, because the way they were made depended upon the art that was employed in the design. The way you designed them depended upon the art that you would employ in manufacturing. There were no rules of design for manufacturing.

Similarly, you could not have existed as an independent maker of logic circuitry or operating systems or core memory because the design of those subsystems was interdependent. The reason for the interdependence was that the product wasn’t good enough. In every product generation, the engineers were compelled by competition to fit the pieces of the system together in a more efficient way to wring the maximum performance possible out of the technology that was available at the time. This meant that you had to do everything in order to do anything. When the way you compete is to make better products, there is a big competitive advantage to being integrated. … In order to compete in that way, to be fast and flexible and responsive, the architecture of the product has to evolve toward modularity. Then, because the functionality is more than good enough, you can afford to have standard interfaces; you can trade off performance to get the advantages of speed and flexibility. These standard interfaces then enable independent providers of pieces of the system to thrive, and the industry comes to be dominated by a population of specialized firms rather than integrated companies.

• Clayton M. Christensen, The Thought Leader Interview

I would argue that we are still very much at the point where the current VDOM libraries aren’t good enough yet. They aren’t yet ready to be commoditised, and the best option is to tightly integrate.

Options from here (with some conjecture)

1. Someone can make a PR to Reagent to add support for Preact. It will probably take a while to get merged because it is a significant change. Once it is merged and released, there will probably need to be several rounds of revisions before it is ready to go. Because Reagent moves relatively slowly, this will take a while.

   Reagent also has a large number of production users, so new releases need to be well tested and stable. Adding Preact to the mix is going to slow this down further.

2. Someone can make a fork of Reagent (let’s say it’s called Preagent). You can run wild experimenting with what is the best way to use Preact in Preagent, take advantage of all of the great features Preact has, and have a much faster turnaround time for releasing and using it. You will be able to work out what is the right API and integration points for Preact because you have room to experiment with it, without the weight and responsibility of bringing the rest of the Reagent users along with you.

   At some point in the future, you could review merging Preagent back into Reagent, given all that you now know. You would also have the weight of evidence on your side where you can demonstrate the benefits of Preact and can show how many users want Preact. This would let you make a much better case for including Preact, give you what you want in the meantime, and likely provide a higher quality integration in the future.

   Alternatively, you may decide that Preagent is better served going its own way and integrating more closely with Preact. This is also a good option.

Abstraction is not free

The point I have been trying to drive through this post is that abstraction is not free. Over-abstraction is a common anti-pattern and it saps productivity. I had a friend who recently left a Clojure job and started a Java one. He quipped to me about how he’d forgotten what it was like to trace code through five layers of abstraction to get to the concrete implementation. As programmers, we’re trained to solve problems by adding layers of abstraction, but that isn’t always the best way to solve the problem.

Lastly, this isn’t a personal attack on you or your ideas. I’m all for innovation in ClojureScript webapps and I think that it is worth investigating Preact and how it could work in the ClojureScript ecosystem 😄. I’m not against Preact. I would consider using it if there was a measurable benefit. I’m just suggesting that the best way to go about this is probably not to integrate it into Reagent as the first step.

I sent the first newsletter for The REPL two years ago, and I have really enjoyed writing it. I’ve learnt a lot from what people have written and created to share with the Clojure community. Sometimes when I’m writing the newsletter I’ve thought “That’s fascinating, I’d love to hear more about the technical details of that”. Now I have an outlet for doing just that.

I first had the idea of producing a Clojure podcast around January 2018. I didn’t have the time for it then, but the idea kept swirling around the back of my brain. When Michael Drogalis’s Pyrostore was acquired, I wanted to hear more about the acquisition, so I contacted Michael to see if he wanted to talk about it. That became the first episode.

My goal for The REPL is to talk with people about the technical details of the projects they are working on. The Clojure community has a ton of interesting people doing really creative work with Clojure. I’ve really enjoyed talking with a bunch of people already, and I’m looking forward to talking to more of them in the future. You can find it therepl.net. It’s available on Apple Podcasts, RSS and all of the other common podcasting apps.

Cognitect has recently released the results of their State of Clojure Survey for 2018. For the last two Clojure survey’s, I have reviewed the free-form answers at the end of the survey and tried to summarise the zeitgeist of community feeling. I enjoy it each time, and so here’s this years analysis.

Some comments have been lightly edited for spelling, clarity, and brevity.

Error messages

Error messages have been one of the top complaints about Clojure since the surveys started, and this year they have gotten worse with the introduction of specs on core macros. I’m not the only one who has noticed this either. In 2015 Colin Fleming gave a talk on improving Clojure’s error messages with Grammars. In previous surveys there was hope that spec would be able to use this approach to improve error messages; Ben Brinckerhoff has recently shown a proof-of-concept that gives some similar (excellent) error messages in Expound.

• My primary wish would be for more helpful error messages (e.g a la Elm) in Clojure and Clojurescript.
• I was excited about spec initially; however, I find the error messages so large, noisy and unhelpful that I don’t want to use it. To put it another way: when there’s a spec failure, my heart sinks; knowing that the next few minutes are probably going to be frustrating and painful.
• Spec was supposed to help with error messages, but it just made them worse. Stuff like (let [f/x 42] f/x) now emit insane error messages. But it looks like the Clojure dev team has moved on to improving command line tools now? Can we please just get error messages fixed before a ton of people get frustrated and jump ship? The current state of errors in Clojure is an embarrassment to the community.
• Error messages is a big show stopper. I am trying to get Clojurescript adopted for web and react-native development withing our organization for few years, but I always see the same pattern: people get some error during project setup/configuration and get stuck forever because they have no idea what the error means at all and how get it fixed. Please please improve error reporting. :)
• A big problem for new starters, an issue that has been singled out time and time again over the years, is poor error messages that lead to difficult debugging of code structure. That Clojure 1.9 has actually made these error messages harder to understand is a disaster. […] I hope I’m wrong and that I’ll understand the grand plan eventually.
• Please please please please enhance the error messages. An incredible error reporting would make this language out of this world, and that would make the biggest difference.

Spec

Comments ranged from loving spec to disliking it. There were also requests for more guidance on how to apply spec to real-world use-cases and how to integrate it into whole programs.

• OMG Spec is amazing. It’s dramatically changing the way I program. I’ve been starting new projects out by writing specs before nearly any other code. The result is that I am able to extend my hammock time to the keyboard, and think through my domain model with composable, high level descriptions implemented as specs. The fact that these high level descriptions then become so useful in so many different ways — generative testing, value conformation for data api parsing, more useful error messages, etc — is a stunning thing to behold. I’m excited to see how the fruits of these ideas permeate through our and other communities.
• The presence of “spec” on some of those lists made me want to rate everything else “important” so I could show how unimportant it is. […]
• spec is awesome, but its dogmatism leaves us needing tools like spec-tools, which makes working with spec clunky.
• There needs to be more guidance, development and reliance on spec from both Cognitect and community.

Docs

Documentation in all forms has been another perrenial complaint but this is starting to improve. clojure.org continues to add more guides this year on Programming at the REPL and Higher Order Functions among other things. Martin Klepsch’s cljdoc holds a lot of promise to improve the state of third-party documentation, by automatically generating and publishing docs for published JARs.

• We should take example of how Rust community/documenation is run: BIG focus on newcomers, features MUST be perfectly documented before being shipped. This serve advanced users as well. No offense, but Rich Hickey should understand what good documentation is, or delegate completely to people who do so that he can focus and making a even greater language (his strong point).
• I feel the true weak points are confusing error messages, not enough friendly guides/tutorials for the early to beginner Clojurist, and a lack of solid CLJ/S example projects. These are key because it is difficult to know the right way without “blessed” examples.
• Please provide a good set of tutorials on Javascript Interop such as how to translate a ES6 or higher JS file into ClojureScript.
• The documentation is really awful - incomplete, vague, no examples, no semantic precision - not up to the standards of what one normally sees for programming language documentation.
• Please improve the docstrings. They are the worst of any programming language I have ever used.
• Really, the core documentation is not written for beginners or even intermediates
• What I miss most is a book which teaches how to approach problems with Clojure. I mean primarily not “best practices” or “design patterns” or language independent algorithms. Nor a kind of language reference or few lines long mini examples, but complex, advanced examples with detailed explanations about the pros and cons of certain solutions applied/not applied.

Startup time

Startup time continues to be an issue for people. As serverless computing becomes more mainstream, this may end up pushing people from Clojure to ClojureScript, or elsewhere.

• Startup time is really really really really really long
• Startup time never used to be an issue for me, as my services were long running and persistent. Lately however, I’ve been using Docker for jobs. Startup time becomes an issue here. Continued improvements in total app startup time will pay dividends going forward.
• Startup time regression. Clojure 1.9 startup time is 1/3 longer then Clojure 1.8 (1.6s vs 1.1s on my humble machine).

Marketing/adoption/staffing

There were lots of comments this year in this category. Even after 10 years, Clojure still unknown or seen as being extremely niche to many people, especially among business stakeholders. Hiring seems to be less of an issue than in years past. There is still a large contingent of programmers looking to work with Clojure in remote positions, although it’s hard to say whether this proportion is any higher than in other language communities.

• I recently hired a programmer with no Clojure experience. She was writing lovely idiomatic Clojure within a month. The biggest difficulty was overcoming her initial fear of parentheses.
• To use Clojure(Script) more it would help if there was marketing for it. It’s still largely unknown, and therefore hard to sell to other companies.
• There is always an uphill battle in pushing Clojure as an enterprise solution, this seems to mostly occur to lack of marketing. Non-technical decision makers care about branding.
• I’ve found that some folks really dislike lisps passionately. One of our vendors indicated that his customers and employees were highly critical of the use of Clojure as an extension point.
• Unfortunately, this year my organization in ceasing its use of Clojure for production projects after many years of using it. Staffing and on-boarding external resources are the 2 main reasons for doing so.
• Hard to convince managers at work to use Clojure because of the perception that its a fringe language
• I wish it was easier to find a remote (non US-only) remote Clojure/Clojurescript job for people with no major Clojure experience :)
• It’s very important to address the enterprise software community. We need formal tools, formal approaches, standard IDEs, SonarQube, Eclipse, Jenkins, e.t.c.
• I believe “JavaScript converts” will be an important source of growth for Clojure moving forward.
• We find Clojure a joy to use and hiring Clojure engineers or those excited and capable of learning Clojure has been a big advantage, contrary to common wisdom. Clojure positions tend to self select for strong candidates.

Language

• Enterprise Java tends to lean heavily on reflection to pull components together (e.g., Spring, annotation-driven JUnit and JAX-RS). To incorporate Clojure into those environments, lots of component-specific wrappers are needed. It is easy to slip a little Java into a Clojure codebase, but this has made the reverse difficult.
• I would love to see some more work on improving clojure.test. It works, but I wish there was less stateful macro magic and more flexibility with fixturing and test structure
• I’m increasingly finding that I’m writing more cljc than clj or cljs but find it really frustrating that cljs seems to be a second citizen to clj. It would be much better if there was actually a core Clojure library/api that was actively supported by both, with clear variances between them.
• I would love to see some work towards using JVM MethodHandle’s to obviate the need for the compiler’s direct linking. Seems we could have our var cake, and eat it too.
• We have a relatively large codebase and 20+ build targets (fat jars/advance compilation js files). we’d like better AOT support and ability to incrementally compile this code base as most changes only affect a few targets. right now AOT compile is pretty black box and not very amenable to building tooling.

Language Development Process

This hasn’t changed in a long time, and I don’t see it changing in the future, but it has been persistently highlighted by many people as a big issue.

• For long term success of Clojure/ClojureScript, it’s important for the Clojure core team & project to be more open and welcoming like other open source projects. The view from outside of the Cognitect bubble is that the language and community aren’t healthy at all, and I fear for it’s long run viability.
• The core team seems to be extra cautious and move really slowly on the language and core libraries. And the roadmap for releasing new versions is really opaque to the community.
• Cognitect has continued to be good technical stewards of Clojure, but poor stewards of the relationship between the core team and the rest of the community. Many (most?) senior Clojure programmers want nothing to do with contributing to Clojure or any projects owned by Core (like the site) because the process is designed to frustrate and prevent meaningful contributions.
• Cognitect need to climb a few rungs down the ivory tower and engage with the community as if it were in fact made up of human beings trying to do their jobs.
• I don’t know what direction the core team is taking Clojure, and the top issues (startup time, poor error messages, lack of involvement with the community and numerous improvements to Java interop in JIRA being ignored) have not changed significantly over the past 5 years. I never felt welcomed to contribute, and seeing how Skummet and other large efforts from the community were largely ignored was not encouraging.
• More attention to JIRA issues/patches/contributions would be great. […] I painstakingly write a patch and jump through the JIRA hoops; but there it languishes. The dev team’s complete silence is not encouraging. Makes me feel more of an outsider, less of a part of the community.

Community

By and large, people found the community welcoming, and personally I have found it to be one of the best things about working with Clojure. However there is a persistent undercurrent of eliteness from some Clojure programmers which really puts people off the language.

• I find Clojure a great language, but I haven’t noticed any increase in market (or mind) share. Quite the opposite, it seems to be going nowhere, possibly dying a slow death I hope I’m wrong, but, for now anyway, I’ve given up on it.
• Thanks, Alex et al! I’m still extremely happy with Clojure, and appreciate the community. I worry a bit about the possibility of clj/s losing user share and dwindling, which I didn’t worry about a year ago. I hope like hell that that won’t happen, and try to do at least a bit toward preventing it.
• That defn podcast is bloody marvelous
• Being in the Clojure community since around 1.2, I loved the language then, and I like the direction, it’s been taking since. The community is great, very friendly and even its most controversial figures are usually polite, intelligent people. Heck, even the local troll from the mailing list is occasionally helpful, stays on point in his threads and sticks with the conversation, even after blowing stuff up.
• clojure.org changes so infrequently (like once a year, if that?) that I’m concerned the language is not being enthusiastically supported and will soon fade away. It makes me nervous to invest time and money in a language that seems to have no vitality.
• Clojure conferences lack diversity in both attendees, speakers, and programs. I hope the merging of Conj and West will improve that this year.
• People who give to the Clojure community are role models. I’m serious about this. I’ve woken up and thought “what would Bruce Hauman do” more than once. I talk with my daughters about the Clojure community.
• Cognitect is doing an awesome job, making programming fun again! Alex Miller is very helpful.
• Effort community makes to promote diversity, particularly in terms of women is good, although there is a long way to go.
• I am doing Clojure for some time now and what does not bring me joy is the fact that the message to other communities (other languages) is clear: “we are better and smarter”. […] I think we can do better without thinking that we are.
• I didn’t want to tick the unpleasant communities cause it is not […] lately [however], […] I started noticing a kind of “elitism” especially from people that are frequent speakers or started working early with the language.
• I have multiple colleagues, all great developers, all former Clojure developers. Whenever Clojure comes up they never fail to mention how alienating the “smarter than you” attitude sometimes present in the community can be.
• I’m worried about the perceived shrinking of the Clojure community
• I’ve gotten to know a few more female developers using Clojure in the past year, and have learned that some parts of the community are scaring/driving women away. This includes unpleasant interactions on Slack and working to avoid certain people at Conj.
• Not to put too much stock in the oversimplified Gartner hype curve model, it does seem like Clojure is passing through what they call the trough of disillusionment.
• Please consider supporting Clojurists Together, even launching initiatives for the community to help through it.

Libraries

More guidance on how to compose and select libraries remains a common common issue for people in this section, as well as improving the state of library documentation.

• The plethora of web app (both client & server side) frameworks is both bad and good. Too many to choose from (no standard like rails) and also lots of options. Also, limited security options, especially w.r.t. use of session state, async handlers & server-less designs.
• I frequently find myself frustrated by looking for a library that will help me do “X”. There are many choices out there, each of which does some subset of what one might want in that arena, but a different subset.
• It is sad that I often find interesting Clojure or ClojureScript libraries that are abandoned and haven’t been updated for a long time
• Need more quality open source projects for machine learning and data visualization libraries. Great to see the advances from conj but for a language that has a front end tool built in, there is no excuse for someone not to build a great visualization library for data science.
• No de-facto web framework, Pedestal is in dire need of documentation, Schema is deprecated and Spec isn’t ready, New CLI tools in Clojure 9 competing with Leiningen/Boot. All of these complaints have something in common: it looks like the Clojure core team is actively hostile towards the larger community. All community-built tools/libraries are open to attack and replacement by the core team and there’s always fear of a major destabilizing revolution everyday.
• Several widely used library started to stall… sustainability might become a concern… [Ed: hopefully Clojurists Together can help here]

Other targets

As in previous years, there are a smattering of people asking for alternative platforms to target other than JS and the JVM. LLVM, Go, and Web Assembly all had interest from people.

• Is it posible to use Clojure with the Go ecosystem? That would be amazing
• It is hard to watch Go get so much mind-share. Clojure on the JVM is great, but it would sure be nice if I could build a Clojure application that didn’t need the JVM.
• I still long for a low-level oriented Clojure dialect to be able to produce small clj executables with low overhead and systems programming capacity, but one can dream!
• webassembly support? and native support? (like a scala-native)

Typing

Requests for static typing were down from previous years. I attribute that mostly to spec gaining wider adoption and understanding.

• Clojure needs to make a better case for why static typing is not necessary for maintainable, large, multi-contributor codebases and how the REPL dev process helps with that.
• I still love Clojure as a whole more than any other. I’d love to address long-term maintenance. The difficulty of stepping back into a codebase after some time can be more challenging than with Haskell or PureScript. I wish the compiler could do more for me. Spec is huge though, really enjoying it. It still takes a good amount of discipline to use well, but I’m thankful for it.
• I used Clojure for one work project because a coworker recommended it. Overall it was a good experience, but the lack of a type system made maintaining that particular project too difficult. Tools like Plumatic Schema and Typed Clojure helped in some ways but didn’t end up pulling their weight. Regardless, the Clojure service continues to run today and it’s better than the Ruby service it replaced.

ClojureScript

ClojureScript continues to improve, although getting the tooling setup remains a pain point, as does integrating NPM modules. There are recent improvements on that front though, so this may not be such an issue in 12 months time. I have heard lots of good things about shadow-cljs, it seems to be a strong tool to investigate for ClojureScript usage, especially if you work with a lot of NPM modules.

• ClojureScript has had a serious push this year… ;)
• For CLJS, we really need to get npm deps working. Right now it’s hard to get right, or at least there’s no up-to-date tutorials available..?
• I find I spend more time than I should configuring my development environment so it works well. This is particularly bad in React Native, but cljs/figwheel can still be frustratingly complex to get right.
• Lack of async/await support in CLJS is holding it back. The JS community has settled on promises and async/await as the primary asynchronous patterns, and although core.async is useful for certain things, shoehorning existing JS code into it is very kludgy.
• Nothing has made a bigger improvement to my ClojureScript workflow in the past couple of years than shadow-cljs.
  • Intelligent externs inference, plus simple ^js or ^clj tagging for anything that’s missed
  • Virtually trouble free usage of other JavaScript modules /npm
  • Way faster cljs compile flow due to smart caching of intermediate artifacts
  • Self-hosted dependencies handled very well
• NPM support in ClojureScript seems to be a little bit tricky and not working all the time for all the libs. But has been improved a lot in the past.
• Probably my biggest frustration has been trying to use libraries from JS in a full-on ClojureScript front-end app.

Setup/tooling

Tooling on the Clojure side seems to be somewhat less of an issue this year, but setting up ClojureScript projects with all the things that you need remains an issue for many people.

• Clojure is now mature and stable enough to start focusing in tooling. Many JVM tools are not Clojure specific and for people coming to Clojure from non-jvm languages diagnosing problems is a barrier (more with systems in production). From GC issues, to debugging exceptions, to attaching to a process in runtime to peek at some memory or patch something. It was pretty easy to do with gdb in linux, it should be much more easily done with Clojure.
• Fast track to CLJS web dev for junior devs, in one piece in one place. There is not that much essential knowledge/tooling needed to get started that it can’t fit in a CreateReactApp-like solution. Too much content, too many sources, don’t help. No need to be exhaustive.
• Git-based deps seem like a bad idea. clj build tool is otherwise cool, if a bit unnecessary given the prevalence and quality of leiningen, along with the stability afforded through maven central/clojars.
• I think we should pay more attention to interactive programming. Like what Bret Victor proposes, beyond the REPL.
• Making changes to libraries then having to jar them just to test is hard (checkouts never worked for me (or others) with cljs). I hope Alex Miller’s recent work will fix that.
• Once Figwheel and Reagent and tests were set up, my experience as a developer was very positive - barring some bad times with advanced Closure optimization bugs. Really happy with ClojureScript myself, but it would be hard to convince a friend to start using it, because there is no clear or standard way to develop.

Compliments

As always, the survey was dominated by positive comments. Here were some of my favourites.

• Clojure has ruined every other language for me. Every time I have to do something in another language, I always wish it was in clojure. It’s just so easy to get things done in clojure. I love the language and hope to use it forever.
• Clojure’s still the language I find myself turning to most frequently when given a choice. It just feels natural, the toolkit simple and yet powerful enough to express pretty much anything at a whim.
• I’ve been using Clojure professionally for at least 4 years and these have been the most productive years of my career. I’m looking forward to using spec more fully. I’m looking forward to seeing what improvements come to the language in the next few years and have no desire to leave. Thanks so much for this language, community, and way of working.
• This. Is. The. Best. Language. Ever. High-level programming done right. Thank you Rich.

Conclusion

When I went to write this years summary, I re-read the previous posts to remind myself of where we were a few years ago. While some areas have improved, I was struck by how many of the comments could have been lifted from previous years survey results. There was a worrying trend in the community and adoption sections that people perceive Clojure to be shrinking. Perception doesn’t always match reality, and it may be (as a commenter noted) that Clojure is entering the trough of disillusionment phase of the Gartner Hype Cycle. However I think there are a lot of low-hanging fruit around that could significantly improve the experience and ecosystem around Clojure without changes to the core language.

When I develop ClojureScript projects, I almost always use Figwheel. It’s a great tool, but sometimes my app ended up using stale files. This led to some very confusing debugging sessions. It only happened some of the time, and was always fixed after a hard refresh. I thought about just disabling the browser cache, but I didn’t like ignoring the issue. After seeing colleagues struggle with stale caching too, I decided to figure out what was going on, and fix it once and for all.

Cache-Control rules everything around me

The first thing to do was to add a Cache-Control: no-cache header to all static file responses. Despite the name, no-cache tells the browser it can cache files, but must always validate them with the server before using them. If the browser’s cached version is up-to-date, a compliant HTTP server should return a 304 Not Modified response, otherwise it serves the new file.

If you don’t provide a caching header to an HTTP response, the browser can choose its own caching behaviour. The browser’s caching heuristics are much more aggressive than you want in development, and lead to the weird caching behaviour I was seeing.

I thought this had fixed the issue, but occasionally I would still notice stale files were being used. After looking closely at the compiled output files, I made a surprising discovery.

ClojureScript copies file modification times

ClojureScript (as of March 2018) copies the last-modified date of ClojureScript source files to the compiled JavaScript target files. This is so that the compiler can detect changes to source files. JavaScript from the Closure compiler (e.g. goog.base), gets a modification time that matches the time it was compiled.

Neither of these dates are particularly useful to use as a Last-Modified date header for caching purposes.

• Closure compiled JavaScript doesn’t change from run to run, so caching based on last modified date will not achieve as high a hit-rate as possible.
• ClojureScript files that use macros from other Clojure files will copy the ClojureScript files modification date to the compiled JavaScript, even if the macro files have changed and are newer. This was leading to the second round of caching issues that I saw.

ETags

To knock both problems on the head once and for all (hopefully), I added a CRC32 checksum based ETag for static file responses. I packaged this up in a library ring-etag-middleware so that other projects could also use it. I also removed the Last-Modified header from the Ring response, as it would bust the browser’s cache unnecessarily.

Summary

As best as I can tell, this has completely solved all of the odd caching issues that I was seeing, while still keeping the app snappy to load by reusing as much of the cache as possible. If you are serving ClojureScript files in development and not using Figwheel, I recommend you follow these three steps:

1. Set a Cache-Control: no-cache header
2. Add an ETag to your static file responses
3. Remove the Last-Modified header

I was excited this morning when I checked Hacker News and saw an article from Cockroach Labs titled “Your Database Should Work Like a CDN”. I’m a big fan of Cockroach DB and enjoy their blog posts. I clicked on this one, but came away disapointed. I don’t think it fairly contrasted Cockroach with it’s competitors, and instead made some incomplete arguments instead of selling Cockroach on it’s merits.

In this article I will analyse sections of the post and add more context where I think some was left out.

Analysis

Availability

To maximize the value of their services, companies and their CTOs chase down the elusive Five Nines of uptime

Only companies with sophisticated operations teams can seriously set an SLA for five nines. It’s doable, but comes with a heavy lift if your service is doing anything non-trivial. It’s certainly not the default position among the industry as far as I can tell. The Google SRE book has a good section on this.

For many companies the cost of moving from a single region deployment to a multi-region one is too great and doesn’t provide enough benefits. Some services don’t actually need to be available to five nines, and if not done well, moving to multi-region deployments may make your system less fault-tolerant, not more.

This is particularly crucial for your customer’s data. If your data’s only located in a single region and it goes down, you are faced with a “non-zero RPO”, meaning you will simply lose all transactions committed after your last backup.

If a whole region dropped into a sinkhole in the earth then you would lose all transactions after your last backup. However that’s not a failure scenario that we tend to worry about (although maybe we should?). Every time a major cloud provider has had a zone/region go down, data was partially/wholly unavailable during the outage, but no committed data was lost once service was restored (that I know of?).

…and Bureaucracy?

This impending legislation requires that businesses receive explicit consent from EU users before storing or even processing their data outside the EU. If the user declines? Their data must always (and only) reside within the EU. If you’re caught not complying to GDPR? You’ll face fines of either 4% of annual global turnover or €20 Million, whichever is greater.

I’m not a GPDR expert, but this seems like an incomplete summary of the GPDR rules around processing, especially as the processing rule applies only to the processing of personal data (AFAICT), and conflates consent for storing/processing personal data with storing the data outside of the EU. From article 6:

Processing shall be lawful only if and to the extent that at least one of the following applies:

a) the data subject has given consent to the processing of his or her personal data for one or more specific purposes;

b) processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract;

c-f) [Public/government interest exceptions)

If you have entered into a contract with the data subject, and that data is necessary for the contract, then I think you are fine (IANAL). As far as I can tell, transfers to Third Countries are OK if there are appropriate safeguards.

This is the part of my critique that I am least confident about, I’d welcome some links confirming/rebutting this.

When you take GDPR in the context of the existing Chinese and Russian data privacy laws––which require you to keep their citizen’s data housed within their countries […]

If you follow the links in the article, you see this for China:

Does the Cybersecurity Law require my company to keep certain data in China?

[…] To that end, the Cybersecurity Law requires “critical information infrastructure” providers to store “personal information” and “important data” within China unless their business requires them to store data overseas and they have passed a security assessment. At this point, it remains unclear what qualifies as “important data,” although its inclusion in the text of the law alongside “personal data” means that it likely refers to non-personal data. […]

“Critical Information Infrastructure” providers are defined a bit more narrowly, but the law still casts a fairly wide net. […] the law names information services, transportation, water resources, and public services, among other service providers, as examples.

and this for Russia:

3.6 Generally, to transfer personal data outside the Russian Federation, the operator will have to make sure, prior to such transfer, that the rights of personal data subjects will enjoy adequate and sufficient protection in the country of destination.

Some companies will need to store data on China/Russian servers, but the laws here are far narrower than “If you have any data from a Chinese or Russian person or company, you must store it in their country”.

Managed & Cloud Databases

Managed and Cloud databases often tout their survivability because they run in “multiple zones.” This often leads users to believe that a cloud database that runs in multiple availability zones can also be distributed across the globe.

It might be an assumption you would make if you have no experience with databases, but it’s not one that I’ve ever seen a software engineer make.

There are caveats to this, of course. For example, with Amazon RDS, you can create read-only replicas that cross regions, but this risks introducing anomalies because of asynchronous replication: and anomalies can equal millions of dollars in lost revenue or fines if you’re audited.

I’m not sure how asynchronous replication lag could result in failing an audit and incurring millions of dollars of fines. I spent a few minutes trying to come up with a scenario and couldn’t. Losing revenue from users also seems speculative, I’m not really clear how this would happen.

Designing a system to run across multiple regions with asynchronous replication is certainly not trivial, but people do it every day. If they were losing millions of dollars from it, they would probably stop.

In addition, this forces all writes to travel to the primary copy of your data. This means, for example, you have to choose between not complying with GDPR or placing your primary replica in the EU, providing poor experiences for non-EU users.

Again, I don’t think GPDR requires this.

NoSQL

For example, NoSQL databases suffer from split-brain during partitions (i.e. availability events), with data that is impossible to reconcile. When partitions heal, you might have to make ugly decisions: which version of your customer’s data to you choose to discard? If two partitions received updates, it’s a lose-lose situation.

This paints all NoSQL databases with a broad brush. While some NoSQL databases are (were?) notorious for losing data, that is not inherent to NoSQL databases.

Certainly if you are using an AP NoSQL database, you need to design your application to correctly handle conflicts, use CRDT’s, or make idempotent writes. Partitions do happen, and it’s not trivial to handle them correctly, but neither is it the case that you always need to discard your customer’s data.

Sharded Relational Databases

Sharded Relational databases come in many shapes and suffer from as many different types of ailments when deployed across regions: some sacrifice replication and availability for consistency, some do the opposite.

I assume the author is referring to systems like Citus. I don’t have enough experience with systems like this to judge the assertion, but this seems fair.

Conclusion

If you do need more reliability/availability than is possible from a single region, then Cockroach is a strong option to consider. I think a far better argument for CockroachDB and against NoSQL, Replicated SQL, and Sharded Relational DBs is minimising complexity and developer time. It is possible for developers to design their application for the nuances of each of these databases, but it’s certainly not easy or cheap, especially if you want it to be correct under failure. The reason Google created Spanner (the inspiration for Cockroach) was that developers found it hard to build reliable systems with weak consistency models.

[…] It is much easier to reason about and write software for a database that supports strong consistency than for a database that only supports row-level consistency, entity-level consistency, or has no consistency guarantees at all. - Life of Cloud Spanner Reads & Writes

CockroachDB provides consistent reads and writes, supports SQL, is able to be deployed multi-region, and in any datacenter. Those are a compelling set of features. If your application can handle the latency and performance tradeoffs that it makes (which are getting better all the time), then it will let you write software against a consistent datastore without spending as much time reasoning about hard consistency problems. Cockroach is a great product, and I think it stands well on it’s merits.

Since Terraform v0.10, Terraform providers are distributed separately from the Terraform binary. This lets them update at different paces, and allows a wider group of people to collaborate on the providers. This is mostly good, but it does introduce a new step for upgrading providers. It is slightly counterintuitive, but to upgrade your providers, run

terraform init -upgrade

To upgrade your modules, run

terraform get -update

I was building VM images for Google Cloud with Packer, and provisioning them with Ansible. Everything had been working in the morning, but in the afternoon one computer wasn’t working after I had upgraded Ansible with Homebrew. I was having a really tough time figuring out why Ansible and Packer were running fine on one computer, and not on the other. I was getting the following error:

googlecompute: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"127.0.0.1\". Make sure this host can be reached over ssh", "unreachable": true}

Investigating the error, I found it could indicate any number of problems, as Ansible is masking the underlying SSH error.

After checking the versions of Python, Ansible, and Packer on the two computers, I found one difference. On the computer that wasn’t working, when running ansible --version it had a config file listed:

$ ansible --version
ansible 2.3.2.0
  config file = /Users/<user>/.ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Dec 18 2016, 07:03:39) [GCC     4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]

Opening that file:

$ cat ~/.ansible.cfg
[ssh_connection]
pipelining = True

I moved that file to another location so it wouldn’t be picked up by Ansible, and after rerunning Packer a second time, got this error:

googlecompute: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '[127.0.0.1]:62684' (RSA) to the list of known hosts.\r\nReceived disconnect from 127.0.0.1 port 62684:2: too many authentication failures\r\nAuthentication failed.\r\n", "unreachable": true}

I then used the extra_arguments option for the Ansible provisioner to pass [ "-vvvv" ] to Ansible. I ran this on both computers and diffed the output. I saw that the dynamically generated key was being successfully provided on the working computer (after one other local key). On the failing computer I had many SSH keys that were being tried before I could get to the dynamic key, and I was getting locked out.

SSH servers only allow you to attempt to authenticate a certain number of times (six by default). All of your loaded keys will be tried before the dynamically generated key provided to Ansible. If you have too many SSH keys loaded in your ssh-agent, the Ansible provisioner may fail authentication.

Running ssh-add -D unloaded all of the keys from my ssh-agent, and meant that the dynamic key Packer was generating was provided first.

I hope this is helpful to someone else, and saves you from hours of debugging!

Postscript

I was very confused by seeing that my computer was trying to connect to 127.0.0.1, instead of the Google Cloud Platform VM. My best guess is that Packer/Google Cloud SDK proxies the connection from my computer to the VM.

While working on Deps I wanted to detect the user’s time zone at signup, to localise the times that I presented to them. I hunted around and found a variety of libraries that offer time zone detection, like moment.js, and jstz. However I wasn’t keen on pulling in a dependency on one of these libraries if I could help it, so I kept looking. I also considered trying to detect the user’s time zone from their IP address using something like the Maxmind GeoIP database, and that probably would have been the method I settled on.

However in my research, just before I started with IP based detection, I found that there is a newish Internationalization API, which will give you the user’s system time zone.

Intl.DateTimeFormat().resolvedOptions().timeZone
=> "Pacific/Auckland"

All current browsers support the Internationalization API including IE11. However there is a slight wrinkle to keep in mind when using it: on some older browser versions that do support Internationalization (IE <=11, and Firefox <=51), the timeZone property is set to undefined rather than the user’s actual time zone. As best as I can tell, at the time of writing (July 2017) all current browsers except for IE11 will return the user’s actual timezone. If you do need to support older browser environments then you could look at moment.js. It uses the newer Internationalization API when it is available, and falls back to other heuristics if it isn’t. For my purposes, my browser statistics show almost no IE usage, so I chose to just use the newer method, while also allowing the user to manually update their time zone if they needed.

Bryan Cantrill on integrity. Edited slightly for clarity and profanity:

We are on a collision course with the Amazon Principles […] What is going on? I’m losing my mind. The wheels are off, everyone has to get out of the car and this is why. All of these [expletive] leadership principles, from all these organisations, where is integrity? Damn it where’s integrity? Amazon has 14 leadership principles and integrity is not on it. [That’s] inexcusable. I’m sorry, if you’ve got one principle in your organisation, its integrity? Right? […] No we’re living in a world that has lost it’s [expletive] mind, I don’t understand it. Why the appetite for territory? Do you not know where your next meal is coming from? Do you not have a roof over your head? I mean I’m sorry, I just don’t get it. We have got an incredible luxury. There’s never been a labour market like this one. Where have we so screwed up with a generation, if not a society where we’ve got people who are so extrinsically motivated? What the [expletive] is wrong with us? I mean its like how is integrity not the only thing you have?

[Andrew Clay Shafer]: Are we doing the full critique of capitalism today?

No it’s not. Bullshit, it’s not capitalism. The finest capitalist is Scott McNealy, a capitalist so pure that when he was being devoured by Oracle he believed that it was his duty to capitalism to die, and I admire that. There is no purer a capitalist than Scott McNealy. And read McNealy’s final email to Sun employees, and it’s almost prescient, McNealy says: “You know what, in 30 years I never had to hide the newspaper from my children.” An achievement that Uber violated on like month two? Where are Travis’ parents? Are you not humiliated? How did you raise him to be so divorced from what really, actually, truly matters? What the living [expletive] is wrong with us? Maybe it’s me?

I continue to be impressed by Joyent, and Bryan Cantrill’s engineering principles. It sounds like a great place to do your best work.

Recorded at GOTO 2017 for an Arrested Devops episode: Old Geeks Yell At Cloud

Last week, there was a mailing list post on the clojure-dev mailing list noting that several of the new predicates in Clojure 1.9 alphas return true or nil. The three new predicates are: qualified-keyword?, qualified-ident?, and qualified-symbol?. Previously, all predicate functions in Clojure core, i.e. functions that end in a ? have returned true/false. In this post I want to examine the pros and cons of this new behaviour.

If I’ve missed anything here, please get in contact and I’ll update the post.

Pros to some predicates returning nil

• The main argument for some predicates returning nil is that the predicate still returns a falsy value. Idiomatic Clojure code usually doesn’t need to distinguish between nil and false.
• The docstrings for qualified-keyword? for example says: “Return true if x is a keyword with a namespace”. It doesn’t say anything about what happens if it doesn’t have a namespace, so returning nil or false are both technically valid interpretations.
• Not coercing the output from qualified-keyword? into a boolean is faster. On my computer qualified-keyword? takes roughly 3.5 ns to run, and a version that returns a boolean value takes around 6.5 ns. In absolute terms they are both pretty small though.

The first two points aren’t strong arguments for returning nil, rather they argue that it doesn’t matter whether the functions return nil or false.

Cons to some predicates returning nil:

• The biggest downside to this change is that it breaks a core convention that Clojure and the community has held around predicates, namely that any function ending in ? returns true or false. This is my biggest concern about the changes. You can find this expectation in:
  • Programming Clojure: “A predicate is a function that returns either true or false. In Clojure, it is idiomatic to name predicates with a trailing question mark, for example true?, false?, nil?, and zero?.” - Programming Clojure, 2nd Ed., Page 27.
  • Clojure’s own library coding standards: “Use ‘?’ suffix for predicates. N.B. - predicates return booleans”. That page also has the disclaimer: “Rules are made to be broken. Know the standards, but do not treat them as absolutes.”
  • The community Clojure Style Guide: “The names of predicate methods (methods that return a boolean value) should end in a question mark (e.g., even?).”
  • All of the Clojure standard library functions that end in a ? return a boolean value. Anyone learning Clojure would be justified in assuming that all functions that end in ? return a boolean value if that’s the convention they have always seen.
  • A mailing list thread on predicates in the aforementioned Programming Clojure book.
  • A mailing list thread about this exact question, whether all Clojure functions that end in ? should return a boolean value.
  • Tutorials Point: “Predicates are functions that evaluate a condition and provide a value of either true or false.”
  • StackOverflow on naming rules: “The main function naming conventions seem to be … Use ? to indicate a predicate that returns true or false: sequential?”

• The second biggest problem is that qualified-keyword? can return three values. “How is that?” you might be asking, “Didn’t you just say that the functions return true or nil?”. I thought so too. However as I was looking at the implementation of these functions, I realised that qualified-keyword? can return false as well, if not given a keyword. For example:

  user=> (qualified-keyword? ::abc)
  true
  user=> (qualified-keyword? :abc)
  nil
  user=> (qualified-keyword? 'abc)
  false
  

  It is fairly common to use group-by or juxt with a predicate function to partition a collection into two groups. If you use any of the new qualified* functions, you will end up with three groups. Based on Alex Miller’s tweet, it seems like this was considered and accepted.

• If these predicates are used at the boundaries of systems or in interop with Java, then it would be easy to forget that the qualified* predicates may return nil, and end up passing null into a Java method as a parameter when you meant to pass false. You could also return false under certain input when you expected the function to only return true or nil, or return nil when you were expecting the result to be a boolean. These kinds of bugs can be very subtle, especially with Clojure’s falsy handling.
• ClojureScript code is a lot smaller and enables more optimisations when it can guarantee that a boolean value is returned.
• Clojure gets similar benefits from the JVM when it can mark the function as returning a Boolean.
• Historically, Clojure has made a distinction between functions that return truthy and those that return true. There are predicate functions in core that return a truthy value like some and every-pred, but they don’t end in a ?. This change starts to dissolve that distinction.

Summary

Personally, I don’t see a strong reason to keep the current behaviour of the new qualified predicate functions, but you might see things differently. Whatever the outcome, I’m glad that in Clojure we don’t have to name all of our boolean functions with an “Eh” suffix.

Update

Alex Miller suggested to file a ticket about this behaviour, so I opened CLJ-2141.

Since my post a few days ago on the Uber-Alphabet lawsuit, there have been a few new developments.

Levandowski consulting for Uber

In that post I wrote:

… no funding appears to have been taken from Uber, or any other company (If there turn out to be payments from Uber to Otto prior to the acquisition then that would be Very Bad for Uber).

Not long after, Johana Bhuiyan at Recode posted a story which suggests that Uber CEO Travis Kalanick hired Otto CEO Anthony Levandowski as a consultant for Uber’s self-driving car around February-March 2016. Even after reading the article several times and sleeping on it, I still can’t quite believe it.

Now, sources are adding more to that claim. Kalanick hired Levandowski as a consultant to Uber’s fledgling self-driving effort as early as February 2016, six months before Levandowski publicly got his company off the ground, according to three sources.

If this was a legitimate business arrangement, the timing makes no sense at all. Levandowski had just left Waymo weeks earlier. While it might have been difficult for Alphabet to bring a lawsuit on these grounds, the optics for Levandowski advising Uber are terrible. Additionally, Uber and Otto are competitors in the self-driving vehicle space, and Uber is famously aggressive. That a CEO of one company would be advising one of their direct competitors within months of forming their own self-funded startup is truly strange.

If this wasn’t a legitimate business arrangement, then it also doesn’t make much sense purely because it looks so suspicious, especially in light of Alphabet’s trade secrets lawsuit. It brings the start of Uber’s direct involvement with Otto to within weeks of Otto being formed. If the plan all along was for Otto to join Uber, then it does make some kind of twisted sense for him to be advising the Uber self-driving car team.

A little birdie tells me that we haven’t seen the last on this particular thread of the story.

Naked Capitalism on Uber

Hubert Horan has just published another article in his series on Uber and whether it can ever increase overall economic welfare.

The answer is that Uber is attempting to implement an innovative and disruptive strategy for achieving market control — a strategy radical different from every prior tech oriented startup — but those disruptive innovations have nothing to do with technology or the efficient production of urban car services.

Otto purchased another Lidar company

A reader sent me a link to this article by Reuters discussing an acquisition of Tyto LIDAR. The title is “Uber’s self-driving unit quietly bought firm with tech at heart of Alphabet lawsuit”. The title isn’t accurate, as it was Otto that acquired Tyto (for an undisclosed sum) in May 2016. Uber then acquired Otto/Tyto in August 2016.

Levandowski: “We did not steal any Google IP”

Forbes published an article on 28 February, 2017 which contains part of an interview they did with Levandowski in October 2016.

Q: The question is, Google has been working on this for however many years. You guys (Otto) are a year old or something like that.

A: We started in January. How did we catch up?

We did not steal any Google IP. Just want to make sure, super clear on that. We built everything from scratch and we have all of the logs to make that—just to be super clear.

But this is my fourth autonomy stack. I did the DARPA Grand Challenge.

I wonder if “we have all of the logs” is also going to be Alphabet counsel’s opening argument?

If you’ve been following tech news over the last few weeks, you have probably seen several stories about Uber, all negative (bar this one about flying cars). I suspect that what is coming next will prove to be a far bigger story than all of the other incidents so far.

N.B. all of this article is sourced from filings and allegations that Alphabet has made, as well as reading between the lines. Uber will probably contest these claims in court.

Update: I’ve published some updates on the lawsuit. You can keep track of all the posts related to the Alphabet lawsuit here.

In the last few weeks Alphabet filed a lawsuit against Uber. Alphabet and Waymo (Alphabet’s self-driving car company) allege that Anthony Levandowski, an ex-Waymo manager, stole confidential and proprietary information from Waymo, then used it in his own self-driving truck startup, Otto. Uber acquired Otto in August 2016, so the suit was filed against Uber, not Otto.

This alone is a fairly explosive claim, but the subtext of Alphabet’s filing is an even bigger bombshell. Reading between the lines, (in my opinion) Alphabet is implying that Mr Levandowski arranged with Uber to:

1. Steal LiDAR and other self-driving component designs from Waymo
2. Start Otto as a plausible corporate vehicle for developing the self-driving technology
3. Acquire Otto for $680 million

Below, I’ll present the timeline of events, my interpretation, and some speculation on a possible (bad) outcome for Uber. The timeline references section numbers from Waymo’s amended filing, so you can read the full context yourself. You can also read the original filing.

Timeline of events

The main timeline of important events is as follows:

• Summer 2015 - Anthony Levandowski told Pierre-Yves Droz, a colleague at Waymo, that he had talked with an Uber executive about forming a self-driving car startup and that Uber would be interested in buying that startup. (Droz 28)
• November 17, 2015 - Levandowski registers a domain for 280 Systems, the company that would later become Otto. 280systems.com can be linked to a public email sent February 4, 2016 from someone with a 280systems.com email address looking to do testing of a semi truck with “specialized equipment”. (Filing 41)
• December 3, 2015 - Mr Levandowski searched the Alphabet intranet for instructions on how to access Waymo’s design server on his work laptop. Based on Gary Brown’s deposition (a Google forensic security engineer) this was an SVN server. (Brown 15)
• December 11, 2015 - Anthony Levandowski installed TortoiseSVN and downloaded 9.7 GB of data from the SVN repository. (Brown 17)
• December 14, 2015 - A USB card reader was attached to the laptop for eight hours. Google doesn’t appear to have logged what the laptop did over that time, but the implication is that data was copied from the laptop to a memory card. (Brown 18)
• December 18, 2015 - Levandowski reformatted his work laptop from Windows to Goobuntu (Google’s custom version of Ubuntu). This laptop wasn’t used again after December 21. To be fair, it was only used three times between March and November 2015. Presumably, he was still doing work during this time, just on another computer? (Brown 19, 20)
• January 4, 2016 - Levandowski downloaded five confidential technical Waymo documents from Google Drive to a personal device. (Brown 22)
• January 5, 2016 - Levandowski took a walk with Droz. In Pierre-Yves’ deposition, he claims that Levandowski “told him that he planned to ‘replicate’ Waymo’s technology at a new company he was forming.” (Droz 27)
• January 11, 2016 - Levandowski downloads another file from Google Drive relating to Waymo’s self-driving car development schedule and timeline. (Brown 23)
• January 14, 2016 - Levandowski was seen meeting at Uber’s headquarters and the news travelled back to Droz. Droz asked Levandowski about this, and he admitted he had met with Uber and was looking for investors for his new company. (Droz 29, Filing 48)
• January 15, 2016 - Levandowski officially forms 280 Systems (in stealth mode). Note that this was one day after his meeting with Uber. (Filing 49)
• January 27, 2016 - Levandowski resigns from Waymo without notice. (Filing 49)
• February 1, 2016 - Levandowski forms Otto Trucking (this is also in stealth mode). (Filing 49)
• Spring (March-May) 2016 - “Kalanick began courting Levandowski this spring, broaching the possibility of an acquisition … The two men would leave their offices separately—to avoid being seen by employees, the press, or competitors.” - Bloomberg. Update: I forgot about this article, but was reminded by hammock on Hacker News.

• May 17, 2016 - Otto launches out of stealth mode. As far as I can tell, they never took on any venture funding, instead self-funding (emphasis mine):

  Many of Otto’s founders have done well for themselves over the years, and it shows: the company is entirely self-funded right now without any external investment. (In the wake of the reported $1 billion Cruise Automation sale to General Motors, I ask Ron if the plan is to get acquired, but he’s insistent that they’re focused on bringing a product to market.) Even George Hotz’s scrappy upstart Comma.ai has recently taken on venture funding from Andreessen Horowitz. - The Verge

  In the photo for their announcement I count 35 people. By the time Otto was acquired, they had 91 employees. This seems like a lot of salary commitment to take on via self-funding by Otto’s four co-founders (all ex-Google). On the other hand, depending on the incentive pay they received at Google, they may have had plenty to cover several years of salaries between them.

• August 2016 - Levandowski received his final multi-million dollar payment from Google (presumably a deferred bonus?). (Filing 55)
• August 19, 2016 - Shortly after the final payment was awarded, Uber announced a deal to acquire Otto for $680 million. (Filing 55)
• Summer 2016 - Levandowski’s sudden resignation, Otto’s quick launch, and Uber’s subsequent acquisition of Otto caused Waymo to suspect that their IP had been misused. Waymo investigated this and discovered Levandowski’s actions prior to leaving. (Filing 57)
• December 13, 2016 - A Waymo employee was accidentally copied on an email from one of its LiDAR-component vendors titled OTTO FILES. The email contained a drawing of what appeared to be an Otto circuit board that resembled Waymo’s LiDAR board and shared several unique characteristics with it. (Filing 59)
• December 2016 to February 2017 - Waymo tried to obtain further information on whether Uber was using their LiDAR designs. This is also known as “Getting your ducks in a row”. (Filing 60)
• February 9, 2017 - A Nevada public records request turned up a filing Otto/Uber made that they were using an “in-house custom built 64-laser” LiDAR system. This was enough to confirm to Waymo that they Uber was using a LiDAR system with the same characteristics as Waymo’s. (Filing 61)
• February 23, 2017 - Alphabet makes their first filing against Uber.
• March 10, 2017 - Alphabet amends their filing, and asks for an injunction against Uber’s self-driving car program.

Implications

From Waymo’s filings, it seems that they have Levandowski dead to rights on stealing their LiDAR designs. That alone should be enough to bring Uber’s self-driving car program to a halt and cause some big problems for Levandowski. California’s Trade Secrets law is weaker than other states, but if successful, Waymo will be able to seek an injunction, damages, and attorney’s fees. Because all law is securities law, the SEC may also be able to bring a case against Uber (similarly to their case against Theranos).

I’m guessing, but I think the reason that Alphabet hasn’t directly accused Uber of conspiring with Levandowski is that they don’t have enough evidence. When they get to discovery, they will be looking for it. You would think that no-one would be dumb enough to send emails like that, but you would be wrong.

Several things suggest Otto’s intent to be acquired by Uber:

• Levandowski told Droz in the summer of 2015 that he had talked to an Uber executive about forming a self-driving car company and Uber acquiring them. This is a pretty clear signal!
• Levandowski formed 280 Systems the day after meeting with Uber executives and two weeks before leaving Waymo. He claimed he was talking to them about funding, but no funding appears to have been taken from Uber, or any other company (If there turn out to be payments from Uber to Otto prior to the acquisition then that would be Very Bad for Uber). The timing of Levandowski’s actions suggests certainty about something, and if it wasn’t funding, then what was it?
• According to a Bloomberg article in August 2016, Travis Kalanick and Anthony Levandowski started talking about an acquisition in the Spring of 2016. Spring is usually defined as March-May, which sounds like they may have been talking about an acquisition before Otto had even been publicly unveiled. I’m a bit skeptical about Uber’s statements for this article, given Alphabet’s allegations, but it seems like a weird kind of detail to make up.
• That Otto hadn’t received funding from any VC’s is unusual. With 91 employees, getting paid $150k/year (this might even be too low given they are working on self-driving cars, one of the hottest spaces in tech right now), they would have had a $13.6 million/year burn rate just for salaries. Otto always aimed to get to market quickly, but getting to profitability without funding seems like it would have been very hard, especially on the accelerated timescale they were working towards and the need to likely hire many more people to get to production. All of Otto’s public self-driving car and truck competitors have taken venture funding. However, as someone working on a bootstrapped SaaS application, I’m sympathetic to wanting to self-fund. Update: Counterbalancing this point, it looks like Levandowski had previously sold three startups to Google for nearly $500 million. Those facts are somewhat in dispute, and it’s not clear how much he personally made from the sales. However it does seem plausible that he would have had enough cash to self-fund Otto with his co-founders.
• Otto was acquired only four months after their public launch. While it’s not that unusual for companies to be acquired quickly, it is still very quick, and for a lot of money.
• Uber has considerable internal self-driving car expertise. I’m speculating, but it seems likely that Uber would have found out (or should have found out) during due diligence, that Otto’s LiDAR system could not have been built and developed independently in the six months the company was operating. Updated: Otto purchased Tyto Lidar in 2016, so they may argue that their Lidar system came from Tyto.

There is no smoking gun email (yet), but there is a strong implication that Uber and Otto planned this from the start.

(Possible) Consequences

If this is true and can be proved in court, then it would be a massive blow to Uber. The worst case for Uber would be:

• Uber gets dealt an injunction on their self-driving car project. They have to start again, a long way behind other companies.
• Uber’s name is mud, they struggle to raise more money from investors, especially on good terms. Uber has raised at least $15 billion at a $68 billion valuation.
• Uber is currently losing money at $2-3 billion/year. Uber passengers only pay 41% of the cost of trips, with investor capital making up the difference. Update: I was wrong about Uber passengers only paying 41% of the cost of trips. I can’t find a publicly available number on how the trips are subsidised. Regardless, “Lose a billion here, a billion there, pretty soon, you’re talking real money.” Update 2: See the appendix for more details on this.
• With significant negative margins, no way to become profitable in sight, and a terrible media narrative after the Alphabet lawsuit, sexual harassment, aggressive/illegal behaviour, e.t.c., they cannot IPO.
• Because their self-driving car plans are still so far off, they can’t lower their costs to become revenue neutral.
• I suspect that a large part of Uber’s appeal is their low pricing. If they were to raise prices to cover their driver costs (not even covering the significant costs of their own operations), demand would dry up.
• Without any way to raise more money or reduce their costs, Uber runs out of money and folds.

This is admittedly the very worst case scenario for Uber, and there are lots of places along this downward trajectory that they could pull up from. If it’s proven that Uber intended to acquire Otto while Levandowski was at Google (this might be established more concretely in discovery), then it’s hard to see how Uber’s CEO Travis Kalanick could keep his job. Uber has had a bad month, and it doesn’t look like things are going to be getting better any time soon.

Postscript

This post is made up of a mix of filings from Waymo, reading between the lines, and some speculation. Keep in mind these are all allegations and haven’t been proven in court. Please let me know if I’ve made any mistakes, and I’ll correct them.

I’d encourage you to go over the filings yourself, as they are very readable, and give a lot more context to the story than I was able to here.

Appendix

I originally posted that Uber was only charging 41% of the cost of the trips, based on this article which I misread. The 41% number is the total cost of the trip including driver compensation as well as corporate expenses.

Alyssa Vance sent me the calculations you would make to work out the farebox ratio:

Farebox ratio = fares / total cost of operations Total cost of operations = fares - profits Farebox ratio = fares / (fares - profits)

For 1H2015: Farebox ratio = 3,661 / (3,661 - -987) = 3661 / 4648 = 78.8%.

The State of Clojure 2016 survey results have just been published. Last year I wrote an summary of the free text comment portion of the State of Clojure 2015 Survey. I enjoyed writing it, and it was my second most popular post of the year, so I’m repeating the process again.

In his summary, Justin said that 62% of the responses were positive. That number sounded low to me (but was presumably calculated by SurveyMonkey). I would have estimated closer to 80% after reading through them all. A little under one quarter of the 2420 survey respondents left a comment.

I’m reusing most of the same categories from last year so interested readers can compare with last years post. Some comments have been lightly edited for spelling, clarity, brevity, and niceness.

Error messages

Error messages were the biggest pain point for people last year. This year there were still many complaints about error messages and stack traces, but a lot of people are waiting to see what spec brings. There was a lot of discussion about spec error messages several months ago, but it seems to have gone quiet. I’m really hopeful that spec will improve error messages.

This year Figwheel has started providing very sophisticated reporting of compile errors, and configuration problems. It has made a massive difference to my ClojureScript workflow. Thanks Bruce!

Please Please Please can we get better error messages. …

ClojureScript has come a long way, but I still encounter errors that are totally perplexing. Better debugging would help enormously.

Error messages are very frustrating and it is hard to find where the error occurred and why. Errors coming from underlying Java code are only somehow relevant to the Clojure that produced them. Programming with dynamically typed language rocks when inferred types align with your idea, but when it’s not, things get fuzzy.

Please, please, please, please, improve error messages. This would single handedly improve: new user onboarding, community feedback (let’s admit it - it is a small community), rapid iterations, usage of the REPL, usage of spec and testing, and other areas. …

I consistently see error messages being the biggest barrier for learners of Clojure, and even more so for ClojureScript

I think a concerted focus on improving Clojure’s error reporting & handling would benefit both newcomers to the language and experienced developers alike and would result in considerably less wasted time & effort.

Docs

Last year clojure.org had just been revamped and open sourced. I wrote at the time:

I have high hopes that it will become a central source for high quality, official documentation and ease the pain for newcomers.

clojure.org has had some contributions from the community, but it still doesn’t have the breadth of material needed to be a central source. I would love to see something like Python’s web documentation for Clojure. Renzo Borgatti is writing a mammoth book “The Clojure Standard Library” which will cover the entire Clojure standard library.

… Please improve the documentation. Recently I heard someone describe Clojure documentation as “comically terse”.

A simple way to get someone up and running that is a standard for the community would be great. There is nothing more frustrating than telling new people to “Google” for a solution and pick one you like… They never get further than reading the docs.

Clojure appeared to be a nice language when I started with it and I do not regret this decision. The flip side is that I had hoped that the poor documentation would have gone away by now — 1.4 was the first version I used and it does not seem to have improved. That’s a shame. Documentation helps new users and really, hinders nobody.

Clojure docs are too terse. They assume that the reader already fully understands the concepts being described, and merely needs a refresher. …

Clojure’s community is surprisingly small given it’s output. This leaves documentation and tutorials sparse because they require time and effort that is in short supply. …

I think both clojure.org and clojurescript.org could use a better “Get Started” experience. It’s our best opportunity at a first impression but we rely too heavily on third party docs. For example, on cljs.org the reference section starts with compiler options. Doesn’t really capture the imagination.

Setup/tooling

Again, Figwheel has been leading the way in improving tooling on the ClojureScript side. Tooling is still a pain point, but seems to be less so than in previous years. For Clojure editors, Cider, Cursive, and vim-fireplace are all used pretty much the same amount as last year, but Atom has entered fourth place, presumably coupled with proto-repl.

One thing we’re used to from C#/Java that would be awesome to have is a better unit testing framework with IDE integration. …

The build toolchain for cljs has been a pain. All of my learning has come from cloning public repos and seeing how they do it.

A huge thanks to the community for invaluable tools like Leiningen, Parinfer and the various IDE plugins like ProtoREPL or Cursive. And of course huge thanks to the core team for excellent stewardship of Clojure itself.

While tooling for Clojure is improving there needs to be a lot more resources devoted to it. I am currently in a Java shop and the only realistic option is Cursive.

Startup time

Startup time was another persistent complaints this year. Unfortunately there wasn’t always enough information about whether startup time was a problem in development or production. There are several good options for avoiding constantly restarting Clojure REPLs in development (and in production), but perhaps for newcomers these are too complicated to setup, or they don’t know about them?

Love Clojure … hate its startup time!

Love the language and will keep at it. Primary frustration is with the start up time and tooling. In particular, I’m finding it difficult to quickly test the backend and frontend of my application with Boot.

… We love functional programming, but hate Clojure’s long startup times. We do what we can to minimize this (using test-refresh, REPLs, etc.), but it is still very painful, and we find ourselves twiddling our thumbs waiting for a restart several times a day. Similarly, our Clojurescript compiles take ~60 seconds (:simple optimizations) or ~30 seconds (:none optimizations). This slows us down substantially. Coming from Python, where compile and startup times are sub-second, this is the biggest complaint from our team.

Clojure would be perfect with faster startup time and a little less memory usage. Seriously I don’t even mind the java stack traces.

REPL startup time is all I care about. I use emacs+cider, and it takes an age. Otherwise, I am completely happy with Clojure :)

Marketing/Adoption

Justin specifically mentioned marketing and adoption as a common theme amongst the feedback:

One relatively stronger theme this year was the need for better marketing for the purposes of expanding or introducing Clojure within organizations, which is a great area for contribution from the entire community.

In my opinion, one of the areas where Clojure could improve on marketing is on the main Clojure and ClojureScript websites. I think Scala in particular has done a really good job of this. Elm and Rust also do really well in presenting the main reasons why you would want to use their languages. The community isn’t able to help with this, as contributions to style and templates are not currently wanted.

Last year I mentioned Cognitect’s case studies on Clojure adoption, this year Juxt has done some great work with their case studies of companies using Clojure in Europe.

Anecdotally, I’ve seen fewer people coming along to the Auckland Clojure Meetup than in previous years, although we’re working on a fairly small sample size. I’m not sure what (if anything) to make of that.

I would do 100% of my development in Clojure if the enterprise-y company I worked for allowed it. “Reason for not using Clojure as much as you would like”? Management not on board.

I’d love to see more “serious” support for ClojureScript: when I present ClojureScript to colleagues, it looks like a thing supported by several enthusiasts rather than a platform supported by a company.

Clojure is very solid. As a frontend developer, a stable build system, DCE and syntax stability are very valuable for me. I failed to convince my CTO on Clojure and instead we stuck to ES6/ES7. The main concern of management was hiring and training people. Although the learning curve for Clojure is easy, people still have a perception that LISP syntax is esoteric and difficult to pick up for someone completely new to the ecosystem. This myth has to be busted on a larger scale.

Language is great, community is great, the “marketing” is not that great. You guys have a great language and you are struggling to sell it.

[I] would love to see better marketing from Cognitect and other Clojure centric shops. Selling Clojure to a Java shop is not easy. It can and should be easier. (and simple, but I’ll take “easier” here first)

As an average developer fluent in Java, I would love to use Clojure more, but the biggest hurdle for me is the lack of peers interested in Clojure. At the local meetup people report the same situation at their organisations. Pity, because Clojure is super cool! Thank you.

Love Clojure, really. Can’t convince my peers that it’s worth investing the time to learn it though.

The Clojure community needs a “Rails killer,” and only Arachne holds serious promise for that.

Clojure/ClojureScript ecosystem could benefit from stronger stewardship from Cognitect to propel it into the mainstream with a focus on a killer app or a industry domain similar to the way Light Bend has been focusing on enterprise and reactive system to promote the Scala ecosystem. …

I’m in Shanghai and we barely have full-time Clojure developers in China. We formed online groups, but I guess we need help to push it to the next level.

And the age old developer question:

How can we get management to understand what they don’t understand?

Language Development

Like last year, there are still concerns about the contribution process, and the opaque decision making process for Clojure’s development.

Engagement of community is still sub-optimal, there are talented individuals who could be engaged more and productively so without opening it up to the peanut gallery.

I wish that the core Clojure development process was more flexible and friendly. I (and all the Clojure developers I know) have pretty much given up on trying to make suggestions or tickets about Clojure because I usually wind up banging my head against the wall and getting frustrated. The core team are under no obligations to explain themselves or their motivations, but they would save a lot of frustration from new community members wishing to contribute if there was a prominent document outlining the philosophy, and what kinds of changes and suggestions are and aren’t welcome.

A more open approach to development and contributing to Clojure would be appreciated. I’d love to contribute to Clojure but the mess of a lot of the code in Clojure and the JIRA-based contribution model is a barrier. I know that’s what Rich likes and see the rest of Cognitect are happy contributing in this way but I’d love to see something more open and approachable.

The slow rate at which JIRA issues are addressed remains a frustration, and provides a disincentive for contributing. Features that Cognitect cares about get rushed into the language half-baked, while well-thought out patches from years ago languish.

A little concerned over how ‘arbitrary’ some design decisions seem to be and how inflexible the core team is. Running instrumented post-condition specs is such an obvious idea for example but it has been deemed not necessary “from on high”.

The issue tracker is well tended during the earlier workflow stages. There seems to be a bottleneck later on. I hope 2017 will bring the project’s founders to modes of community contribution that are more efficient of the art and tradecraft of the many.

Community

I would say that Clojure’s community is one of it’s greatest assets. There were mostly positive comments for the community, though still some concerns. The Clojurians Slack has been a great resource for a lot of newcomers, and spawned many channels for libraries and topics. IRC is still around, but seems to be less popular, and doesn’t have the breadth of topics that Clojurians does.

Alex Miller is doing a great job addressing the community.

CLJS has come a long way!! to the point we can use it for production development! this community is great! Justin_Smith stands out as ridiculously helpful on IRC … if the world (this community) had more folks like him, it’ll elevate all of our games!

The thing that makes me wary of continuing to invest in Clojure professionally is how few paid opportunities there are for women and people of color. I don’t know what the exact problem is - they are left of networks, don’t have an opportunity to build their skills, or just straight-up sexism and racism - but it is definitely a problem. The current state of things means I will likely not be able to be on teams with women or people of color, and that is a big turn-off for me.

Clojure has the best community after PostgreSQL, in my opinion. I came for the language but I stayed for the community.

Be more inviting to new-comers.. so they stop learning Ruby.

Libraries

A “Rails for Clojure” was a common request so people can start delivering value quickly. Arachne was commonly raised as one possibility to provide this.

Clojure/ClojureScript it’s very cool, but not very practical nor efficient in many typical business case scenarios. I’m missing a full stack framework like Grails or Ruby on Rails or Django: there’s nothing like that for the Clojure world: it’s not about a project template but about the quick prototyping (with the customer by your side) these frameworks allow.

… There is also no good Clojure Web Development story: Pedestal looks interesting, but not actively developed and unsupported, Arachne is years away from being complete and useful, Luminus/Duct are initial scaffold project generators. It’s hard to delivering some value quickly - user have to dig through all underlying libs and how they work together.

… Clojure web development needs a “Rails” to overcome the chicken-and-egg problem of getting enough developers to be able to hire for it. …

I hope to write a lot more Clojure. No, I hope to do a lot more with less Clojure. And looking forward to using Arachne.

Other targets

Many people are looking to run Clojure on alternative platforms or reduce host dependence. LLVM was a common request. Some are looking to move away from the JVM, while others are looking to move into environments that Java and JavaScript aren’t suited for. This is an interesting request, though one of Clojure’s original goals (and still on the Clojure homepage) is to be a hosted language. I’m not sure whether it’s possible for these two desires to ever meet? Graal and Truffle are also interesting possibilities.

The biggest thing I’ve noticed for beginners on forums / blog comments is they are afraid of the JVM. …

I’ll never rewrite all my Scala into Clojure unless some sort of Cognitect Clojure OFFICIAL LLVM effort is made. Scala Native is [beating] Clojure.

Clojure’s hosted nature is its biggest downfall, in my opinion. As a systems programmer looking to lisp + immutability for saner concurrency, the cost of Clojure’s startup time and memory usage is very high. …

The one thing that I think would drive Clojure to a new level of adoption across the industry would be for someone to do an actual LLVM implementation of Clojure. Not a “close variant” of Clojure, but real Clojure. This would probably yield some performance boost beyond Java, but more importantly it would open up Clojure to environments that are simply not amenable to Java (or Javascript) for whatever reason. Some people have moved in this direction, but either have not completed the work, or have implemented their own “variants” of Clojure. What I wish someone would do is implement something with LLVM that is at least as close to Clojure as Clojurescript is.

I really think an opportunity is being missed by dismissing the c++ ecosystem, and I think that something that should be given somewhat serious consideration to by somebody would be to leverage LLVM with JIT and get a clojurescript running in that setting. There are significant legacy environments to be liberated there.

I would love for there to be some sort of compile-to-bundle with Clojure such that you have a single artifact that doesn’t require an external JVM installation. Like Go programs. An embedded JVM (perhaps).

Typing

As we’ve all learned this year, people are worried about types. Spec is the biggest movement in this direction, and the number of type related comments has dropped a lot since last year.

I know that Spec should help a lot with the error messages once we move to 1.9, although people coming from a non-Java background still have a lot of learning to do.

It is just sad that people who love Clojure (Colin Fleming, Steve Yegge, Adam Bard), have to use Kotlin sometimes instead (better startup times, static typing and Java interop like fast multidimensional arrays)

I hope Spec will find its way into tooling, especially Cursive.

Looking forward to learn and use spec

I’d most like to see improvements to the Clojure compiler. Error messages, performance, better Java interop and simple static type checking seem most important.

Compliments

As I mentioned at the start, the overwhelming majority of comments had positive things to say about Clojure. Alex Miller also deserves a special mention for his work with the community, and maintaining the Clojure infrastructure.

I sincerely thank Clojure(Script) community to make LISP revolution back to live stage and making us realize the importance of focusing on core things which matter

I have been programming since 1979. Clojure is a work of art.

The community has been unbelievably great and I hope it stays true to its roots! Keep up the good work and I can’t wait to see where Clojure/Script goes next!

The community is amazing, with a special shoutout to Alex Miller for being such a great part of it.

Happy Clojure user for over 4 years. Still loving every minute :-).

Finally, I found a comment that I think sums up the general feeling of the community (and my own):

No one and nothing is perfect, but I (for one) am very appreciative of the work that has been done and is on-going around Clojure/ClojureScript and the community in general. People tend to ‘harp’ on the things that aren’t just right (in their opinion) and forget about all the things that are amazing. I just want to say thanks, and keep up the superb work! :-)

I think the future of Clojure is bright.

Shockwaves rang out through the Clojuresphere today with the news that Datomic is changing their licensing to drop the per-process limits. This is big news if you were limited in the number of processes you wanted to run.

The major changes in a nutshell:

• There are now no limits on process count
• Datomic Starter is now limited to one year of updates, then you need to pay $5,000 for Datomic Pro, or stay on the current version when your license expires.
• Datomic Starter can now run the Memcached cache.
• All Datomic Pro licenses are now $5,000, and annual maintenance is also $5,000/year.

If I’ve got any of the following wrong, please get in touch via email and I’ll update my post.

The old pricing model

Datomic launched in March 2012 with a paid option and in July 2012 added Datomic Free. In November 2013, Cognitect launched Datomic Pro Starter Edition. The old pricing page is on archive.org here. This model was easy to understand, as it mapped well to existing database licensing patterns of ‘perpetually free with limitations’ or ‘paid without limitations on a per node basis’.

Datomic Free

• Free (as the name indicates)
• Limited to storing data in memory on the transactor, or on the transactor’s disk.
• Freely redistributable (i.e. in on-premise or open source software)
• Only able to run two peers (clients) and a single transactor. If the single transactor fails, then you won’t be able to write to the database until another one starts.
• Community support

Datomic Pro Starter Edition

• Free
• Support for all storage backends (SQL, DynamoDB, Cassandra, Riak, Couchbase, Infinispan)
• Limited to 2 peers and a single transactor
• Not redistributable
• Community support
• Limited to 1 year of maintenance and software updates, though you were able to renew your Datomic Pro Starter Edition license each year for free (more on this later).
• When your updates expire, you can continue using that version of Datomic, but you won’t be able to use any future versions.

Datomic Pro

• Paid
• Support for all storage backends
• Able to run a second transactor in HA standby to take over if the first one fails or for rolling updates.
• Ability to use Memcache to cache segments rather than all peers needing to talk directly to the storage backend.
• Support is included while maintenance is current.
• Pricing scaled linearly from $3,000 for five processes (this includes peers and transactors) up to 30 processes for $16,000. You could upgrade later to higher tiers for the price difference between the old license and the new license.
• Annual maintenance for support and updates was half of the upfront license price
• If an organisation had special needs (more processes, custom EULA, 24x7 support, redistributing Datomic e.t.c.) then you could talk to Cognitect to negotiate terms.

The new model

All versions of Datomic apart from Free now support unlimited peers, high availability, Memcached support, and all storages (more on storage later). This is a significant change, as you can now use Datomic in an almost unlimited fashion for free. There is also a new Client API which is suitable for smaller, short-lived processes, like microservices and serverless computing. The changes are smart, as it frees users up from having to architect their systems around their Datomic license limitations. The new pricing model rearranges the tiers. There is now Datomic Free (unchanged), Datomic Starter, Datomic Pro, and Datomic Enterprise.

Datomic Starter

• Similar intent to the previous Datomic Pro Starter
• Free
• Maintenance and updates limited to 1 year. However, based on discussion on Hacker News, it seems that you can no longer renew your Datomic Starter license. This means that you will need to pay for Datomic Pro to get updates after one year. You can still use whichever version was available when your updates expire. Discussion in the #datomic Slack channel on clojurians matches up with this too.
• Update: Alex Miller said that you can sign up for multiple Datomic Starter licenses for different systems you’re running.

Datomic Pro

• Similar intent to the previous Datomic Pro
• $5,000/year per system including maintenance and updates. For organisations using 16 processes or less, then maintenance will be more expensive (previously $1,500 - $4,500 depending on process count).
• 2 Day Business-Hours-Only Support. The former Datomic Pro didn’t have a published SLA for support, but I suspect that this is just formalising what was previously there.

Datomic Enterprise

• Enterprise integration support (professional services)
• Negotiated license terms
• 24x7 Support
• While this is a new tier in the pricing grid, these options were available as a “Contact Us” note on the former Datomic Pro.

Other changes

Datomic has a snazzy new documentation site. It also looks like as part of the licensing changes, Riak, Couchbase, and Infinispan are now considered legacy storage and are only available to be used under an enterprise license. Standard editions of Datomic only support SQL, DynamodDB, and Cassandra. This change hasn’t been mentioned on the Datomic mailing list or release notes, but probably will be soon.

Datomic has deprecated the REST API in favour of the new Client API.

There is also a new customer feedback portal where you can suggest features. You can access this after logging in to my.datomic.com.

Summary

If you are a Datomic Pro user then your maintenance is probably going to be higher, although in absolute terms it’s still not a lot compared to developer salaries. If you were on Datomic Pro Starter and want to stay current, then you are now looking at moving Datomic Free, or paying $5,000/year for Datomic Pro. If you were using Riak, Couchbase, or Infinispan then it seems like you’ll need to get Datomic Enterprise.

Datomic from the beginning has always felt like a database that understood the Cloud, and the zeitgeist of computing. It supported AWS DynamoDB and CloudFormation from early on, and their architecture has always felt well suited to cloud systems. The license changes to accommodate the trend towards microservices and serverless computing are a continuation of that.

Before we get started, I want to be clear. I don’t support tobacco, cluster bomb manufacturers, nuclear weapons manufacturers, and the other socially harmful businesses mentioned in the recent brouhaha about Kiwisaver investments into the aforementioned companies. However, I think the reporting that NZ Herald did on this was misleading, in search of a good story at the expense of accuracy.

To recap: the Herald has recently been reporting on Kiwisaver, the NZ government retirement savings scheme (like an IRA). Their big headline was that $150mm has been invested by the major Kiwisaver providers into companies that produce cluster bombs, landmines, and other socially harmful products, and that they may be breaking a law banning investments into cluster bomb manufacturers. When you look into the details, their bold claims don’t look so strong.

Here are a few points that should have been included in the reporting:

• The biggest problem is that the Herald doesn’t distinguish between active management (where fund managers or algorithms chose particular businesses to invest in), and passive management that tracks an index (say agriculture, energy, or the USA stock market). If some of these funds were directly invested in these companies, and banks really were breaking the law, that would be a real story. It’s not clear to me from the reporting whether there was any direct investment into the companies, or investment into clusterbomb index funds, or whether the funds were invested in broad indexes. You can make an argument for both being bad, but active investment in these companies is very different from passively investing in an index fund of the total stock market. The Herald implies that these companies deliberately or knowingly invested in the companies, but I couldn’t see this from the data they provided.
• The total amount invested in Kiwisaver is $32.5 billion, and the amount invested in socially harmful businesses is $150 million. This works out to around 0.46% of the total funds invested. This is not nothing, but it is a tiny fraction of the total assets invested.
• Kiwisaver funds that aren’t investing in these companies are likely doing so through active management of stocks. In general, this will result in higher fees, and depending on who you ask, lower performance (especially once fees are taken into account).
• The charts that NZ Herald produced all give absolute numbers for how much is invested in socially harmful companies, without giving the percentage invested. Westpac, ANZ, and ASB all feature high on the list of investing into these harmful companies, but there is no context given for what percentage of the investments each have made. Westpac has 0.76% in socially harmful businesses. ANZ has 1.27%, and ASB has 0.13% ¹. Without more details on total amount invested in stocks (active and passive) by each fund, it’s hard to tell where and why the differences here come from.
• The amount invested in the socially harmful parts of Northrop Grumman² and General Dynamics ³ are very small, compared to their overall businesses.

Data journalism can be used to illuminate complex topics and explain them for a wide audience. It seems in this reporting that the story came first, and the numbers were presented in a misleading way to back it up. There is a nuanced discussion that could have been had about the ethics of index funds, and socially responsible investing, but that wasn’t what we got here.

N.B.: I may have read the article wrong, and all of the figures provided were active investments (it’s not clear at all to me which is being included). If that’s the case my conclusion would be quite different.

Update: More discussion index/active investments may be coming later this week.

Every week in The REPL I have a section called: “People are worried about Types”. This is based on a section that Matt Levine (my favourite columnist) includes in his daily column “People are worried about bond market liquidity”, and the related “People are worried about unicorns”, and “People are worried about stock buybacks”.

The title is a joke (as is his), I don’t really think there are people worried about types, but it is interesting to see a small but steady flow of type or type related (schema, spec, e.t.c.) articles relating to Clojure each week.