Daniel Compton

The personal blog of Daniel Compton - Projects

How to upgrade Terraform providers and modules

Since Terraform v0.10, Terraform providers are distributed separately from the Terraform binary. This lets them update at different paces, and allows a wider group of people to collaborate on the providers. This is mostly good, but it does introduce a new step for upgrading providers. It is slightly counterintuitive, but to upgrade your providers, run

terraform init -upgrade

To upgrade your modules, run

terraform get -update

How to upgrade Terraform provider plugins and modules

Since Terraform v0.10, Terraform providers are distributed separately from the Terraform binary. This lets them update at different paces, and allows a wider group of people to collaborate on the providers. This is mostly good, but it does introduce a new step for upgrading providers. It is slightly counterintuitive, but to upgrade your providers, run

terraform init -upgrade

To upgrade your modules, run

terraform get -update

Fixing Ansible error: “SSH Error: data could not be sent to remote host "". Make sure this host can be reached over ssh”

I was building VM images for Google Cloud with Packer, and provisioning them with Ansible. Everything had been working in the morning, but in the afternoon one computer wasn’t working after I had upgraded Ansible with Homebrew. I was having a really tough time figuring out why Ansible and Packer were running fine on one computer, and not on the other. I was getting the following error:

googlecompute: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"\". Make sure this host can be reached over ssh", "unreachable": true}

Investigating the error, I found it could indicate any number of problems, as Ansible is masking the underlying SSH error.

After checking the versions of Python, Ansible, and Packer on the two computers, I found one difference. On the computer that wasn’t working, when running ansible --version it had a config file listed:

$ ansible --version
  config file = /Users/<user>/.ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Dec 18 2016, 07:03:39) [GCC     4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]

Opening that file:

$ cat ~/.ansible.cfg
pipelining = True

I moved that file to another location so it wouldn’t be picked up by Ansible, and after rerunning Packer a second time, got this error:

googlecompute: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '[]:62684' (RSA) to the list of known hosts.\r\nReceived disconnect from port 62684:2: too many authentication failures\r\nAuthentication failed.\r\n", "unreachable": true}

I then used the extra_arguments option for the Ansible provisioner to pass [ "-vvvv" ] to Ansible. I ran this on both computers and diffed the output. I saw that the dynamically generated key was being successfully provided on the working computer (after one other local key). On the failing computer I had many SSH keys that were being tried before I could get to the dynamic key, and I was getting locked out.

SSH servers only allow you to attempt to authenticate a certain number of times (six by default). All of your loaded keys will be tried before the dynamically generated key provided to Ansible. If you have too many SSH keys loaded in your ssh-agent, the Ansible provisioner may fail authentication.

Running ssh-add -D unloaded all of the keys from my ssh-agent, and meant that the dynamic key Packer was generating was provided first.

I hope this is helpful to someone else, and saves you from hours of debugging!


I was very confused by seeing that my computer was trying to connect to, instead of the Google Cloud Platform VM. My best guess is that Packer/Google Cloud SDK proxies the connection from my computer to the VM.

Detecting the users’s time zone using pure JavaScript

While working on Deps I wanted to detect the user’s time zone at signup, to localise the times that I presented to them. I hunted around and found a variety of libraries that offer time zone detection, like moment.js, and jstz. However I wasn’t keen on pulling in a dependency on one of these libraries if I could help it, so I kept looking. I also considered trying to detect the user’s time zone from their IP address using something like the Maxmind GeoIP database, and that probably would have been the method I settled on.

However in my research, just before I started with IP based detection, I found that there is a newish Internationalization API, which will give you the user’s system time zone.

=> "Pacific/Auckland"

All current browsers support the Internationalization API including IE11. However there is a slight wrinkle to keep in mind when using it: on some older browser versions that do support Internationalization (IE <=11, and Firefox <=51), the timeZone property is set to undefined rather than the user’s actual time zone. As best as I can tell, at the time of writing (July 2017) all current browsers except for IE11 will return the user’s actual timezone. If you do need to support older browser environments then you could look at moment.js. It uses the newer Internationalization API when it is available, and falls back to other heuristics if it isn’t. For my purposes, my browser statistics show almost no IE usage, so I chose to just use the newer method, while also allowing the user to manually update their time zone if they needed.

Bryan Cantrill on Integrity

Bryan Cantrill on integrity. Edited slightly for clarity and profanity:

We are on a collision course with the Amazon Principles […] What is going on? I’m losing my mind. The wheels are off, everyone has to get out of the car and this is why. All of these [expletive] leadership principles, from all these organisations, where is integrity? Damn it where’s integrity? Amazon has 14 leadership principles and integrity is not on it. [That’s] inexcusable. I’m sorry, if you’ve got one principle in your organisation, its integrity? Right? […] No we’re living in a world that has lost it’s [expletive] mind, I don’t understand it. Why the appetite for territory? Do you not know where your next meal is coming from? Do you not have a roof over your head? I mean I’m sorry, I just don’t get it. We have got an incredible luxury. There’s never been a labour market like this one. Where have we so screwed up with a generation, if not a society where we’ve got people who are so extrinsically motivated? What the [expletive] is wrong with us? I mean its like how is integrity not the only thing you have?

[Andrew Clay Shafer]: Are we doing the full critique of capitalism today?

No it’s not. Bullshit, it’s not capitalism. The finest capitalist is Scott McNealy, a capitalist so pure that when he was being devoured by Oracle he believed that it was his duty to capitalism to die, and I admire that. There is no purer a capitalist than Scott McNealy. And read McNealy’s final email to Sun employees, and it’s almost prescient, McNealy says: “You know what, in 30 years I never had to hide the newspaper from my children.” An achievement that Uber violated on like month two? Where are Travis’ parents? Are you not humiliated? How did you raise him to be so divorced from what really, actually, truly matters? What the living [expletive] is wrong with us? Maybe it’s me?

I continue to be impressed by Joyent, and Bryan Cantrill’s engineering principles. It sounds like a great place to do your best work.

Recorded at GOTO 2017 for an Arrested Devops episode: Old Geeks Yell At Cloud

The case for and against Clojure predicates returning nil

Last week, there was a mailing list post on the clojure-dev mailing list noting that several of the new predicates in Clojure 1.9 alphas return true or nil. The three new predicates are: qualified-keyword?, qualified-ident?, and qualified-symbol?. Previously, all predicate functions in Clojure core, i.e. functions that end in a ? have returned true/false. In this post I want to examine the pros and cons of this new behaviour.

If I’ve missed anything here, please get in contact and I’ll update the post.

Pros to some predicates returning nil

The first two points aren’t strong arguments for returning nil, rather they argue that it doesn’t matter whether the functions return nil or false.

Cons to some predicates returning nil:


Personally, I don’t see a strong reason to keep the current behaviour of the new qualified predicate functions, but you might see things differently. Whatever the outcome, I’m glad that in Clojure we don’t have to name all of our boolean functions with an “Eh” suffix.


Alex Miller suggested to file a ticket about this behaviour, so I opened CLJ-2141.

Uber-Alphabet Lawsuit Updates

Since my post a few days ago on the Uber-Alphabet lawsuit, there have been a few new developments.

Levandowski consulting for Uber

In that post I wrote:

… no funding appears to have been taken from Uber, or any other company (If there turn out to be payments from Uber to Otto prior to the acquisition then that would be Very Bad for Uber).

Not long after, Johana Bhuiyan at Recode posted a story which suggests that Uber CEO Travis Kalanick hired Otto CEO Anthony Levandowski as a consultant for Uber’s self-driving car around February-March 2016. Even after reading the article several times and sleeping on it, I still can’t quite believe it.

Now, sources are adding more to that claim. Kalanick hired Levandowski as a consultant to Uber’s fledgling self-driving effort as early as February 2016, six months before Levandowski publicly got his company off the ground, according to three sources.

If this was a legitimate business arrangement, the timing makes no sense at all. Levandowski had just left Waymo weeks earlier. While it might have been difficult for Alphabet to bring a lawsuit on these grounds, the optics for Levandowski advising Uber are terrible. Additionally, Uber and Otto are competitors in the self-driving vehicle space, and Uber is famously aggressive. That a CEO of one company would be advising one of their direct competitors within months of forming their own self-funded startup is truly strange.

If this wasn’t a legitimate business arrangement, then it also doesn’t make much sense purely because it looks so suspicious, especially in light of Alphabet’s trade secrets lawsuit. It brings the start of Uber’s direct involvement with Otto to within weeks of Otto being formed. If the plan all along was for Otto to join Uber, then it does make some kind of twisted sense for him to be advising the Uber self-driving car team.

A little birdie tells me that we haven’t seen the last on this particular thread of the story.

Naked Capitalism on Uber

Hubert Horan has just published another article in his series on Uber and whether it can ever increase overall economic welfare.

The answer is that Uber is attempting to implement an innovative and disruptive strategy for achieving market control — a strategy radical different from every prior tech oriented startup — but those disruptive innovations have nothing to do with technology or the efficient production of urban car services.

Otto purchased another Lidar company

A reader sent me a link to this article by Reuters discussing an acquisition of Tyto LIDAR. The title is “Uber’s self-driving unit quietly bought firm with tech at heart of Alphabet lawsuit”. The title isn’t accurate, as it was Otto that acquired Tyto (for an undisclosed sum) in May 2016. Uber then acquired Otto/Tyto in August 2016.

Levandowski: “We did not steal any Google IP”

Forbes published an article on 28 February, 2017 which contains part of an interview they did with Levandowski in October 2016.

Q: The question is, Google has been working on this for however many years. You guys (Otto) are a year old or something like that.

A: We started in January. How did we catch up?

We did not steal any Google IP. Just want to make sure, super clear on that. We built everything from scratch and we have all of the logs to make that—just to be super clear.

But this is my fourth autonomy stack. I did the DARPA Grand Challenge.

I wonder if “we have all of the logs” is also going to be Alphabet counsel’s opening argument?

The Uber Bombshell About to Drop

If you’ve been following tech news over the last few weeks, you have probably seen several stories about Uber, all negative (bar this one about flying cars). I suspect that what is coming next will prove to be a far bigger story than all of the other incidents so far.

N.B. all of this article is sourced from filings and allegations that Alphabet has made, as well as reading between the lines. Uber will probably contest these claims in court.

Update: I’ve published some updates on the lawsuit. You can keep track of all the posts related to the Alphabet lawsuit here.

In the last few weeks Alphabet filed a lawsuit against Uber. Alphabet and Waymo (Alphabet’s self-driving car company) allege that Anthony Levandowski, an ex-Waymo manager, stole confidential and proprietary information from Waymo, then used it in his own self-driving truck startup, Otto. Uber acquired Otto in August 2016, so the suit was filed against Uber, not Otto.

This alone is a fairly explosive claim, but the subtext of Alphabet’s filing is an even bigger bombshell. Reading between the lines, (in my opinion) Alphabet is implying that Mr Levandowski arranged with Uber to:

  1. Steal LiDAR and other self-driving component designs from Waymo
  2. Start Otto as a plausible corporate vehicle for developing the self-driving technology
  3. Acquire Otto for $680 million

Below, I’ll present the timeline of events, my interpretation, and some speculation on a possible (bad) outcome for Uber. The timeline references section numbers from Waymo’s amended filing, so you can read the full context yourself. You can also read the original filing.

Timeline of events

The main timeline of important events is as follows:


From Waymo’s filings, it seems that they have Levandowski dead to rights on stealing their LiDAR designs. That alone should be enough to bring Uber’s self-driving car program to a halt and cause some big problems for Levandowski. California’s Trade Secrets law is weaker than other states, but if successful, Waymo will be able to seek an injunction, damages, and attorney’s fees. Because all law is securities law, the SEC may also be able to bring a case against Uber (similarly to their case against Theranos).

I’m guessing, but I think the reason that Alphabet hasn’t directly accused Uber of conspiring with Levandowski is that they don’t have enough evidence. When they get to discovery, they will be looking for it. You would think that no-one would be dumb enough to send emails like that, but you would be wrong.

Several things suggest Otto’s intent to be acquired by Uber:

There is no smoking gun email (yet), but there is a strong implication that Uber and Otto planned this from the start.

(Possible) Consequences

If this is true and can be proved in court, then it would be a massive blow to Uber. The worst case for Uber would be:

This is admittedly the very worst case scenario for Uber, and there are lots of places along this downward trajectory that they could pull up from. If it’s proven that Uber intended to acquire Otto while Levandowski was at Google (this might be established more concretely in discovery), then it’s hard to see how Uber’s CEO Travis Kalanick could keep his job. Uber has had a bad month, and it doesn’t look like things are going to be getting better any time soon.


This post is made up of a mix of filings from Waymo, reading between the lines, and some speculation. Keep in mind these are all allegations and haven’t been proven in court. Please let me know if I’ve made any mistakes, and I’ll correct them.

I’d encourage you to go over the filings yourself, as they are very readable, and give a lot more context to the story than I was able to here.


I originally posted that Uber was only charging 41% of the cost of the trips, based on this article which I misread. The 41% number is the total cost of the trip including driver compensation as well as corporate expenses.

Alyssa Vance sent me the calculations you would make to work out the farebox ratio:

Farebox ratio = fares / total cost of operations Total cost of operations = fares - profits Farebox ratio = fares / (fares - profits)

For 1H2015: Farebox ratio = 3,661 / (3,661 - -987) = 3661 / 4648 = 78.8%.

State of Clojure Survey 2016 Analysis

The State of Clojure 2016 survey results have just been published. Last year I wrote an summary of the free text comment portion of the State of Clojure 2015 Survey. I enjoyed writing it, and it was my second most popular post of the year, so I’m repeating the process again.

In his summary, Justin said that 62% of the responses were positive. That number sounded low to me (but was presumably calculated by SurveyMonkey). I would have estimated closer to 80% after reading through them all. A little under one quarter of the 2420 survey respondents left a comment.

I’m reusing most of the same categories from last year so interested readers can compare with last years post. Some comments have been lightly edited for spelling, clarity, brevity, and niceness.

Error messages

Error messages were the biggest pain point for people last year. This year there were still many complaints about error messages and stack traces, but a lot of people are waiting to see what spec brings. There was a lot of discussion about spec error messages several months ago, but it seems to have gone quiet. I’m really hopeful that spec will improve error messages.

This year Figwheel has started providing very sophisticated reporting of compile errors, and configuration problems. It has made a massive difference to my ClojureScript workflow. Thanks Bruce!

Please Please Please can we get better error messages. …

ClojureScript has come a long way, but I still encounter errors that are totally perplexing. Better debugging would help enormously.

Error messages are very frustrating and it is hard to find where the error occurred and why. Errors coming from underlying Java code are only somehow relevant to the Clojure that produced them. Programming with dynamically typed language rocks when inferred types align with your idea, but when it’s not, things get fuzzy.

Please, please, please, please, improve error messages. This would single handedly improve: new user onboarding, community feedback (let’s admit it - it is a small community), rapid iterations, usage of the REPL, usage of spec and testing, and other areas. …

I consistently see error messages being the biggest barrier for learners of Clojure, and even more so for ClojureScript

I think a concerted focus on improving Clojure’s error reporting & handling would benefit both newcomers to the language and experienced developers alike and would result in considerably less wasted time & effort.


Last year clojure.org had just been revamped and open sourced. I wrote at the time:

I have high hopes that it will become a central source for high quality, official documentation and ease the pain for newcomers.

clojure.org has had some contributions from the community, but it still doesn’t have the breadth of material needed to be a central source. I would love to see something like Python’s web documentation for Clojure. Renzo Borgatti is writing a mammoth book “The Clojure Standard Library” which will cover the entire Clojure standard library.

… Please improve the documentation. Recently I heard someone describe Clojure documentation as “comically terse”.

A simple way to get someone up and running that is a standard for the community would be great. There is nothing more frustrating than telling new people to “Google” for a solution and pick one you like… They never get further than reading the docs.

Clojure appeared to be a nice language when I started with it and I do not regret this decision. The flip side is that I had hoped that the poor documentation would have gone away by now — 1.4 was the first version I used and it does not seem to have improved. That’s a shame. Documentation helps new users and really, hinders nobody.

Clojure docs are too terse. They assume that the reader already fully understands the concepts being described, and merely needs a refresher. …

Clojure’s community is surprisingly small given it’s output. This leaves documentation and tutorials sparse because they require time and effort that is in short supply. …

I think both clojure.org and clojurescript.org could use a better “Get Started” experience. It’s our best opportunity at a first impression but we rely too heavily on third party docs. For example, on cljs.org the reference section starts with compiler options. Doesn’t really capture the imagination.


Again, Figwheel has been leading the way in improving tooling on the ClojureScript side. Tooling is still a pain point, but seems to be less so than in previous years. For Clojure editors, Cider, Cursive, and vim-fireplace are all used pretty much the same amount as last year, but Atom has entered fourth place, presumably coupled with proto-repl.

One thing we’re used to from C#/Java that would be awesome to have is a better unit testing framework with IDE integration. …

The build toolchain for cljs has been a pain. All of my learning has come from cloning public repos and seeing how they do it.

A huge thanks to the community for invaluable tools like Leiningen, Parinfer and the various IDE plugins like ProtoREPL or Cursive. And of course huge thanks to the core team for excellent stewardship of Clojure itself.

While tooling for Clojure is improving there needs to be a lot more resources devoted to it. I am currently in a Java shop and the only realistic option is Cursive.

Startup time

Startup time was another persistent complaints this year. Unfortunately there wasn’t always enough information about whether startup time was a problem in development or production. There are several good options for avoiding constantly restarting Clojure REPLs in development (and in production), but perhaps for newcomers these are too complicated to setup, or they don’t know about them?

Love Clojure … hate its startup time!

Love the language and will keep at it. Primary frustration is with the start up time and tooling. In particular, I’m finding it difficult to quickly test the backend and frontend of my application with Boot.

… We love functional programming, but hate Clojure’s long startup times. We do what we can to minimize this (using test-refresh, REPLs, etc.), but it is still very painful, and we find ourselves twiddling our thumbs waiting for a restart several times a day. Similarly, our Clojurescript compiles take ~60 seconds (:simple optimizations) or ~30 seconds (:none optimizations). This slows us down substantially. Coming from Python, where compile and startup times are sub-second, this is the biggest complaint from our team.

Clojure would be perfect with faster startup time and a little less memory usage. Seriously I don’t even mind the java stack traces.

REPL startup time is all I care about. I use emacs+cider, and it takes an age. Otherwise, I am completely happy with Clojure :)


Justin specifically mentioned marketing and adoption as a common theme amongst the feedback:

One relatively stronger theme this year was the need for better marketing for the purposes of expanding or introducing Clojure within organizations, which is a great area for contribution from the entire community.

In my opinion, one of the areas where Clojure could improve on marketing is on the main Clojure and ClojureScript websites. I think Scala in particular has done a really good job of this. Elm and Rust also do really well in presenting the main reasons why you would want to use their languages. The community isn’t able to help with this, as contributions to style and templates are not currently wanted.

Last year I mentioned Cognitect’s case studies on Clojure adoption, this year Juxt has done some great work with their case studies of companies using Clojure in Europe.

Anecdotally, I’ve seen fewer people coming along to the Auckland Clojure Meetup than in previous years, although we’re working on a fairly small sample size. I’m not sure what (if anything) to make of that.

I would do 100% of my development in Clojure if the enterprise-y company I worked for allowed it. “Reason for not using Clojure as much as you would like”? Management not on board.

I’d love to see more “serious” support for ClojureScript: when I present ClojureScript to colleagues, it looks like a thing supported by several enthusiasts rather than a platform supported by a company.

Clojure is very solid. As a frontend developer, a stable build system, DCE and syntax stability are very valuable for me. I failed to convince my CTO on Clojure and instead we stuck to ES6/ES7. The main concern of management was hiring and training people. Although the learning curve for Clojure is easy, people still have a perception that LISP syntax is esoteric and difficult to pick up for someone completely new to the ecosystem. This myth has to be busted on a larger scale.

Language is great, community is great, the “marketing” is not that great. You guys have a great language and you are struggling to sell it.

[I] would love to see better marketing from Cognitect and other Clojure centric shops. Selling Clojure to a Java shop is not easy. It can and should be easier. (and simple, but I’ll take “easier” here first)

As an average developer fluent in Java, I would love to use Clojure more, but the biggest hurdle for me is the lack of peers interested in Clojure. At the local meetup people report the same situation at their organisations. Pity, because Clojure is super cool! Thank you.

Love Clojure, really. Can’t convince my peers that it’s worth investing the time to learn it though.

The Clojure community needs a “Rails killer,” and only Arachne holds serious promise for that.

Clojure/ClojureScript ecosystem could benefit from stronger stewardship from Cognitect to propel it into the mainstream with a focus on a killer app or a industry domain similar to the way Light Bend has been focusing on enterprise and reactive system to promote the Scala ecosystem. …

I’m in Shanghai and we barely have full-time Clojure developers in China. We formed online groups, but I guess we need help to push it to the next level.

And the age old developer question:

How can we get management to understand what they don’t understand?

Language Development

Like last year, there are still concerns about the contribution process, and the opaque decision making process for Clojure’s development.

Engagement of community is still sub-optimal, there are talented individuals who could be engaged more and productively so without opening it up to the peanut gallery.

I wish that the core Clojure development process was more flexible and friendly. I (and all the Clojure developers I know) have pretty much given up on trying to make suggestions or tickets about Clojure because I usually wind up banging my head against the wall and getting frustrated. The core team are under no obligations to explain themselves or their motivations, but they would save a lot of frustration from new community members wishing to contribute if there was a prominent document outlining the philosophy, and what kinds of changes and suggestions are and aren’t welcome.

A more open approach to development and contributing to Clojure would be appreciated. I’d love to contribute to Clojure but the mess of a lot of the code in Clojure and the JIRA-based contribution model is a barrier. I know that’s what Rich likes and see the rest of Cognitect are happy contributing in this way but I’d love to see something more open and approachable.

The slow rate at which JIRA issues are addressed remains a frustration, and provides a disincentive for contributing. Features that Cognitect cares about get rushed into the language half-baked, while well-thought out patches from years ago languish.

A little concerned over how ‘arbitrary’ some design decisions seem to be and how inflexible the core team is. Running instrumented post-condition specs is such an obvious idea for example but it has been deemed not necessary “from on high”.

The issue tracker is well tended during the earlier workflow stages. There seems to be a bottleneck later on. I hope 2017 will bring the project’s founders to modes of community contribution that are more efficient of the art and tradecraft of the many.


I would say that Clojure’s community is one of it’s greatest assets. There were mostly positive comments for the community, though still some concerns. The Clojurians Slack has been a great resource for a lot of newcomers, and spawned many channels for libraries and topics. IRC is still around, but seems to be less popular, and doesn’t have the breadth of topics that Clojurians does.

Alex Miller is doing a great job addressing the community.

CLJS has come a long way!! to the point we can use it for production development! this community is great! Justin_Smith stands out as ridiculously helpful on IRC … if the world (this community) had more folks like him, it’ll elevate all of our games!

The thing that makes me wary of continuing to invest in Clojure professionally is how few paid opportunities there are for women and people of color. I don’t know what the exact problem is - they are left of networks, don’t have an opportunity to build their skills, or just straight-up sexism and racism - but it is definitely a problem. The current state of things means I will likely not be able to be on teams with women or people of color, and that is a big turn-off for me.

Clojure has the best community after PostgreSQL, in my opinion. I came for the language but I stayed for the community.

Be more inviting to new-comers.. so they stop learning Ruby.


A “Rails for Clojure” was a common request so people can start delivering value quickly. Arachne was commonly raised as one possibility to provide this.

Clojure/ClojureScript it’s very cool, but not very practical nor efficient in many typical business case scenarios. I’m missing a full stack framework like Grails or Ruby on Rails or Django: there’s nothing like that for the Clojure world: it’s not about a project template but about the quick prototyping (with the customer by your side) these frameworks allow.

… There is also no good Clojure Web Development story: Pedestal looks interesting, but not actively developed and unsupported, Arachne is years away from being complete and useful, Luminus/Duct are initial scaffold project generators. It’s hard to delivering some value quickly - user have to dig through all underlying libs and how they work together.

… Clojure web development needs a “Rails” to overcome the chicken-and-egg problem of getting enough developers to be able to hire for it. …

I hope to write a lot more Clojure. No, I hope to do a lot more with less Clojure. And looking forward to using Arachne.

Other targets

Many people are looking to run Clojure on alternative platforms or reduce host dependence. LLVM was a common request. Some are looking to move away from the JVM, while others are looking to move into environments that Java and JavaScript aren’t suited for. This is an interesting request, though one of Clojure’s original goals (and still on the Clojure homepage) is to be a hosted language. I’m not sure whether it’s possible for these two desires to ever meet? Graal and Truffle are also interesting possibilities.

The biggest thing I’ve noticed for beginners on forums / blog comments is they are afraid of the JVM. …

I’ll never rewrite all my Scala into Clojure unless some sort of Cognitect Clojure OFFICIAL LLVM effort is made. Scala Native is [beating] Clojure.

Clojure’s hosted nature is its biggest downfall, in my opinion. As a systems programmer looking to lisp + immutability for saner concurrency, the cost of Clojure’s startup time and memory usage is very high. …

The one thing that I think would drive Clojure to a new level of adoption across the industry would be for someone to do an actual LLVM implementation of Clojure. Not a “close variant” of Clojure, but real Clojure. This would probably yield some performance boost beyond Java, but more importantly it would open up Clojure to environments that are simply not amenable to Java (or Javascript) for whatever reason. Some people have moved in this direction, but either have not completed the work, or have implemented their own “variants” of Clojure. What I wish someone would do is implement something with LLVM that is at least as close to Clojure as Clojurescript is.

I really think an opportunity is being missed by dismissing the c++ ecosystem, and I think that something that should be given somewhat serious consideration to by somebody would be to leverage LLVM with JIT and get a clojurescript running in that setting. There are significant legacy environments to be liberated there.

I would love for there to be some sort of compile-to-bundle with Clojure such that you have a single artifact that doesn’t require an external JVM installation. Like Go programs. An embedded JVM (perhaps).


As we’ve all learned this year, people are worried about types. Spec is the biggest movement in this direction, and the number of type related comments has dropped a lot since last year.

I know that Spec should help a lot with the error messages once we move to 1.9, although people coming from a non-Java background still have a lot of learning to do.

It is just sad that people who love Clojure (Colin Fleming, Steve Yegge, Adam Bard), have to use Kotlin sometimes instead (better startup times, static typing and Java interop like fast multidimensional arrays)

I hope Spec will find its way into tooling, especially Cursive.

Looking forward to learn and use spec

I’d most like to see improvements to the Clojure compiler. Error messages, performance, better Java interop and simple static type checking seem most important.


As I mentioned at the start, the overwhelming majority of comments had positive things to say about Clojure. Alex Miller also deserves a special mention for his work with the community, and maintaining the Clojure infrastructure.

I sincerely thank Clojure(Script) community to make LISP revolution back to live stage and making us realize the importance of focusing on core things which matter

I have been programming since 1979. Clojure is a work of art.

The community has been unbelievably great and I hope it stays true to its roots! Keep up the good work and I can’t wait to see where Clojure/Script goes next!

The community is amazing, with a special shoutout to Alex Miller for being such a great part of it.

Happy Clojure user for over 4 years. Still loving every minute :-).

Finally, I found a comment that I think sums up the general feeling of the community (and my own):

No one and nothing is perfect, but I (for one) am very appreciative of the work that has been done and is on-going around Clojure/ClojureScript and the community in general. People tend to ‘harp’ on the things that aren’t just right (in their opinion) and forget about all the things that are amazing. I just want to say thanks, and keep up the superb work! :-)

I think the future of Clojure is bright.

A guide to the Datomic licensing changes

Shockwaves rang out through the Clojuresphere today with the news that Datomic is changing their licensing to drop the per-process limits. This is big news if you were limited in the number of processes you wanted to run.

The major changes in a nutshell:

If I’ve got any of the following wrong, please get in touch via email and I’ll update my post.

The old pricing model

Datomic launched in March 2012 with a paid option and in July 2012 added Datomic Free. In November 2013, Cognitect launched Datomic Pro Starter Edition. The old pricing page is on archive.org here. This model was easy to understand, as it mapped well to existing database licensing patterns of ‘perpetually free with limitations’ or ‘paid without limitations on a per node basis’.

Datomic Free

Datomic Pro Starter Edition

Datomic Pro

The new model

All versions of Datomic apart from Free now support unlimited peers, high availability, Memcached support, and all storages (more on storage later). This is a significant change, as you can now use Datomic in an almost unlimited fashion for free. There is also a new Client API which is suitable for smaller, short-lived processes, like microservices and serverless computing. The changes are smart, as it frees users up from having to architect their systems around their Datomic license limitations. The new pricing model rearranges the tiers. There is now Datomic Free (unchanged), Datomic Starter, Datomic Pro, and Datomic Enterprise.

Datomic Starter

Datomic Pro

Datomic Enterprise

Other changes

Datomic has a snazzy new documentation site. It also looks like as part of the licensing changes, Riak, Couchbase, and Infinispan are now considered legacy storage and are only available to be used under an enterprise license. Standard editions of Datomic only support SQL, DynamodDB, and Cassandra. This change hasn’t been mentioned on the Datomic mailing list or release notes, but probably will be soon.

Datomic has also deprecated the REST API in favour of the new Client API.

There is a new customer feedback portal where you can suggest features.


If you are a Datomic Pro user then your maintenance is probably going to be higher, although in absolute terms it’s still not a lot compared to developer salaries. If you were on Datomic Pro Starter and want to stay current, then you are now looking at moving Datomic Free, or paying $5,000/year for Datomic Pro. If you were using Riak, Couchbase, or Infinispan then it seems like you’ll need to get Datomic Enterprise.

Datomic from the beginning has always felt like a database that understood the Cloud, and the zeitgeist of computing. It supported AWS DynamoDB and CloudFormation from early on, and their architecture has always felt well suited to cloud systems. The license changes to accommodate the trend towards microservices and serverless computing are a continuation of that.

On cluster bombs and data journalism

Before we get started, I want to be clear. I don’t support tobacco, cluster bomb manufacturers, nuclear weapons manufacturers, and the other socially harmful businesses mentioned in the recent brouhaha about Kiwisaver investments into the aforementioned companies. However, I think the reporting that NZ Herald did on this was misleading, in search of a good story at the expense of accuracy.

To recap: the Herald has recently been reporting on Kiwisaver, the NZ government retirement savings scheme (like an IRA). Their big headline was that $150mm has been invested by the major Kiwisaver providers into companies that produce cluster bombs, landmines, and other socially harmful products, and that they may be breaking a law banning investments into cluster bomb manufacturers. When you look into the details, their bold claims don’t look so strong.

Here are a few points that should have been included in the reporting:

Data journalism can be used to illuminate complex topics and explain them for a wide audience. It seems in this reporting that the story came first, and the numbers were presented in a misleading way to back it up. There is a nuanced discussion that could have been had about the ethics of index funds, and socially responsible investing, but that wasn’t what we got here.

N.B.: I may have read the article wrong, and all of the figures provided were active investments (it’s not clear at all to me which is being included). If that’s the case my conclusion would be quite different.

Update: More discussion index/active investments may be coming later this week.

  1. Westpac has $3.2 billion invested in their funds, ANZ has $5.5 billion, ASB has $5 billion

  2. Northrop Grumman is blacklisted by the NZ Superannuation Fund for selling mines. They had sales last year of $23.5 billion. $15 billion in products and $10.5 billion in services. This is split amongst Aerospace systems, Electronic systems, Information systems, and Technical services. Northrop Grumman doesn’t break out a landmine line item (and only mention mines once in their annual report, to say they have nothing to disclose about mine safety), but it looks like it is part of the Electronic systems segment, which did $5.5 billion in product sales and $1.3 billion in services (23% of total sales). Electronic systems also includes radar, space intelligence, navigation, land & self protection systems (probably where mines go, but also includes missiles, air defence, e.t.c.). 

  3. General Dynamics is blacklisted by the NZ Superannuation fund for selling cluster bombs. They had $31.4 billion in sales in 2015. They also don’t break out a clusterbomb line item (I’m sensing a pattern here), but it probably fits into the Combat Systems group which had $5.6 billion in sales (18%), and which also includes wheeled combat and tactical vehicles, tanks, weapons systems, and maintenance. 

Why are people worried about types?

Every week in The REPL I have a section called: “People are worried about Types”. This is based on a section that Matt Levine (my favourite columnist) includes in his daily column “People are worried about bond market liquidity”, and the related “People are worried about unicorns”, and “People are worried about stock buybacks”.

The title is a joke (as is his), I don’t really think there are people worried about types, but it is interesting to see a small but steady flow of type or type related (schema, spec, e.t.c.) articles relating to Clojure each week.

Announcing The REPL - A weekly Clojure newsletter

I’ve started a weekly newsletter about Clojure and ClojureScript. Each week will have a curated selection of links (both recent, and older) and a sentence or two about why I think they’re worth reading. You can sign up at therepl.net.

Requiring records in Clojure and ClojureScript

One of the datatypes Clojure gives you is the Record. It is used in both Clojure and ClojureScript, but due to implementation differences, the syntax to use records from one namespace in another is different.


Clojure generates a Java class for every defrecord that you create. You need to :import them, just like you would a standard Java class. Lets say we have a small music store app that sells vinyl:

(ns myapp.cool-music)

(defrecord Vinyl [speed])

;;; meanwhile ...

(ns myapp.music-store
  (:import [myapp.cool_music Vinyl]))

(instance? Vinyl v)

;; or

(ns myapp.music-store
  (:import [myapp.cool_music.Vinyl]))

(instance? myapp.cool_music.Vinyl v)

There’s a few things to note here:

  1. Because Clojure is generating Java classes, they follow the naming conventions of Java classes, where dashes get converted to underscores.
  2. You should probably prefer the first example where the record is aliased, over the second one where the record is fully qualified.
  3. Clojure dynamically generates the class for the record when that namespace is required (ignoring AOT compiling). If your code never requires the namespace that the record lives in, then it won’t be created. This will cause ClassNotFoundException errors when you try to import the class. In our trivial example, that would mean changing the ns import to:

    (ns myapp.music-store
      (:require [myapp.cool-music]
      (:import [myapp.cool_music Vinyl]))


ClojureScript generates the code for a record in the namespace it is defined in. You can require, refer, and alias a record just like you would any other function in a ClojureScript namespace.

(ns myapp.cool-music)

(defrecord Vinyl [speed])

;;; meanwhile ...

(ns myapp.music-store
  (:require [myapp.cool-music :refer [Vinyl]))

(instance? Vinyl v)

;; or

(ns myapp.music-store
  (:require [myapp.cool-music :as m))

(instance? m/Vinyl v)

;; or

(ns myapp.music-store
  (:require [myapp.cool-music))

(instance? myapp.cool-music/Vinyl v)

There are no hidden gotchas with requiring records in ClojureScript. At one point ClojureScript supported importing records like Clojure, but this was backed out later.

Clojure and ClojureScript

If you want to work with Clojure and ClojureScript (and let’s face it, who doesn’t?) then you can use Reader Conditionals to require a record in both language variants at the same time.

(ns myapp.music-store
  (:require #?(:clj [myapp.cool-music]
               :cljs [myapp.cool-music :refer [Vinyl]]))
     (:import [myapp.cool_music Vinyl])))

(instance? Vinyl v)

git check-ignore: Explaining why git has ignored a file

Most of the time I don’t have any issues with git’s .gitignore system but occasionally it doesn’t behave as I expect. Usually, I add and remove lines until I get the result I’m after and leave feeling vaguely unsatisfied with myself. I always had a nagging feeling that there must be a smarter way to do this, but I was never able to find it. Until today!

Enter git check-ignore. You pass it files on the command line, and it tells you if they’re ignored or not, and why. Sounds pretty perfect right? I won’t give you a run down of all the options, as the man page is surprisingly readable (especially for a git man page!). However the 30-second version is git check-ignore --verbose --non-matching <filepath to check>:

$ echo "b" > .gitignore
$ # a is a file that would not be ignored
$ git check-ignore --verbose --non-matching a
::    a
$ # b is a file that would be ignored (matched)
$ git check-ignore --verbose --non-matching b
.gitignore:1:b    b

--verbose prints the explanation of why a file was ignored. --non-matching prints the file name out, even if Git would not ignore it. If a file is going to be ignored, check-ignore prints out the path to the gitignore file and line that matched. Interestingly, the files don’t even need to exist, git just checks what it would do if they did exist.

Using check-ignore I was able to solve my problem faster, and learn a bit more about git’s arcane exclusion rules. Another handy tool to add to my unix utility belt.

Speeding up Clojure(Script) sorting by 100x

I’ve recently been working on a new ClojureScript application as part of a contract, and I was digging around for things to polish before launch. The app was mostly fast, but I noticed that when the main list of content got to around 40 items, it was a little bit slow to render. I also noticed that it seemed like it got almost twice as slow when I added another 10 items. At this point, you might already be having alarm bells go off in your head suggesting what the problem was likely to be. I didn’t, so I dived into the code to look at the part of the app rendering the main list.

I looked over the code path that rendered the list, and wrapped time around a few suspect pieces of code. After a few checks, I found that a sort-by function in the view was the slow part, though it wasn’t immediately clear why sorting a list of 40 items would take a second. We were using sort-by to order items by state, then reverse date order (newest items first).

sort-by takes a custom sort key function. Our key function was doing some date parsing to parse a string into a date, then subtracting the current unix time from the parsed unix time to give an integer value. The lowest numbers are the most recent dates. I suspected that the date parsing could be the problem, but I wasn’t really sure. As an experiment, I disabled all of the date parsing, and returned the string directly. My sorting was the wrong way around, but it went from taking 1000 ms to 10 ms, a factor of 100x speedup!

A standard sort of the dates (which were in ISO 8601 order, e.g. 2016-04-02T08:24:31+00:00) sorted the oldest dates as the first ones in the list. After a few minutes thinking, I remembered I had recently read the clojure.org guide on comparators. In it, it discusses a reverse comparator:

(fn [a b] (compare b a))

This comparator is exactly the same as a normal comparator, but it will return the opposite result to what the normal one would. Passing this comparator to sort-by kept the 100x speedup, but sorted in the correct order. The list rendered in 10-15 ms, and was now plenty fast enough.

One question remained though, why was the list getting so much slower to render as I added a few more items to it? Of course reading this now, you Dear Reader are probably thinking to yourself, “aha! ClojureScript’s sorting algorithm will be O(n log(n)), and the slow key function will therefore be called O(n log(n)) times.” It took me a bit more thinking than you (I didn’t have a blog post explaining why my code was slow to work from), but I got to this conclusion in the end too.

I really enjoyed this debugging session, as it is not very often that I can both speed up real world code by 100x, and get exposed directly to algorithmic complexity issues. A++, would debug again.

Decompress - A new podcast I’m hosting

I’m excited to announce I’m starting a new podcast Decompress, hosted by myself and my long time friend Nathan Tiddy. Decompress is a technology podcast, with a New Zealand perspective. We also talk about the wider world of things that touch technology, and how it relates to our lives.

If you like technology, and podcasts, then you may like this too. You can subscribe through iTunes, or in your podcast client of choice, by searching for “Decompress”.