Daniel Compton

The personal blog of Daniel Compton - Projects

Tracking down memory leaks while testing on Circle CI

In December, I tried to set up ClojureScript testing with doo and it was going surprisingly well. Everything was running well locally, so I setup a circle.yml file, and pushed my changes up to CircleCI. They ran there on the first time as well, more surprise! CircleCI have clearly put a lot of energy into getting a good setup for browser testing. I was very pleased, until after a few builds, I started running out of memory while running the tests. Circle CI only allows you 4GB which should be plenty for most applications, and certainly for compiling ClojureScript. I tried rerunning the build, sometimes it would complete successfully, and other times it would hit the RAM limit. I tested it on my machine and the tests ran fine, and only used ~ 2GB RAM.

I wasn’t quite sure what was going on so I gave it a break for a few hours. When I came back I ssh’d onto the build machine and ran top while the test was running. I could see my process running and taking more and more memory, getting closer and closer to 4GB. Then the tests finished and the memory was reclaimed. While I was running top, I saw Mem: 251902904k total and thought that was curious. It looked like it was saying I had 240GB, but I thought I was only allowed 4?

This triggered a thought, if the JVM saw that it had 240 GB of memory, then it might easily blow past 4GB of memory while running a ClojureScript build. I checked the project.clj file and saw that it didn’t have any :jvm-opts set in it. I checked the default JVM memory options on the build server:

$ java -XX:+PrintFlagsFinal -version | grep HeapSize 
uintx ErgoHeapSizeLimit = 0 {product} 
uintx HeapSizePerGCThread = 87241520 {product} 
uintx InitialHeapSize := 2147483648 {product} 
uintx LargePageHeapSizeThreshold = 134217728 {product} 
uintx MaxHeapSize := 32126271488 {product}

2147483648 bytes for intial heap size (xms) converts to 2.15 GB, and 32126271488 bytes for max heap size (xmx) converts to 32.13 GB. Once I saw that, the problem became very clear. The JVM was starting with 2 GB of RAM, and had a max heap size of 32 GB. It had no need to GC anywhere near 4GB, so its memory usage would just keep climbing. Depending on how fast the build ran, my tests were sometimes able to finish before the CI hit the 4 GB limit. I set :jvm-opts ^:replace ["-Xms256m" "-Xmx2g"] and reran the build a few times. It was able to successfully run every time. Success!

A Context Free Grammar

The Everything Store has a story in it about how Jeff Bezos came up with the idea for Amazon, an ‘everything store’.

Bezos concluded that a true everything store would be impractical—at least at the beginning. He made a list of twenty possible product categories, including computer software, office supplies, apparel, and music. The category that eventually jumped out at him as the best option was books. They were pure commodities; a copy of a book in one store was identical to the same book carried in another, so buyers always knew what they were getting. There were two primary distributors of books at that time, Ingram and Baker and Taylor, so a new retailer wouldn’t have to approach each of the thousands of book publishers individually. And, most important, there were three million books in print worldwide, far more than a Barnes & Noble or a Borders superstore could ever stock.

We all know how Amazon became a success in the dotcom boom. What isn’t quite so well remembered were the imitators: pets.com, eToys.com, and other websites sprung up trying to be the Amazon for pets or toys. These businesses tried to copy Amazon’s visible activities of selling a category of product online, without understanding the context, business model, and reason for choosing books to sell. The x for y phenomenon is still going strong today. A few years ago it used to be Facebook or Pinterest of Y. Today it’s Uber for Y. These companies are copying the visible parts of a business, but they can’t copy the context, and underlying reasons for a business’s success.

The same thing happens when companies copy design. It’s one thing to copy the trade dress of a product, and this takes some effort and thought. But you can’t copy the thinking that went in to the design of a product, and you’re likely to miss the small touches when you do so.

A few years ago I read how Stripe handles group email with Google Group lists and I wanted to replicate it at my job. I imagined how efficient this would let us be in our communications and how it would help us deal with projects much more efficently. But in the back of my head I had a nagging feeling that it wouldn’t work for us. At the time I didn’t know exactly why, but I parked the idea. Looking back I understand it now. I was trying to copy a practice that had grown out of Stripe’s specific company culture and values and transplant it into my own workplace with completely different values and world views. I could copy the system but I couldn’t copy the context.

When looking to learn from another business, instead of looking at the superficial exterior that is easy to see and understand, we should look at the core of why a product or process exists, and why people use it. Only once we have that understanding, can we see how to apply it to our own contexts or look for new opportunities based on that insight. Clayton Christensen’s ‘Jobs to be done’ theory is a powerful tool for doing this.

A context free grammar may be useful for describing regular expressions and automata, but it’s not so useful for analysing businesses, products and processes.

State of Clojure Survey 2015 Analysis

Cognitect has released the State Of Clojure 2015 results. Justin Gehtland did a good analysis on the quantitative parts of the survey, and summarizing the comments section at the end. I read (most) all of them and got a feel for the zeitgeist of where the Clojure community is looking for improvements. After compliments, error messages and documentation far and away received the most comments. Below are some quotes grouped by subject, to help you get a feel for the responses. You can read all of the comments on the SurveyMonkey page.

Error messages

Error messages have been a pain point with Clojure since the first Clojure survey in 2010. Colin Fleming’s talk on Improving Clojure’s Error Messages with Grammars represents a bright spot on the horizon, and I hope that some variant of this comes to Clojure and ClojureScript. I’d highly recommend watching the talk if you haven’t already.

I’d love to see elm-like approach to hard tasks like improving on error reporting and documentation. Clojure team is so skilled that it won’t bother even with low hanging fruit. But it could be hurtful in the long run.

Regarding potential future enhancements, improved error reporting would be very helpful. This has been a major stumbling block for our junior developers in particular. Too many error messages require a high level of understanding of Clojure internals to understand. Overall, I’m very appreciative of the well-thought-out design decisions in both Clojure and its ecosystem.

Please fix/finish transitive AOT, and please make error messages better. I know error messages continue to receive this sort of “well, if you wanna make ‘em better, patches welcome…” treatment, but I think it’s a mistake to keep beating up on them, insinuating they don’t really matter. As someone who’s spent a fair amount of time debugging Clojure programs, some of them do matter, and I think the wrong message is being sent to would-be contributors about the value in improving them.

Better error messages from macros! It can be done with PEGs quite easily.

Please improve error messages - Elm developers are a good example to follow.

2+ years with Clojure. First one was strong, but the community has been solving non-problems (transducers, reducers) instead of improving error messages and general debugability.

Highest on my list of annoyances are NullPointerExceptions missing stack traces. They they disproportionately more time to debug that other failures. We rarely see them in production but during development they are a constant scourge.

Clojure has the problem that many languages have —- the people capable of providing a better UX aren’t interested in solving that problem. (Myself included.) We could learn from the Go core team’s willingness to solve common UX problems (fast compiler, auto-formatter) and other projects like Elm’s compiler work to support good error messages.


clojure.org has undergone a massive visual refresh between the survey being taken, and the results being released. I have high hopes that it will become a central source for hiqh quality, official documentation and ease the pain for newcomers.

Community & resource wise, everything things feel extremely federated, in a bad way. Documentation This is where I had the worst experience with Clojure. Documentation is extremely federated. To understand clojure.zip & clojure.core.logic took me so much work […]

Many things that disappoint you about Clojure at first (“why errors are so awful?”, “why the contribution process is so hard?”, ” why don’t you implement this one feature I like? “), they fade away as you learn Clojure and Clojure Way more. Still, it would nice to have a sort of explain-like-I’m-five FAQ somewhere on clojure.org that covers these non-obvious points so that people skip the frustration part and understand the reasoning behind those decisions immediately.

CLJS could use better official documentation.

I still don’t know how to get externs working with either lein or boot!! Evaluating code in vim doesn’t always work. Documentation on google are just not reliable because they are outdated or assumes too much pre-knowledge. Applying what’s read in the docs most often do not work, or I just don’t understand. The documentation must really be much more comprehensive and not assume the readers are clojure pros.

Documentation! The clojure.org documentation is so terse that it is unusable for beginners. The clojuredocs.org documentation is confusing for someone who wants the learn the basics. Some examples are too cryptic and appears to be written by very smart people showing off edge uses of the feature they’re describing.

Clojure needs more outreach. Clojure.org or clojuredocs.org needs more tutorials, application-development documentation pointers (For example: “here are your choices if you want to make webapps with Clojure”).

Very happy with progress on cljs.js (tho remains sparsely documented). Much of my pain is related to the compilation step. Lots of errors still happen only after compilation, & not easy to know how to split large code base and load only necessary modules, etc.

Re. documentation: Features added to clojure.core usually have far inferior documentation on initial release than many libraries. Sometimes this is eventually remedied by the community, but sometimes not. This has been the case for reducers, core.async, the new threading macros, and transducers. Even the core library functions didn’t have much example-based documentation until the community took it upon themselves to make clojuredocs. […] Actual documentation, with practical examples, would lead to faster and more widespread adoption of new features.

We just had a Clojure user group meeting here in PDX with several new faces, new to Clojure. […] One person pointed out how the Clojure home page makes no mention of Leiningen, the de-facto standard build tool. The references to running Clojure apps with “java -jar …” are that much more intimidating for anyone coming from a non-Java background. Some of these issues were well described in the “ClojureScript for Skeptics” talk at the Conj. Consensus: people love, or want to love, the language (I do!) but the rough edges have not been smoothed out, and it is still intimidating to get started in Clojure. — Howard


Setup and tooling has historically been another pain point in Clojure. In the ClojureScript world, Figwheel seems to be solving a lot of hard problems (REPL, reloading code) in a relatively easy way to set up, and I’m sure it will get even better this year. As an alternative to emacs, Cursive (which I use) is a great tool for getting a full Clojure development environment setup with minimal hassle.

By far, the biggest complaints I hear from friends and at meetups is the startup time and the work required to get a ClojureScript workflow set up.

Clojurescript project setup remains really involved.

Tooling tooling tooling

[…] EMACS SETUP: IMPOSSIBLE I used and liked emacs 15 years ago and prefer to use emacs for clojure development now, but after a full day of trying I still can’t get all the emacs pieces together. It’s too complex, there are too many half-baked and out-of-date how-to examples. Will the real viable emacs setup tutorial please stand up?

Cursive and it’s debugger are a Godsend - I couldn’t ever convince myself to use Emacs and all the alternatives were inferior. Cursive made writing and debugging Clojure a really pleasant experience. Same with boot in Clojurescript world - getting lein + cljbsuild + figwheel working well was a pain. Boot+boot-cljs+boot-reload+boot-cljs-repl is a pleasure. I think those two things will do a lot good towards Clojure and Clojurescript adoption.


Convincing coworkers to adopt Clojure also was a pain point for some of the respondents, and additional marketing materials could help here. Cognitect’s case studies are good, and could be ported to the official Clojure website.

The clj / cljs / om / figwheel websites make Clojure appear like a niche ecosystem. Seems to be time for marketing!

needs better marketing. clojure has bad marketing (great product, bad marketing) typesafe has good marketing (complex product, good marketing) 10gen has great marketing (bad product, great marketing) it is hurting companies wide adoption: “Clojure? Um.. What is it you said?” “Scala, yea, I’ve heard good things about it, it’s solid.” “MongoDB? Oh, yea my teams tell me it’s really good.”

Love clojure and clojureScript. Where I need help is on the “evangelical” side. If we had more people singing the praises of clojure that would help my business secure more clojure business and everyone else’s work as well. This brings in more money and talent which can then be reinvested into tooling, systems, turn-around development time,etc. just look at what’s happened over in the JS world once their reputation improved.

Overall, very happy with the current direction of Clojure. What feels missing is awareness in the market about the goods of is as a technology, as the biggest impediment with using Clojure more has been finding customers willing to embrace the risk. Alternatives like Scala managed to have a better name within the industry.

Clojure deserves a better website and documentation.

Still trying to persuade co-workers to use Clojure more.


In my opinion, the Clojure community is one of Clojure’s greatest assets. I’ve found it to be overwhelmingly positive, and there are a massive amount of innovative libraries coming from a very young community. There were also some concerns about the contribution process, and how many of the decisions about Clojure happen inside Cognitect.

Many thanks to Rich, Stu, David, Alex and many other people from community and Cognitect. You are doing amazing job. I love the culture of community you created around Clojure: open community with emphasis on learning, design, thinking hard and solving the right problem. It is inspiring place to be in.

What an amazing language. Relatively frequent, consistently stable releases. A pleasure to use. A friendly, smart community. I feel very lucky to be a Clojure user! Thank you for all of your hard work.

The Clojure contrib process frustrates me more than any technical or community aspect of the language.

Clojure gets a lot right, but as has been repeatedly discussed the pace of evolution and the maintainership’s dim view of 3rd party non-bugfix work flatly leads to worthy but minor work such as type predicates going largely ignored and certainly unmerged. In most open source projects, contributors can impact priorities by giving of their time to support work which isn’t high priority. The work which Nola Stowe did on clojure.string was awesome and I think we can see more of that if Alex et all allocate more capacity to working with contributors.

I found some of the recent back-and-forth about tuples to be very disheartening—-if I were Zach Tellman I would decide not to bother attempting to make improvements to the core of Clojure. […]

i have become a little worried about the future of the language. The core team are way ahead in terms of design and concepts, but seem to lack any kind of empathy with beginners.

Things I think should be a priority for the long term future of Clojure: […] 3. More open core development process. I worry about the long term future of Clojure when it appears to be driven by internal decisions within a single company (i.e. Cognitect). I am uncomfortable with this approach, and am hesitant to commit fully to Clojure as a language until this becomes more open. Maybe a “Clojure Foundation” would be a good idea?

Thanks to everyone again for another great year in CLJ(S)land. Best community for any language.

I participate in a local meetup group which has helped greatly and I find the community very welcoming (especially being a woman, I’m not finding as many issues as I do with other communities). Thanks!

Keep up the great work! Its been a joy to participate in this community :)

I would like to thank entire Cognitect staff for creating such a great environment and community.


Although there are some guides like Clojure Toolbox, awesome-clojure and the Luminus useful libraries list, it seems like there are still pain points around library discovery.

Finding good libraries has not been difficult, but it would be helpful to have some mechanism in the community for evaluating which libraries are viable in the long term as well as which are “best in class”, so to speek. In many cases, I want a library that will provide a useful component for a multi-year project. The library does what I need and is available, but the modifications on github are 3+ years ago. Should I invest in using this library in my core code? Are there better, more current and maintained options? And so forth. Some community aggregation that makes it easy to evaluate libraries in this sense would be very helpful. […]

Libraries Not sure where the main place is to find clojure libraries. Some are in maven (with no ability to evaluate the library). Clojars also feels incomplete and hard to evaluate. The landing page http://clojure.org/libraries even suggests checking maven, clojars, and github, it feels like a dead end. Google is better resource than these, because you can find blogs evaluating the libraries.


I was surprised to see so many comments about static typing, but it was a recurring theme. They can roughly be divided into three groups:

  1. People who were interested in structural typing (Prismatic Schema), or gradual typing (core.typed).
  2. People who were getting push back from their organisation for using a dynamic language.
  3. People who wanted a type system like Haskell’s.

There seems to be a lot of interest in core.typed but people are waiting at the sidelines until it reaches more maturity.

I was surprised to see that the survey doesn’t include reference to usage of typing or contract tools like core.typed or Schema. I’m interested in understanding how widespread their usage is. Schema’s been an important tool in helping to encourage Clojure’s more widespread use at our enterprise and I’ve been largely happy with it.

Clojure and ClojureScript are amazing projects and the community is pushing the edges of programming today (Om Next, Datomic, cljc). I’ve been close to using both many times, but at this point I don’t think I could leave the expressive syntax or type systems in Haskell and PureScript. I wish they had the same community and dev experience, and am interested in using the Clojure community’s ideas there.

People in my organization feel strongly that dynamic typing won’t scale to projects with more than a few developers. I am concentrating on small projects. Also, the learning curve for Java devs is a bit steep. Like anything else, one has to really want it.

The two things holding back Clojure/ClojureScript use at work are corporate policy and a strong fear of weakly typed production languages by the team.

I love clojure but I also like static type systems. Don’t know if its possible to two in a way that works well. Haskell and scala are languages I like for its type system

The discussion we have been having lately though is around static typing. I have a Haskell background (and a Phd in type systems, even), so I’m aware of the benefits they can provide but I don’t think they’re a silver bullet either. None-the-less, improved static support seems incredibly useful. A gradual-typing approach does seem ideal, allowing you to specify either types up front, or to add annotations as the design solidifies, perhaps. I’m eagerly watching Ambrose’s work, and perhaps prismatic-schema is enough — it would be great to get some of the core community’s thoughts on the matter though.

I really, really, really want to see some form of official support for static typing. This could be core.typed, but it needs to be blessed by the core Clojure dev team and development to allow core.typed (and other future tools) to better integrate with the compiler is necessary. Type annotations provide helpful documentation and (optional) static type checking can help speed up development for those already annotating for documentation purposes by detecting type errors before executing programs. I see the lack of support for (optional) static typing as Clojure’s biggest weakness and the main reason I have considered moving to a different language.


Like Justin said, over half of the responses were happy or positive. Here’s a few to finish on an upbeat note. I’m looking forward to seeing what 2016 brings for Clojure, and especially ClojureScript. The tooling story seems to be coming together quite well. If enhanced error messages make it into Clojure 1.9, and the documentation story improves, a lot of the pain in using Clojure could be minimised.

(= “Clojure” “Awesome!”) => true

Thanks for everything that you do. :)

Love it! Thank you for everything that you do.

I just love programming in clojure.

Very nice thing. I like to program in this Lisp a lot

I would never have written PartsBox.io if it wasn’t for Clojure and ClojureScript. Only these tools allow for such a productivity increase that a single person can write a complete medium-size application.

You rock guys, I love you!!!

Onward and upward!