Before we get started, I want to be clear. I don’t support tobacco, cluster bomb manufacturers, nuclear weapons manufacturers, and the other socially harmful businesses mentioned in the recent brouhaha about Kiwisaver investments into the aforementioned companies. However, I think the reporting that NZ Herald did on this was misleading, in search of a good story at the expense of accuracy.

To recap: the Herald has recently been reporting on Kiwisaver, the NZ government retirement savings scheme (like an IRA). Their big headline was that $150mm has been invested by the major Kiwisaver providers into companies that produce cluster bombs, landmines, and other socially harmful products, and that they may be breaking a law banning investments into cluster bomb manufacturers. When you look into the details, their bold claims don’t look so strong.

Here are a few points that should have been included in the reporting:

• The biggest problem is that the Herald doesn’t distinguish between active management (where fund managers or algorithms chose particular businesses to invest in), and passive management that tracks an index (say agriculture, energy, or the USA stock market). If some of these funds were directly invested in these companies, and banks really were breaking the law, that would be a real story. It’s not clear to me from the reporting whether there was any direct investment into the companies, or investment into clusterbomb index funds, or whether the funds were invested in broad indexes. You can make an argument for both being bad, but active investment in these companies is very different from passively investing in an index fund of the total stock market. The Herald implies that these companies deliberately or knowingly invested in the companies, but I couldn’t see this from the data they provided.
• The total amount invested in Kiwisaver is $32.5 billion, and the amount invested in socially harmful businesses is $150 million. This works out to around 0.46% of the total funds invested. This is not nothing, but it is a tiny fraction of the total assets invested.
• Kiwisaver funds that aren’t investing in these companies are likely doing so through active management of stocks. In general, this will result in higher fees, and depending on who you ask, lower performance (especially once fees are taken into account).
• The charts that NZ Herald produced all give absolute numbers for how much is invested in socially harmful companies, without giving the percentage invested. Westpac, ANZ, and ASB all feature high on the list of investing into these harmful companies, but there is no context given for what percentage of the investments each have made. Westpac has 0.76% in socially harmful businesses. ANZ has 1.27%, and ASB has 0.13% ¹. Without more details on total amount invested in stocks (active and passive) by each fund, it’s hard to tell where and why the differences here come from.
• The amount invested in the socially harmful parts of Northrop Grumman² and General Dynamics ³ are very small, compared to their overall businesses.

Data journalism can be used to illuminate complex topics and explain them for a wide audience. It seems in this reporting that the story came first, and the numbers were presented in a misleading way to back it up. There is a nuanced discussion that could have been had about the ethics of index funds, and socially responsible investing, but that wasn’t what we got here.

N.B.: I may have read the article wrong, and all of the figures provided were active investments (it’s not clear at all to me which is being included). If that’s the case my conclusion would be quite different.

Update: More discussion index/active investments may be coming later this week.

Every week in The REPL I have a section called: “People are worried about Types”. This is based on a section that Matt Levine (my favourite columnist) includes in his daily column “People are worried about bond market liquidity”, and the related “People are worried about unicorns”, and “People are worried about stock buybacks”.

The title is a joke (as is his), I don’t really think there are people worried about types, but it is interesting to see a small but steady flow of type or type related (schema, spec, e.t.c.) articles relating to Clojure each week.

I’ve started a weekly newsletter about Clojure and ClojureScript. Each week will have a curated selection of links (both recent, and older) and a sentence or two about why I think they’re worth reading. You can sign up at therepl.net.

One of the datatypes Clojure gives you is the Record. It is used in both Clojure and ClojureScript, but due to implementation differences, the syntax to use records from one namespace in another is different.

Clojure

Clojure generates a Java class for every defrecord that you create. You need to :import them, just like you would a standard Java class. Lets say we have a small music store app that sells vinyl:

(ns myapp.cool-music)

(defrecord Vinyl [speed])

;;; meanwhile ...

(ns myapp.music-store
  (:import [myapp.cool_music Vinyl]))

(instance? Vinyl v)

;; or

(ns myapp.music-store
  (:import [myapp.cool_music.Vinyl]))

(instance? myapp.cool_music.Vinyl v)

There’s a few things to note here:

1. Because Clojure is generating Java classes, they follow the naming conventions of Java classes, where dashes get converted to underscores.
2. You should probably prefer the first example where the record is aliased, over the second one where the record is fully qualified.

3. Clojure dynamically generates the class for the record when that namespace is required (ignoring AOT compiling). If your code never requires the namespace that the record lives in, then it won’t be created. This will cause ClassNotFoundException errors when you try to import the class. In our trivial example, that would mean changing the ns import to:

   (ns myapp.music-store
     (:require [myapp.cool-music]
     (:import [myapp.cool_music Vinyl]))
   

ClojureScript

ClojureScript generates the code for a record in the namespace it is defined in. You can require, refer, and alias a record just like you would any other function in a ClojureScript namespace.

(ns myapp.cool-music)

(defrecord Vinyl [speed])

;;; meanwhile ...

(ns myapp.music-store
  (:require [myapp.cool-music :refer [Vinyl]))

(instance? Vinyl v)

;; or

(ns myapp.music-store
  (:require [myapp.cool-music :as m))

(instance? m/Vinyl v)

;; or

(ns myapp.music-store
  (:require [myapp.cool-music))

(instance? myapp.cool-music/Vinyl v)

There are no hidden gotchas with requiring records in ClojureScript. At one point ClojureScript supported importing records like Clojure, but this was backed out later.

Clojure and ClojureScript

If you want to work with Clojure and ClojureScript (and let’s face it, who doesn’t?) then you can use Reader Conditionals to require a record in both language variants at the same time.

(ns myapp.music-store
  (:require #?(:clj [myapp.cool-music]
               :cljs [myapp.cool-music :refer [Vinyl]]))
  #?(:clj
     (:import [myapp.cool_music Vinyl])))

(instance? Vinyl v)

Most of the time I don’t have any issues with git’s .gitignore system but occasionally it doesn’t behave as I expect. Usually, I add and remove lines until I get the result I’m after and leave feeling vaguely unsatisfied with myself. I always had a nagging feeling that there must be a smarter way to do this, but I was never able to find it. Until today!

Enter git check-ignore. You pass it files on the command line, and it tells you if they’re ignored or not, and why. Sounds pretty perfect right? I won’t give you a run down of all the options, as the man page is surprisingly readable (especially for a git man page!). However the 30-second version is git check-ignore --verbose --non-matching <filepath to check>:

$ echo "b" > .gitignore
$ # a is a file that would be matched
$ git check-ignore --verbose --non-matching a
::    a
$ # b is a file that would be ignored
$ git check-ignore --verbose --non-matching b
.gitignore:1:b    b

--verbose prints the explanation of why a file was ignored. --non-matching makes sure the file is printed out, even if it won’t be ignored. If a file is going to be ignored, check-ignore prints out the path to the gitignore file and line that matched. Interestingly, the files don’t even need to exist, git just checks what it would do if they did exist.

Using check-ignore I was able to solve my problem faster, and learn a bit more about git’s arcane exclusion rules. Another handy tool to add to my unix utility belt.

I’ve recently been working on a new ClojureScript application as part of a contract, and I was digging around for things to polish before launch. The app was mostly fast, but I noticed that when the main list of content got to around 40 items, it was a little bit slow to render. I also noticed that it seemed like it got almost twice as slow when I added another 10 items. At this point, you might already be having alarm bells go off in your head suggesting what the problem was likely to be. I didn’t, so I dived into the code to look at the part of the app rendering the main list.

I looked over the code path that rendered the list, and wrapped time around a few suspect pieces of code. After a few checks, I found that a sort-by function in the view was the slow part, though it wasn’t immediately clear why sorting a list of 40 items would take a second. We were using sort-by to order items by state, then reverse date order (newest items first).

sort-by takes a custom sort key function. Our key function was doing some date parsing to parse a string into a date, then subtracting the current unix time from the parsed unix time to give an integer value. The lowest numbers are the most recent dates. I suspected that the date parsing could be the problem, but I wasn’t really sure. As an experiment, I disabled all of the date parsing, and returned the string directly. My sorting was the wrong way around, but it went from taking 1000 ms to 10 ms, a factor of 100x speedup!

A standard sort of the dates (which were in ISO 8601 order, e.g. 2016-04-02T08:24:31+00:00) sorted the oldest dates as the first ones in the list. After a few minutes thinking, I remembered I had recently read the clojure.org guide on comparators. In it, it discusses a reverse comparator:

(fn [a b] (compare b a))

This comparator is exactly the same as a normal comparator, but it will return the opposite result to what the normal one would. Passing this comparator to sort-by kept the 100x speedup, but sorted in the correct order. The list rendered in 10-15 ms, and was now plenty fast enough.

One question remained though, why was the list getting so much slower to render as I added a few more items to it? Of course reading this now, you Dear Reader are probably thinking to yourself, “aha! ClojureScript’s sorting algorithm will be O(n log(n)), and the slow key function will therefore be called O(n log(n)) times.” It took me a bit more thinking than you (I didn’t have a blog post explaining why my code was slow to work from), but I got to this conclusion in the end too.

I really enjoyed this debugging session, as it is not very often that I can both speed up real world code by 100x, and get exposed directly to algorithmic complexity issues. A++, would debug again.

I’m excited to announce I’m starting a new podcast Decompress, hosted by myself and my long time friend Nathan Tiddy. Decompress is a technology podcast, with a New Zealand perspective. We also talk about the wider world of things that touch technology, and how it relates to our lives.

If you like technology, and podcasts, then you may like this too. You can subscribe through iTunes, or in your podcast client of choice, by searching for “Decompress”.

Plumatic (nee Prismatic) Schema is a great tool for adding structural typing to your code.

Specifying standard Clojure types is fairly straightforward, but I recently needed to specify that a value was a goog.date.UtcDateTime. I did a bit of searching on Google, didn’t find anything. I then looked in the company codebase and found it. Here’s a minimal example:

(ns myns
  (:require [goog.date.UtcDateTime]))

(def Date js/goog.date.UtcDateTime)

Cognitect has released the State Of Clojure 2015 results. Justin Gehtland did a good analysis on the quantitative parts of the survey, and summarizing the comments section at the end. I read (most) all of them and got a feel for the zeitgeist of where the Clojure community is looking for improvements. After compliments, error messages and documentation far and away received the most comments. Below are some quotes grouped by subject, to help you get a feel for the responses. You can read all of the comments on the SurveyMonkey page.

Error messages

Error messages have been a pain point with Clojure since the first Clojure survey in 2010. Colin Fleming’s talk on Improving Clojure’s Error Messages with Grammars represents a bright spot on the horizon, and I hope that some variant of this comes to Clojure and ClojureScript. I’d highly recommend watching the talk if you haven’t already.

I’d love to see elm-like approach to hard tasks like improving on error reporting and documentation. Clojure team is so skilled that it won’t bother even with low hanging fruit. But it could be hurtful in the long run.

Regarding potential future enhancements, improved error reporting would be very helpful. This has been a major stumbling block for our junior developers in particular. Too many error messages require a high level of understanding of Clojure internals to understand. Overall, I’m very appreciative of the well-thought-out design decisions in both Clojure and its ecosystem.

Please fix/finish transitive AOT, and please make error messages better. I know error messages continue to receive this sort of “well, if you wanna make ‘em better, patches welcome…” treatment, but I think it’s a mistake to keep beating up on them, insinuating they don’t really matter. As someone who’s spent a fair amount of time debugging Clojure programs, some of them do matter, and I think the wrong message is being sent to would-be contributors about the value in improving them.

Better error messages from macros! It can be done with PEGs quite easily.

Please improve error messages - Elm developers are a good example to follow.

2+ years with Clojure. First one was strong, but the community has been solving non-problems (transducers, reducers) instead of improving error messages and general debugability.

Highest on my list of annoyances are NullPointerExceptions missing stack traces. They they disproportionately more time to debug that other failures. We rarely see them in production but during development they are a constant scourge.

Clojure has the problem that many languages have —- the people capable of providing a better UX aren’t interested in solving that problem. (Myself included.) We could learn from the Go core team’s willingness to solve common UX problems (fast compiler, auto-formatter) and other projects like Elm’s compiler work to support good error messages.

Docs

clojure.org has undergone a massive visual refresh between the survey being taken, and the results being released. I have high hopes that it will become a central source for hiqh quality, official documentation and ease the pain for newcomers.

Community & resource wise, everything things feel extremely federated, in a bad way. Documentation This is where I had the worst experience with Clojure. Documentation is extremely federated. To understand clojure.zip & clojure.core.logic took me so much work […]

Many things that disappoint you about Clojure at first (“why errors are so awful?”, “why the contribution process is so hard?”, ” why don’t you implement this one feature I like? “), they fade away as you learn Clojure and Clojure Way more. Still, it would nice to have a sort of explain-like-I’m-five FAQ somewhere on clojure.org that covers these non-obvious points so that people skip the frustration part and understand the reasoning behind those decisions immediately.

CLJS could use better official documentation.

I still don’t know how to get externs working with either lein or boot!! Evaluating code in vim doesn’t always work. Documentation on google are just not reliable because they are outdated or assumes too much pre-knowledge. Applying what’s read in the docs most often do not work, or I just don’t understand. The documentation must really be much more comprehensive and not assume the readers are clojure pros.

Documentation! The clojure.org documentation is so terse that it is unusable for beginners. The clojuredocs.org documentation is confusing for someone who wants the learn the basics. Some examples are too cryptic and appears to be written by very smart people showing off edge uses of the feature they’re describing.

Clojure needs more outreach. Clojure.org or clojuredocs.org needs more tutorials, application-development documentation pointers (For example: “here are your choices if you want to make webapps with Clojure”).

Very happy with progress on cljs.js (tho remains sparsely documented). Much of my pain is related to the compilation step. Lots of errors still happen only after compilation, & not easy to know how to split large code base and load only necessary modules, etc.

Re. documentation: Features added to clojure.core usually have far inferior documentation on initial release than many libraries. Sometimes this is eventually remedied by the community, but sometimes not. This has been the case for reducers, core.async, the new threading macros, and transducers. Even the core library functions didn’t have much example-based documentation until the community took it upon themselves to make clojuredocs. […] Actual documentation, with practical examples, would lead to faster and more widespread adoption of new features.

We just had a Clojure user group meeting here in PDX with several new faces, new to Clojure. […] One person pointed out how the Clojure home page makes no mention of Leiningen, the de-facto standard build tool. The references to running Clojure apps with “java -jar …” are that much more intimidating for anyone coming from a non-Java background. Some of these issues were well described in the “ClojureScript for Skeptics” talk at the Conj. Consensus: people love, or want to love, the language (I do!) but the rough edges have not been smoothed out, and it is still intimidating to get started in Clojure. — Howard

Setup/tooling

Setup and tooling has historically been another pain point in Clojure. In the ClojureScript world, Figwheel seems to be solving a lot of hard problems (REPL, reloading code) in a relatively easy way to set up, and I’m sure it will get even better this year. As an alternative to emacs, Cursive (which I use) is a great tool for getting a full Clojure development environment setup with minimal hassle.

By far, the biggest complaints I hear from friends and at meetups is the startup time and the work required to get a ClojureScript workflow set up.

Clojurescript project setup remains really involved.

Tooling tooling tooling

[…] EMACS SETUP: IMPOSSIBLE I used and liked emacs 15 years ago and prefer to use emacs for clojure development now, but after a full day of trying I still can’t get all the emacs pieces together. It’s too complex, there are too many half-baked and out-of-date how-to examples. Will the real viable emacs setup tutorial please stand up?

Cursive and it’s debugger are a Godsend - I couldn’t ever convince myself to use Emacs and all the alternatives were inferior. Cursive made writing and debugging Clojure a really pleasant experience. Same with boot in Clojurescript world - getting lein + cljbsuild + figwheel working well was a pain. Boot+boot-cljs+boot-reload+boot-cljs-repl is a pleasure. I think those two things will do a lot good towards Clojure and Clojurescript adoption.

Marketing/Adoption

Convincing coworkers to adopt Clojure also was a pain point for some of the respondents, and additional marketing materials could help here. Cognitect’s case studies are good, and could be ported to the official Clojure website.

The clj / cljs / om / figwheel websites make Clojure appear like a niche ecosystem. Seems to be time for marketing!

needs better marketing. clojure has bad marketing (great product, bad marketing) typesafe has good marketing (complex product, good marketing) 10gen has great marketing (bad product, great marketing) it is hurting companies wide adoption: “Clojure? Um.. What is it you said?” “Scala, yea, I’ve heard good things about it, it’s solid.” “MongoDB? Oh, yea my teams tell me it’s really good.”

Love clojure and clojureScript. Where I need help is on the “evangelical” side. If we had more people singing the praises of clojure that would help my business secure more clojure business and everyone else’s work as well. This brings in more money and talent which can then be reinvested into tooling, systems, turn-around development time,etc. just look at what’s happened over in the JS world once their reputation improved.

Overall, very happy with the current direction of Clojure. What feels missing is awareness in the market about the goods of is as a technology, as the biggest impediment with using Clojure more has been finding customers willing to embrace the risk. Alternatives like Scala managed to have a better name within the industry.

Clojure deserves a better website and documentation.

Still trying to persuade co-workers to use Clojure more.

Community

In my opinion, the Clojure community is one of Clojure’s greatest assets. I’ve found it to be overwhelmingly positive, and there are a massive amount of innovative libraries coming from a very young community. There were also some concerns about the contribution process, and how many of the decisions about Clojure happen inside Cognitect.

Many thanks to Rich, Stu, David, Alex and many other people from community and Cognitect. You are doing amazing job. I love the culture of community you created around Clojure: open community with emphasis on learning, design, thinking hard and solving the right problem. It is inspiring place to be in.

What an amazing language. Relatively frequent, consistently stable releases. A pleasure to use. A friendly, smart community. I feel very lucky to be a Clojure user! Thank you for all of your hard work.

The Clojure contrib process frustrates me more than any technical or community aspect of the language.

Clojure gets a lot right, but as has been repeatedly discussed the pace of evolution and the maintainership’s dim view of 3rd party non-bugfix work flatly leads to worthy but minor work such as type predicates going largely ignored and certainly unmerged. In most open source projects, contributors can impact priorities by giving of their time to support work which isn’t high priority. The work which Nola Stowe did on clojure.string was awesome and I think we can see more of that if Alex et all allocate more capacity to working with contributors.

I found some of the recent back-and-forth about tuples to be very disheartening—-if I were Zach Tellman I would decide not to bother attempting to make improvements to the core of Clojure. […]

i have become a little worried about the future of the language. The core team are way ahead in terms of design and concepts, but seem to lack any kind of empathy with beginners.

Things I think should be a priority for the long term future of Clojure: […] 3. More open core development process. I worry about the long term future of Clojure when it appears to be driven by internal decisions within a single company (i.e. Cognitect). I am uncomfortable with this approach, and am hesitant to commit fully to Clojure as a language until this becomes more open. Maybe a “Clojure Foundation” would be a good idea?

Thanks to everyone again for another great year in CLJ(S)land. Best community for any language.

I participate in a local meetup group which has helped greatly and I find the community very welcoming (especially being a woman, I’m not finding as many issues as I do with other communities). Thanks!

Keep up the great work! Its been a joy to participate in this community :)

I would like to thank entire Cognitect staff for creating such a great environment and community.

Libraries

Although there are some guides like Clojure Toolbox, awesome-clojure and the Luminus useful libraries list, it seems like there are still pain points around library discovery.

Finding good libraries has not been difficult, but it would be helpful to have some mechanism in the community for evaluating which libraries are viable in the long term as well as which are “best in class”, so to speek. In many cases, I want a library that will provide a useful component for a multi-year project. The library does what I need and is available, but the modifications on github are 3+ years ago. Should I invest in using this library in my core code? Are there better, more current and maintained options? And so forth. Some community aggregation that makes it easy to evaluate libraries in this sense would be very helpful. […]

Libraries Not sure where the main place is to find clojure libraries. Some are in maven (with no ability to evaluate the library). Clojars also feels incomplete and hard to evaluate. The landing page http://clojure.org/libraries even suggests checking maven, clojars, and github, it feels like a dead end. Google is better resource than these, because you can find blogs evaluating the libraries.

Typing

I was surprised to see so many comments about static typing, but it was a recurring theme. They can roughly be divided into three groups:

1. People who were interested in structural typing (Prismatic Schema), or gradual typing (core.typed).
2. People who were getting push back from their organisation for using a dynamic language.
3. People who wanted a type system like Haskell’s.

There seems to be a lot of interest in core.typed but people are waiting at the sidelines until it reaches more maturity.

I was surprised to see that the survey doesn’t include reference to usage of typing or contract tools like core.typed or Schema. I’m interested in understanding how widespread their usage is. Schema’s been an important tool in helping to encourage Clojure’s more widespread use at our enterprise and I’ve been largely happy with it.

Clojure and ClojureScript are amazing projects and the community is pushing the edges of programming today (Om Next, Datomic, cljc). I’ve been close to using both many times, but at this point I don’t think I could leave the expressive syntax or type systems in Haskell and PureScript. I wish they had the same community and dev experience, and am interested in using the Clojure community’s ideas there.

People in my organization feel strongly that dynamic typing won’t scale to projects with more than a few developers. I am concentrating on small projects. Also, the learning curve for Java devs is a bit steep. Like anything else, one has to really want it.

The two things holding back Clojure/ClojureScript use at work are corporate policy and a strong fear of weakly typed production languages by the team.

I love clojure but I also like static type systems. Don’t know if its possible to two in a way that works well. Haskell and scala are languages I like for its type system

The discussion we have been having lately though is around static typing. I have a Haskell background (and a Phd in type systems, even), so I’m aware of the benefits they can provide but I don’t think they’re a silver bullet either. None-the-less, improved static support seems incredibly useful. A gradual-typing approach does seem ideal, allowing you to specify either types up front, or to add annotations as the design solidifies, perhaps. I’m eagerly watching Ambrose’s work, and perhaps prismatic-schema is enough — it would be great to get some of the core community’s thoughts on the matter though.

I really, really, really want to see some form of official support for static typing. This could be core.typed, but it needs to be blessed by the core Clojure dev team and development to allow core.typed (and other future tools) to better integrate with the compiler is necessary. Type annotations provide helpful documentation and (optional) static type checking can help speed up development for those already annotating for documentation purposes by detecting type errors before executing programs. I see the lack of support for (optional) static typing as Clojure’s biggest weakness and the main reason I have considered moving to a different language.

Compliments

Like Justin said, over half of the responses were happy or positive. Here’s a few to finish on an upbeat note. I’m looking forward to seeing what 2016 brings for Clojure, and especially ClojureScript. The tooling story seems to be coming together quite well. If enhanced error messages make it into Clojure 1.9, and the documentation story improves, a lot of the pain in using Clojure could be minimised.

(= “Clojure” “Awesome!”) => true

Thanks for everything that you do. :)

Love it! Thank you for everything that you do.

I just love programming in clojure.

Very nice thing. I like to program in this Lisp a lot

I would never have written PartsBox.io if it wasn’t for Clojure and ClojureScript. Only these tools allow for such a productivity increase that a single person can write a complete medium-size application.

You rock guys, I love you!!!

Onward and upward!

The Everything Store has a story in it about how Jeff Bezos came up with the idea for Amazon, an ‘everything store’.

Bezos concluded that a true everything store would be impractical—at least at the beginning. He made a list of twenty possible product categories, including computer software, office supplies, apparel, and music. The category that eventually jumped out at him as the best option was books. They were pure commodities; a copy of a book in one store was identical to the same book carried in another, so buyers always knew what they were getting. There were two primary distributors of books at that time, Ingram and Baker and Taylor, so a new retailer wouldn’t have to approach each of the thousands of book publishers individually. And, most important, there were three million books in print worldwide, far more than a Barnes & Noble or a Borders superstore could ever stock.

We all know how Amazon became a success in the dotcom boom. What isn’t quite so well remembered were the imitators: pets.com, eToys.com, and other websites sprung up trying to be the Amazon for pets or toys. These businesses tried to copy Amazon’s visible activities of selling a category of product online, without understanding the context, business model, and reason for choosing books to sell. The x for y phenomenon is still going strong today. A few years ago it used to be Facebook or Pinterest of Y. Today it’s Uber for Y. These companies are copying the visible parts of a business, but they can’t copy the context, and underlying reasons for a business’s success.

The same thing happens when companies copy design. It’s one thing to copy the trade dress of a product, and this takes some effort and thought. But you can’t copy the thinking that went in to the design of a product, and you’re likely to miss the small touches when you do so.

A few years ago I read how Stripe handles group email with Google Group lists and I wanted to replicate it at my job. I imagined how efficient this would let us be in our communications and how it would help us deal with projects much more efficently. But in the back of my head I had a nagging feeling that it wouldn’t work for us. At the time I didn’t know exactly why, but I parked the idea. Looking back I understand it now. I was trying to copy a practice that had grown out of Stripe’s specific company culture and values and transplant it into my own workplace with completely different values and world views. I could copy the system but I couldn’t copy the context.

When looking to learn from another business, instead of looking at the superficial exterior that is easy to see and understand, we should look at the core of why a product or process exists, and why people use it. Only once we have that understanding, can we see how to apply it to our own contexts or look for new opportunities based on that insight. Clayton Christensen’s ‘Jobs to be done’ theory is a powerful tool for doing this.

A context free grammar may be useful for describing regular expressions and automata, but it’s not so useful for analysing businesses, products and processes.

In December, I tried to set up ClojureScript testing with doo and it was going surprisingly well. Everything was running well locally, so I setup a circle.yml file, and pushed my changes up to CircleCI. They ran there on the first time as well, more surprise! CircleCI have clearly put a lot of energy into getting a good setup for browser testing. I was very pleased, until after a few builds, I started running out of memory while running the tests. Circle CI only allows you 4GB which should be plenty for most applications, and certainly for compiling ClojureScript. I tried rerunning the build, sometimes it would complete successfully, and other times it would hit the RAM limit. I tested it on my machine and the tests ran fine, and only used ~ 2GB RAM.

I wasn’t quite sure what was going on so I gave it a break for a few hours. When I came back I ssh’d onto the build machine and ran top while the test was running. I could see my process running and taking more and more memory, getting closer and closer to 4GB. Then the tests finished and the memory was reclaimed. While I was running top, I saw Mem: 251902904k total and thought that was curious. It looked like it was saying I had 240GB, but I thought I was only allowed 4?

This triggered a thought, if the JVM saw that it had 240 GB of memory, then it might easily blow past 4GB of memory while running a ClojureScript build. I checked the project.clj file and saw that it didn’t have any :jvm-opts set in it. I checked the default JVM memory options on the build server:

$ java -XX:+PrintFlagsFinal -version | grep HeapSize 
uintx ErgoHeapSizeLimit = 0 {product} 
uintx HeapSizePerGCThread = 87241520 {product} 
uintx InitialHeapSize := 2147483648 {product} 
uintx LargePageHeapSizeThreshold = 134217728 {product} 
uintx MaxHeapSize := 32126271488 {product}

2147483648 bytes for intial heap size (xms) converts to 2.15 GB, and 32126271488 bytes for max heap size (xmx) converts to 32.13 GB. Once I saw that, the problem became very clear. The JVM was starting with 2 GB of RAM, and had a max heap size of 32 GB. It had no need to GC anywhere near 4GB, so its memory usage would just keep climbing. Depending on how fast the build ran, my tests were sometimes able to finish before the CI hit the 4 GB limit. I set :jvm-opts ^:replace ["-Xms256m" "-Xmx2g"] and reran the build a few times. It was able to successfully run every time. Success!

We cannot solve our problems with the same thinking we used when we created them. — Albert Einstein

In my (relatively short) career as a software engineer, I’ve seen a paradoxical behaviour among coworkers and myself: at the point when people most need better debugging tools, they are the least likely to use them.

This behaviour doesn’t seem to be exclusive to any one language, domain, or Operating System. The pattern I see is:

1. Get stuck with a bug
2. Try to debug the problem using the methods you’re comfortable with
3. Keep going, getting more frustrated/nervous/impatient
4. Realise that you’re running out of time, and double down twice as hard on your current methods.

It’s steps 3 and 4 that are most interesting to me, and where the paradoxical behaviour occurs. I can think of a few reasons for why people do this:

• When you are debugging something, you will usually spend some time debugging it with your normal methods. By the time you realise that you are stuck, you may be behind schedule, so you double down on your existing method.
• When you don’t feel like you have enough time, you are not as likely to explore alternative tools and debugging approaches.
• Often when debugging you can feel like the answer is just around the corner (even if it’s still a few hours away).
• If you’re already neck deep in one confusing situation, adding a new unfamiliar tool doesn’t seem like it will help.

I’m not sure if there is a good way to combat this problem in the heat of the moment. Two solutions I can see:

1. Try debugging with someone more experienced than you who is familiar with these tools. As a bonus explaining the problem to them may help you solve the issue.
2. Invest in learning new debugging tools ahead of time. Debugging a problem is a really stressful situation, particularly if you are under time pressure, so having your tools and techniques practiced and ready to go beforehand can be helpful here. Athletes and musicians put in hours of practice before they actually perform, I think Software Developers could take a lesson from them.

Sometimes when you’re working with Clojure maps you want to get both the key and the value together. To do this, use find. find returns a clojure.lang.MapEntry which looks and smells like a two element vector, and can be manipulated like one.

(find {:key :val} :key)
;; => [:key :val]

(type (find {:key :val} :key))
;; => clojure.lang.MapEntry