Daniel Compton

In December, I tried to set up ClojureScript testing with doo and it was going surprisingly well. Everything was running well locally, so I setup a circle.yml file, and pushed my changes up to CircleCI. They ran there on the first time as well, more surprise! CircleCI have clearly put a lot of energy into getting a good setup for browser testing. I was very pleased, until after a few builds, I started running out of memory while running the tests. Circle CI only allows you 4GB which should be plenty for most applications, and certainly for compiling ClojureScript. I tried rerunning the build, sometimes it would complete successfully, and other times it would hit the RAM limit. I tested it on my machine and the tests ran fine, and only used ~ 2GB RAM.

I wasn’t quite sure what was going on so I gave it a break for a few hours. When I came back I ssh’d onto the build machine and ran top while the test was running. I could see my process running and taking more and more memory, getting closer and closer to 4GB. Then the tests finished and the memory was reclaimed. While I was running top, I saw Mem: 251902904k total and thought that was curious. It looked like it was saying I had 240GB, but I thought I was only allowed 4?

This triggered a thought, if the JVM saw that it had 240 GB of memory, then it might easily blow past 4GB of memory while running a ClojureScript build. I checked the project.clj file and saw that it didn’t have any :jvm-opts set in it. I checked the default JVM memory options on the build server:

$ java -XX:+PrintFlagsFinal -version | grep HeapSize 
uintx ErgoHeapSizeLimit = 0 {product} 
uintx HeapSizePerGCThread = 87241520 {product} 
uintx InitialHeapSize := 2147483648 {product} 
uintx LargePageHeapSizeThreshold = 134217728 {product} 
uintx MaxHeapSize := 32126271488 {product}

2147483648 bytes for intial heap size (xms) converts to 2.15 GB, and 32126271488 bytes for max heap size (xmx) converts to 32.13 GB. Once I saw that, the problem became very clear. The JVM was starting with 2 GB of RAM, and had a max heap size of 32 GB. It had no need to GC anywhere near 4GB, so its memory usage would just keep climbing. Depending on how fast the build ran, my tests were sometimes able to finish before the CI hit the 4 GB limit. I set :jvm-opts ^:replace ["-Xms256m" "-Xmx2g"] and reran the build a few times. It was able to successfully run every time. Success!