I recently was looking at how to get more than one display connected to my new M1 MacBook Air. DisplayLink technology had a lot of limitations but was the only option I could see. I looked at different docks and ended up purchasing the ALogic DX3.
These are my thoughts after a few months use. I’ve split the review into DisplayLink and ALogic DX3 sections. Most of my complaints are about DisplayLink and I think would be the same no matter which dock/USB adapter you choose.
Alogic DX3
The DX3 has 3 DisplayPort ports, but only two of them are DisplayLink compatible. The third one uses Alternate Mode (I think?) over USB-C to pass the DisplayPort signal directly to the monitor. This was good for me as I could make my center display DisplayPort and use DisplayLink on the peripheral display.
The dock comes with a hefty external power supply almost the size and weight of the dock. The dock has a power button on the front which I thought was unnecessary at first but came in handy as I often need to restart the dock for the Mac to detect one or both of the displays. I’ve had this issue on a 2019 Intel Macbook Pro with directly connected DisplayPort cables so I suspect this may be more of a macOS problem. Having the power button on the front is much more convenient than unplugging the displays or turning the displays off and on.
I haven’t had any issues with the USB ports. I haven’t pushed them very hard, but I also haven’t noticed anything “wonky” about them either. I have a 10-port USB hub that I plug into the dock and it seems to work fine including for an audio interface.
DisplayLink
With a 4K display, macOS will let you choose a higher resolution to render at, which it will then downscale to the 4K pixels that are actually on the display. When I first bought the DX3, DisplayLink version 1.4 wouldn’t allow you to do this, the maximum resolution you could pick was 1920x1080 points at 2k (native 4K resolution). When writing this article, I went back to the feature request and found that they had released version 1.5 which was meant to fix this issue. It lets you choose 2560x1440 points but also renders at 2560x1440 pixels, instead of 5120x2880 pixels, downscaled to 4K. The good news is that it looks like they are continuing to work on the software and addressing limitations so hopefully this will be fixed in a future update.
DisplayLink 1.4 was doing something which prevented you from using Apple Watch to unlock your computer. I discovered that it also stopped you from using your Watch as an admin password. However, the 1.5 update has a checkbox to “Use Apple Watch to unlock on login screen”. When you check that, the Watch unlock appears to work at the login screen and in everywhere else Watch unlock would work. I’m not sure why that wouldn’t just be a default setting?
Note that you can’t evaluate DisplayLink CPU usage just by looking at the “DisplayLinkUserAgent” process. You also need to look at the WindowServer process, as DisplayLink causes work to be done there when rendering to the virtual display.
The DisplayLink software could do with more diagnostics on what is happening. Currently you can’t see which displays are controlled by DisplayLink and which are directly connected.
The DisplayLink software doesn’t seem to have any way to automatically update or even notify you of updates so you’ll need to check the website yourself.
I haven’t noticed any latency from the compression/decompression signal transfer, but I’m only using the displays for writing code and web browsing so I wouldn’t expect many problems.
Other
I’ve had a weird bug where sometimes the MacBook display won’t turn on for 10-60 seconds after opening the laptop, even when I’ve been using it only a few minutes prior. I’m not sure if this is related to DisplayLink software or just a macOS bug.
Summary
Overall, the ALogic DX3 dock serves it’s purpose well. If you had any other option, I wouldn’t recommend choosing DisplayLink technology, but with an M1 Mac this your best (and only) option. However, I’m encouraged that DisplayLink are continuing to improve the Mac user experience. When I upgrade to a new MacBook Pro, I’ll either use the DX3 without DisplayLink (if that will work), or sell it and purchase a standard Thunderbolt 4 dock.
I needed to buy a new Mac recently. I couldn’t bring myself to buy a legacy Intel Mac, but the new M1 Macs only support up to two displays. For the MacBook Air, MacBook Pro, and iMac that means you can connect one additional display along with the built-in display. For the Mac Mini, you can add two displays. I wanted a MacBook Air and already had two external displays which I wanted to keep using.
Update: I ended up purchasing the ALogic DX3 and wrote a follow-up post about my experiences with it.
DisplayLink
What are your options if you want to run more than two displays? Enter DisplayLink. DisplayLink (not to be confused with DisplayPort) is the name of a technology created by a company also named DisplayLink. It lets you send a video signal to a display over USB or Wi-Fi instead of via DisplayPort or HDMI. DisplayLink technology was first used in laptop docking stations but can now be found in other video-related products including adapter cables and monitors.
DisplayLink has two components - a software driver installed on your computer and a hardware chip in the dock or adapter. The software driver presents itself as one or more displays to the computer. The computer sends pixel data to the software driver, which then compresses the data and sends it over USB. The DisplayLink chip decompresses the data and sends the display signal to the (real) display.
I was pleasantly surprised to see that DisplayLink’s drivers have been updated for Apple Silicon and don’t require a kernel extension. From what I can tell, DisplayLink seems to be publishing regular updates to their drivers and is actively working on new features.
DisplayLink Limitations
Sending compressed video over USB is a pretty neat trick! However, DisplayLink has quite a few downsides to be aware of.
An uncompressed 4K60 video signal requires 12 Gbps of bandwidth. The latest DisplayLink chips run over USB 3.0 which has a bandwidth of 5 Gbps. Even a single 4K signal is too big, let alone multiple displays. This is why DisplayLink needs to compress the captured screen.
For desktop browsing, email, software development, and other general computer tasks where the display doesn’t change much from frame to frame, DisplayLink compression can be very efficient and is unlikely to be noticeable. DisplayLink claims “Pixel-perfect graphics”, and “Compression tuned for video content and high-quality graphics” on their newer chipsets. If you’re a video/photo/graphics professional, you’d probably want to check and see if this works for you.
It doesn’t sound like it will be as good for gaming though:
While DisplayLink technology is targeted primarily to productivity and video applications, it is suitable for the casual gamer. If you’re a “power gamer”, looking for every edge possible over your opponents, you might want to go another route.
Another downside to DisplayLink is that you need to install an optional program to show the Lock Screen on a DisplayLink monitor. Otherwise, the Lock Screen will only show up on the built-in display.
Unlock with Apple Watch doesn’t work when using DisplayLink. Apple disables Unlock with Apple Watch with the message “Unlocking with Apple Watch is not available while your screen is being shared”.
DisplayLink has a few other limitations on Macs: HDCP is not supported so you may not be able to watch Netflix, Hulu, iTunes Movies, etc.; only 4 external displays are supported; screen rotation is not supported; Clamshell mode is partially supported. Manufacturers publish blog posts announcing that new DisplayLink drivers are validated with their hardware, it’s probably good to check with your manufacturer before updating.
There are quite a few limitations to using DisplayLink to extend your displays. However, it’s also currently the only way to run more than two displays on a Mac, so if you need that then you’ll need to look at a DisplayLink powered Dock.
Choosing a Dock
If you already have a dock without DisplayLink, your cheapest option may be to purchase a DisplayLink USB adapter. You could connect your monitor to the adapter, and then the adapter to a free USB port. DisplayLink keeps a list of USB adapters you can purchase. Check compatibility as not all adapters list support for macOS, although sometimes the reviews will mention that they do still work.
DisplayLink sells chipsets to dock manufacturers. The chipset has their DisplayLink hardware, along with other ports like USB, Ethernet, Audio. This means that products from different manufacturers like Targus, Dell, Alogic, Plugable, Kensington can end up with very similar capabilities because they are all using the same electronics.
Unfortunately, all of the M1 compatible docks I saw had a maximum of one USB-C port. If you have a lot of USB-C devices you’ll either need to purchase USB-A->USB-C adapters or buy a non-DisplayLink dock and a DisplayLink adapter.
If you’re investigating docks, you may see reference to USB-C Alternate Mode. This is a way for a USB-C port to run non-USB signals over it like DisplayPort and Thunderbolt(!). Alternate Mode doesn’t let you run any more displays than you could otherwise do natively.
Dock Options
If you’re looking for a list of DisplayLink docks, DisplayLink has a list of DisplayLink docks. Here’s a list of docks that the manufacturers have said are supported on M1 Macs as a starting point:
UD-3900PDZ, $160 - 3 HDMI ports, 1 supports 4K30, the other 2 only support 1080p/60Hz. 60W Power Delivery, 6 USB-A 3.0 ports, 3.5mm audio jack.
UD-ULTC4k, $220 on Amazon - 2 DisplayPort ports that support 4K60 over DisplayLink, 1 HDMI port that supports 4K30. 60W Power Delivery, 4 USB-A 3.0 ports, 1 downstream USB Type-C port, 3.5mm audio jack.
Note that Plugable mentioned that their Ethernet ports are limited to 300Mbps on macOS due to driver issues. I’m unsure if that extends to all DisplayLink docks.
DOCK190, $310 - 2 DisplayPort/HDMI ports (only 2 video ports can be used at the same time), 100W Power Delivery, 4 USB-A, 1 USB-C, Gigabit Ethernet, 3.5mm audio jack. Thunderbolt alt mode is supported.
DOCK570, $415 - 4 DisplayPort/HDMI Ports, 4 USB-A, 1 USB-C, Gigabit Ethernet, 3.5mm audio jack. Thunderbolt alt mode is supported.
Dell has some docks with DisplayLink technology, but they don’t list compatibility with Macs, so probably safest to avoid them unless you’re able to try them first.
Caldigit docks are highly regarded but don’t contain DisplayLink chips. This means you can’t run multiple displays on them. However, they seem to work if you only need a single external display on an M1 Mac, or purchase a USB display adapter. There is a video of someone using a Caldigit TS3 Thunderbolt dock with 4 DisplayLink USB adapters.
Disclaimer
I’m not an expert on any of this, I’ve just read a lot and compiled it into one place. Please let me know if I’ve missed something or if you have any suggestions.
Summary
There are lots of moving parts involved with picking a USB-C dock, let alone one also running DisplayLink. You need one which:
Supports DisplayLink.
Supports the number of displays you have, at the resolution and refresh rate you want.
Has enough USB ports to support other peripherals, or is reliable enough to add a USB hub.
Supports USB Power Delivery with your laptop, with enough wattage.
Has enough bandwidth for all your devices.
Doesn’t have audible fan noise.
Works reliably and doesn’t cause other issues.
As you can tell from the number of limitations mentioned throughout this post, picking a dock that works with your setup is far from a simple task. I recommend checking the specs closely, checking out reviews, and ideally trying it out at home before committing.
Good morning. I was going to ask if everyone’s feeling awake, but I think after that karaoke, I think everyone’s pretty well awake by now.
This talk is called Building Stable Foundations, and it’s about building stable long lasting foundations for the Clojure community.
[00:00:30] Bozhidar has already introduced me slightly, but I’ll go back over it. My name’s Daniel Compton, I’m an open-source project maintainer. I am an administrator at Clojars. I write and record a Clojure podcast and newsletter at therepl.net, and I am working at a startup called Falcon.
Falcon is hiring for Clojure developers, remote Clojure developers. If you’re sort of in the U.S.-ish region, or you can work [00:01:00] on a similar kind of time zone, check out falconproj.com [Ed: Falcon shut down 😢] or come and talk to me, and I can tell you more about what we’re working on.
The other thing that you may know me from is an organization called Clojurists Together. With the help of our members, we fund and support open-source Clojure projects. The question is, well, why is that important? Firstly, who has heard of Clojurists Together?
Great. All right. Just about everyone, and who [00:01:30] here is a member or your company is a member of Clojurists Together. Great. The reason why Clojurists Together is important, or I think it’s important, is that Clojure is foundational, and open-source Clojure is foundational to what we work on. Of course, the Clojure language is open-source, but the vast majority of tools, and libraries, and services, that we all use as day-to-day Clojure programmers come from open-source software.
It’s important, not just for [00:02:00] us as a community, but open-source Clojure is really foundational for businesses too. Businesses are investing lots and lots of money into the Clojure ecosystem, and they’re relying on these investments lasting a long amount of time. I think it’s really important to sort of remember that context of the money being invested into Clojure and making sure that that is well-spent and it’s going to be [00:02:30] a long lasting investment.
Who here is lucky enough to get paid to work with Clojure? All right. Of those of you who had your hands up, who uses open-source Clojure code in your day job? It’s a bit of a silly question, isn’t it? That the idea that as a working Clojure programmer you wouldn’t be using a significant amount of open-source code as part of your work. Maybe there are some people. I suspect there are a couple, but for most of us a lot of the work we do is built on top of these really important open-source foundations. Open-source creates an incredible amount of value for everybody that gets to use it. It’s the shared resource that doesn’t run out when somebody uses it, that doesn’t mean that anyone else can use less of it. This is a really amazing property of software and open-source software.
But a few years ago, I started to notice [00:03:30] developers talking about burning out. In the Clojure community and also outside of the Clojure community, and this was kind of worrying to me, because a lot of these people were really important to the Clojure community. They were doing really important work.
I started to see these kinds of things being said. People saying things like, “Any tips for a post burnout return to open-source? I’m exhausted, but want to keep going.”
This [00:04:00] really broke my heart to see these people who I cared about, many friends and that they were not feeling supported. They were thinking of leaving, and I was really worried about this. We have this problem that the foundations of our community are perhaps not as strong as they could be, or they’re not being as supported as they could be.
Before we talk more about our open-source foundations, I think we can learn some things from building [00:04:30] foundations. This is what a building site looks like before you start building on it itself. The first thing you have to do when you’re building is to prepare these foundations, everything else goes on top of them. So you have to get this right first.
When you’re building foundations, there’s a few things you need to do. The first thing you need to do is survey the land. You need to find out, do you have the rights to work on this land that you think you own and is the structure that you’re going to [00:05:00] be building allowed to be built on this land. This is really important to get right, because if you get it wrong, you might invest all of this money into building something which you then need to take back down again.
Next thing you need to do is check under the ground. What is going on underneath the surface? You can see what’s on top of the ground, but it’s really important to find out what’s going on underneath there before you start building, so that you don’t end up being surprised by things later on. The last thing, and this is the most important thing and [00:05:30] what we most often think of when we’re thinking of building foundations is reinforcing the ground. A fully built house can weigh 50 to a hundred tons or more, and that’s a lot of weight to be putting into the ground, and so you really want to make sure that those foundations are going to be able to hold up to that kind of weight.
This is quite similar to what we do when we’re picking dependencies. The first thing we do is we’ve got to check the license. I hope we check the license and [00:06:00] we also will usually want to review the codebase. Is it well-written Clojure? Does it have tests? Is it actively maintained? These are all important things to be aware of before you pick a library, and the last thing, if this is going to be used in the critical path of your production software, you probably want to test it, find out how fast is it. Can it stand up to the full weight of production? Are there memory leaks, that sort of thing?
Foundations are really important [00:06:30] to building a house, but it turns out that they’re actually pretty cheap. In the U.S. on a $400,000 house build. They cost about $10,000, two and a half percent, so not really that much of the cost of the build. This is because it’s a very well understood process. It’s been done many times every day, and for the most part, it goes pretty smoothly.
Has anyone ever seen a house like this before?
[00:07:00] What’s happening here is that the foundation on this house wasn’t built correctly the first time. It’s a little bit hard to tell, but this house has been lifted right up off the ground, quite a few meters up. Stacked up on these Jenga blocks so they could fit machinery underneath to repair the foundations, and then they’re going to slowly lower that house back down.
If you look closely, I’m not sure if it’s going to come through very well here, but there are some people standing underneath this house, [00:07:30] very brave people. Does anyone relate to this picture? Has anyone had to fix the foundational dependency in your codebase and feel like there’s a lot of things bearing down on top of you? As you can imagine, this is not a cheap process. It can cost $20,000 to $100,000 to do this kind of repair, which is two to 10 times more than the cost of building the foundations correctly in the first place
[00:08:00] Yesterday, Tiago talked to us about resilience, and I’m glad he really dug into this, because I don’t need to go too deeply, but we’ll remember resilience as the ability to recover from a stress or a change. This is what you’re going for when you’re building foundations, you want the foundations to be able to stand up to the elements and all of the things that a house is going to face over a hundred year lifetime. What we can see here is there’s two houses, on the left… If you look at them, superficially they kind of look the same. They’re [00:08:30] both held up by wooden poles. They’ve both got thatch kind of roofs. They’re both orangy. They’re both roughly the same size, but the one on the left is not very resilient at all, and the one on the right is incredibly resilient, even though it’s over the water.
This house on the right is, well, it’s a community. Firstly, it’s a community of houses. It’s not just a single house, and it’s also held up by many stilts, so that even if you lost one, or two or even a dozen of these stilts, [00:09:00] the whole structure would still be sound. Whereas on the left, you’ve got just two poles holding this house up. And if one of those two goes down, the whole thing is coming down. If you agree with me that open-source software is the foundation of the Clojure community, then we need to be asking this question, how resilient are our foundations? A few years ago, as I was looking around the Clojure community, it seemed to me that we were perhaps closer to the left-hand side than [00:09:30] the right-hand side.
We had a few people doing a tremendous amount of work for the Clojure community. People were depending on their work every day, they were using it. It was really important, but they were feeling unappreciated, perhaps burnt out, and there was this refrain of, “I’m exhausted, but want to keep going.” I thought they were at severe risk of burning out. If we lost these people, this would be somewhat equivalent to that house we saw, having to be lifted up on the blocks, [00:10:00] and there would be a huge investment. We would lose a huge amount if we lost some of these people, and it would cost us a lot of time and effort to regain that knowledge and hard won wisdom.
This led me and others to create this organization called Clojurists Together. We have a very simple mission. We want to fund and support open-source software, infrastructure, and documentation that’s important to the Clojure and Clojure script community.
We [00:10:30] fund three kinds of projects. The first kind of project is maintenance projects, and these are projects that are often not very interesting to fund. Maybe they’re very stable, used by lots of people, but over time, software needs maintenance, and if people aren’t being paid, or aren’t getting dedicated time to work on it these maintenance things can pile up over time. So this is a place where I think Clojurists Together is really well suited to help these projects, [00:11:00] because it’s work that sometimes wouldn’t get done otherwise, or it would take a lot longer. We really liked being able to fund these kinds of low-level boring kind of work, but these are really important improvements to make.
The next kind of work we like to fund is new development. This might be a project that is already well understood and used by many people, and they have an idea for a new feature, or a new release, or something that they would like to do, and they need some dedicated [00:11:30] time to work on it. We also take and fund projects who do this kind of work.
The last kind of project is new fledgling of projects, and we fund a few of these kinds of projects where someone comes to us with a seed of an idea, and perhaps they’ve already created proved out that this is an idea, but they would like some funding to go to the next level with it, and we’ve been able to fund some of these projects, and they have [00:12:00] really surpassed my expectations. You’ve done amazing work, so this is another kind of project we like to fund.
We launched in October 2017, and in 2018 and 2019, we’ve funded 11 projects.
We’ve funded Bozhidar on CIDER.
We funded Tim Pope to work on Fireplace, which is a Vim Clojure plugin.
We funded Arne Brasseur, also here, to work on Kaocha, which is a next generation test runner for Clojure.
We funded Mike Fikes to work [00:12:30] on ClojureScript and the ClojureScript compiler.
We funded Nikita Prokotov to work on Datascript.
We funded Thomas Heller to work on Shadow CLJS, which is a ClojureScript and JavaScript build tool.
We funded Lee Hinman to work on clj-http. This is a HTTP client that’s used by many projects directly. I think it’s one of the most downloaded projects on Clojars, but it’s also used as the foundation for many of these other API clients. You may not even be aware that you’re using it, [00:13:00] but improvements to clj-http benefit a large part of the Clojure community.
We also funded Aleph. This is a Netty based web server for Clojure.
We funded Dragan Djuric to work on Neanderthal, and then he worked on documentation for Neanderthal and wrote 150 pages of documentation about deep learning and Clojure from scratch. It’s a really interesting read if you’ve been wanting to understand deep learning. [00:13:30] I’d really recommend checking that out. It’s really good.
We also funded Martin Klepsch to work on cljdoc. Cljdoc automatically builds the documentation for you every time you push to Clojars. This is really nice, because it means that the maintainer of the project doesn’t need to remember to update the API documentation every time they do a deploy. It just happens automatically, and you can look back at older versions of the API documentation, and there’s all sorts of other good things that have come from [00:14:00] having a shared platform that we can all use to improve.
The last project we funded was Bruce Hauman to work on Figwheel. Figwheel is a really key tool for the ClojureScript community. It’s had a lot of influence outside of the Clojure community as well. It’s been cited by many other environments. I’ve seen its influence, and the influence of things that came before it too, spreading across the entire community. Live reloading is one of ClojureScript’s [00:14:30] secret weapons, or perhaps not so secret anymore.
Who in the audience has used one of these projects? All right. What about three projects. Who’s used three or more of these projects? Wow. Okay. Who’s used five or more of these projects? Great.
All right. Is there anybody here who has not [00:15:00] used a single one of these projects?
All right. Well, that’s really good to see. [Nobody put their hand up]. I’m glad that this has benefited people.
I’m really excited to be able to announce today, this Heart of Clojure conference has been timed really well to be able to announce on stage the next funding round. We’re going to be funding four projects, $9,000 over the next three months.
The first project we are funding [00:15:30] is Calva. Calva is a VS Code extension for Clojure. VS Code has been gaining in popularity hugely over the last few years. I think it’s one of the sort of surprise sleeper hits of the programming community, so we’re really excited to be able to fund Calva.
We’re also funding Thomas Heller to work on Shadow CLJS again. He’s been doing incredible work for the Clojure and ClojureScript community on Shadow CLJS. I hear lots of people talking about it and using [00:16:00] it, and it’s a really great tool, and those improvements are benefiting lots of people.
Another project we are funding is Meander. Meander is a really interesting Clojure, ClojureScript, data transformation library. I probably can’t do it justice here, but Michiel Borkent [Ed: this was Timothy Pratley, sorry Michiel!] wrote a really great blog post about this, so if you find him here at the conference, ask him to tell you more about Meander, because it’s a really cool project, transforming data and maps, which [00:16:30] we do every day.
The last project is CIDER. I would say CIDER needs no introduction as well, because many, many people use CIDER and many more people also use Bozhidar’s work on the Orchard and in REPL and all of these other foundational things that sit beneath CIDER that we as a community get to take advantage of. I’m really excited to be able to fund these four projects, thanks to the [00:17:00] support of our members.
The way Clojurists Together works is that we have members sign up throughout the year, and then every quarter we will go to them and say, “Hey, what do you think we should be working on? What’s interesting to you, what do you think is useful? What areas do you think need support?” Then we take that information and we create a call for proposals. We say to the wider Clojure community, “Hey, here’s what our members would like us to fund. Please [00:17:30] give us some proposals,” and every quarter we get an amazing amount of proposals, more than we could pick. The quality of these submissions is really high. Then the committee members vote on the projects that we want to fund, and then we’ll fund those projects for three months. At the end of those three months, we turn around and do it all again, so it’s pretty straightforward.
Clojurists Together has been getting noticed and has influenced beyond [00:18:00] just the Clojure community. This is my friend, Devon, who works at GitHub, and she’s talking at GitHub Satellite earlier this year about GitHub Sponsors. Clojurists Together was mentioned as an influence on GitHub Sponsors. GitHub Sponsors is a project for where you can fund directly the projects that you use on GitHub. I’m really excited to see this. I think GitHub has a really wide reach, clearly, in the development community, and they’re helping to [00:18:30] change attitudes around open-source funding for both, in the Clojure community, but also, in the wider community. This is something I’m really looking forward to.
Devon has this quote, “Our goal is for open-source to be a serious career path people can set up.”
This is kind of the, I think, the next frontier for the Clojure community too. That we have a bunch of projects that are really important, that people are using, and I would really like to see [00:19:00] people be able to be funded to work full-time or nearly full-time on these projects for the good of the community. Clojurists Together, can’t do it all. We’re set up to do a particular kind of funding, and so what I’d like to see is that the community as a whole starts to build this open-source middle class of single or small teams being funded in a meaningful way, and being appreciated by the community so that they can do this [00:19:30] work that benefits everybody.
Part of this is that we’ve just launched a page on our site called Beyond Clojurists Together, where we are collecting Clojure projects and Clojure programmers who are accepting money from Patreon, Open Collective, GitHub Sponsors, or whatever other tools they have, to be funded directly. If you’re interested in funding projects and you don’t really know where to start, this could be a really good place for you to go start looking and finding [00:20:00] some of the dependencies that maybe you or your company is using to fund them directly.
How do we get there? How do we get to this glorious future where all of our dependencies are supported and maintained? It needs two things, time and money. Sometimes people and companies have capabilities for one or the other. We have a lot of people investing, people in companies too, investing in open-source [00:20:30] projects, open-sourcing internal things or maintaining projects. Other times companies don’t really have the time or the capacity to be directly working on open-source projects, but they have some money available that they could put into maintaining them instead.
The other side of the equation is money, and companies spend money on lots of things. They spend it on our salaries, software licenses, cloud hosting, hopefully [00:21:00] green cloud hosting after that talk yesterday, and offices and snacks.
Companies are used to spending money on lots of things, but I think open-source projects often aren’t shaped very well for companies to give money to, and companies need these two things. They need value. They need to be able to say to the decision-makers, to the finance department, “We are getting something of value when we are giving money to this project.” That doesn’t mean [00:21:30] that you need to sell out to the man. There’s lots of different ways that you can provide value to projects, to companies that are both valuable to the companies, that fit with the skills and capacity of what you have, but most importantly, still meet the values that you hold as a person and the values of your project.
This could be things like consulting, feature development, long-term support, maintenance, [00:22:00] and these are things that oftentimes you may be doing anyway, but if you’re able to sell it, and sell this to companies in a way that they are able to understand, and sell onto their decision holders, this can be a really powerful thing.
The other thing that companies need is invoices, and not just invoices, but all of the other financial infrastructure that goes along with things. Companies are not used to using PayPal [00:22:30] donation links to pay for their office rent. They pay money to a bank account, and so this is something where open-source projects, again, aren’t often shaped very well to accept this money.
This is where I’m really excited to see platforms like Open Collective, Patreon, GitHub Sponsors who are providing the shared common infrastructure for projects to be able to work, to be able to provide the kinds of things that companies [00:23:00] need to be able to give money to.
The question now is, are we investing in stable foundations? Are we building stable, resilient foundations that are going to last us the test of time? To answer that question, we need to sort of answer, well, compared to what? The main input to open-source projects is labor, people spending their time on it. I thought perhaps a good comparison to [00:23:30] this will be, how much are we spending on Clojure developer salaries and businesses. To answer that we need two numbers. We need how many Clojures developers there are, and how much they’re getting paid.
Estimates vary from perhaps 20,000 to 50,000 working Clojure developers. We want to be conservative here, so we will just take 20,000, and then we need to figure out how much they’re getting paid. The Stack Overflow Survey for 2019, put the average Clojure developer salary, worldwide, [00:24:00] at 90,000 U.S. dollars. Clojure’s been at the top for the last three years, which is pretty good, but I’ve heard people quibble with this number, say that maybe it’s a bit inflated or unrepresentative in some way, so let’s just take it back a bit more to still be really conservative, and we would just say $80,000 a year.
If you multiply these two numbers together, you end up with $1.6 billion a year being spent on Clojure developers.
This doesn’t even include all [00:24:30] of the other expenses that go around Clojure that you need to run a business, and it doesn’t even include the full cost of hiring an employee, but I think this is a really good number to compare to and to think about.
If we were to spend a relatively small percentage of what we spend on Clojure developers, on supporting the foundations, that could be a meaningful amount of money, say two and a half percent of 1.6 billion would be $ [00:25:00] 40 million a year. I think, I can’t even really imagine quite what that would look like. It’s just such a long way away from where we are currently that it would… Yeah, I can’t imagine quite that what that would look like, but what I’ve seen from funding projects, even at a very small level far below this, is that the return on investment there has been incredible.
I think we would see some really exciting things start to happen, even more than they are currently, [00:25:30] if we were to be able to even approach this level of funding, which itself is, I think, very small compared to the value that we as a community get from these open-source foundations.
We come back to this image of the houses on the water, and this image really appeals to me, because I see some similarities here with the Clojure community. These are not always the most pretty houses, some of [00:26:00] those stilts down below look a little bit rickety, but together as a whole, these come together to form a really solid stable foundation. This is what I’m hoping that we can build together for the Clojure community, and to be clear, this is already happening, well before Clojurists Together. This is still a really strong community before this, and I’m excited to see kind of where we can go in the future.
If this idea appeals to you, here’s some things that you could [00:26:30] do.
First thing is support your dependencies. You can go have a look at that page on Clojurists Together, Beyond Clojurists Together, where you can find some projects that you are using, which you might not even be aware. We’re accepting donations.
Another thing you can do is commit engineering time to maintaining the dependencies that you work on. This is a really key thing to do, and I know many companies already do this, so I really want to recognize that and thank them for their work they’re already doing here.
Another interesting [00:27:00] thing you can do is ask when you are next looking for work, ask the companies that you’re talking to, do you support in some way, the open-source dependencies that you use? I think this is a really interesting question, because sort of below just your funding open-source, I think the primary barrier to that is changing mindset and expectations around what [00:27:30] it means to use open-source, and the potential benefits for the community as a whole to fund it. If you ask companies, when you’re looking for work, “Do you do this?” Companies are looking to stand out to you, and so I think we might start to see the answer more and more become, “Yes.”
Another thing you can do is stand for Clojurists Together elections. We have a board of seven people and every year in November, we have elections, so if you would like to be directly involved in helping us plan where we [00:28:00] go into the future you can stand for these elections.
The last thing you can do is join Clojurists Together. I see many people already in the room are Clojurists Together members, and I really appreciate that. You can sign up as a company, or a developer, as an individual. The choice is yours there.
I want to acknowledge many people here. Firstly, Heart of Clojure, Ana and Martin, they’ve put on a really great conference here, so I really appreciate them bringing me along to speak.
[00:28:30] I’d also like to thank Ruby Together, and André Arko in particular. Ruby Together is the organization that we modeled ourselves afterwards. They’ve done a lot of trailblazing work here and André Arko, in particular, has given us a lot of time so that we can learn from some of the things that they’ve done.
I’d also like to thank the Software Freedom Conservancy. I saw someone here had a [00:29:00] Software Freedom Conservancy shirt on yesterday. The Software Freedom Conservancy, if you’re not aware is our non-profit parent. They handle all of the accounting, and payments, and financial, and legal things that go on that companies need, and that we need to have happen, but that we can rely on them to provide for us. They’ve done an amazing amount of work for us, and so I really appreciate them.
Next group of people I’d like to think is [00:29:30] the projects that have applied. We have more projects apply every quarter than we could fund, and as I read over the applications, I’m really excited, because there’s so many good projects out there, and so I’d like to thank everyone who has applied.
I’d also like to thank the projects that we funded. Sometimes today in the talk, I might’ve slipped and said that we did this work and I wasn’t programming on CIDER, Bozhidar was, so I’d like to thank the projects that we funded, that have done this work. I know that often they’ve taken [00:30:00] time off paid work, or taken holiday time. While we’ve been able to fund them to a certain level, I know often that still means a pay cut for them to be able to do this work, so I really appreciate that.
On the left we have Maria Geller, Daniel Solano Gómez, Larry Staton Jr., Nola Stowe, Fumiko Hanreich, and Laurens Van Houtven. These are all current board members. We also have [00:30:30] board members from the past, Bridget Hillyer, Toby Crawley, Devin Walters, and Rachel Magruder. [inaudible 00:30:36]is our admin assistant. If you’re a Clojurists Together member, you might’ve got some stickers from us recently, and if you’re wondering, why did this come from Spain? It’s because Rachel, our admin assistant lives in Spain.
I’d also like to thank our members. We’ve seen many of these names around the conference already. Pitch, Nubank, JUXT, Metosin, Adgoji, and Funding Circle. [00:31:00] These companies have all been a huge help to Clojurists Together, and have funded us in a really big way that have let us do all of the work that we’ve been able to do.
Also, I’d like to think these other company members, I don’t have time to go over all of their names, but many of these names are going to be familiar to you, and some of them are at the conference as well.
This is probably my favorite slide at the talk, because this is the names of the 200 members, developer members in the Clojure, [00:31:30] Clojurists Together that are members of Clojurists Together. Yeah, there’s far too many here to name them all. I tried to make the text bigger, so that you could even read the names, but then it just went on for slides, and slides, and slides.
I can’t name them all, but I’d really like to say thanks to the 204 developer members, the 34 company members, and also to the rest of the Clojure community.
The Clojure community is a [00:32:00] really special place. It’s warm. It’s very small, and I think we punch above our weight in terms of the impact that we can have, in terms of the tooling and libraries that we create. That’s in large part due to the contributions of all of the Clojure community coming together to build things for everybody. I think a good chunk of the Clojurists Together members are here at this conference, and so again, I especially want to say thanks to you for your support.
I’m thriled to announce that I have finally attained the coveted blue GitHub “Staff” badge!
I’m joining the Social Coding team at GitHub as a product manager. For the last six years I’ve been writing Clojure professionally; product management is going to be quite different, but I’m looking forward to it. There are a few reasons why I’m looking forward to working at GitHub:
Over the last few years working on Clojurists Together I’ve been looking at how to fund open source software that is in the community good. One of the best parts of the role has been talking to open source maintainers to help design funding programs that serve their needs, and the needs of the community. In my new role I’ll get to talk to even more open source maintainers, contributors, and users about their experiences with open source and GitHub.
GitHub is a remote friendly company and the majority of their employees were remote pre-COVID. I’ve only worked in small remote teams and I’m looking forward to seeing how remote work scales to a large organisation.
GitHub has been on a tear in the last few years, releasing many features and products. I’m excited to experience this from the inside.
Im also looking forward to be working for Devon Zuegel. I met Devon 18 months ago when she was doing research for GitHub Sponsors, and I’ve been impressed with the work she and the rest of the GitHub Sponsors team has done since then.
GitHub is the worlds most important social network for developers, and I’m thankful for the opportunity to help build it.
This tax season I got an email from IRD reminding me I had to file my return. This happens every year and is usually not a surprise. However this year, the returns were slightly different. They said:
The 31 March 2019 IIT return for {person} is due 8 July 2019.
and
The 31 March 2019 ITN return for {company} is due 8 July 2019.
I’d never heard of an IIT return or an ITN return. I searched around and couldn’t get any results on Google or IRD’s website search. However, my best guess at the acronyms is:
IIT Return: IR3 - Individual Income Tax Return
ITN Return: IR4 - can’t think of a good explanation for this acronym
I think IRD has renamed these in their recent system upgrades and I suspect they’ve renamed all of their returns. If you come across any other acronyms, let me know and I’ll add them here to help others out.
I’ve been working remotely for about five years full-time. Over that time I’ve talked with colleagues pretty much every work day, so being able to communicate clearly by audio has been crucial. Bad quality audio can quickly turn a good conversation into a frustrating one when you struggle to hear the speaker, or keep needing to ask them to repeat themselves. Videoconferencing is lower bandwidth than in-person communication; I’ve tried to get as good quality when videoconferencing so I can catch as much of what the other person is communicating.
I studied as a musician, and accumulated several pieces of audio gear and knowledge about working with audio which have been very useful for working remotely. A friend asked me for advice about getting some audio gear so I sent him an email. That turned into a post on our company wiki, and now I’m posting it here publicly.
In my experience, for remote videoconferencing to work well, the following things are very beneficial:
A fast, stable, and reliable internet connection
A quiet space free of distractions, and without too much echo
A space you’re able to comfortably talk in during work hours
A good microphone close to your mouth and a set of headphones - Hearing yourself echo on someone else’s call makes it hard to have a conversation.
Improving audio quality when video conferencing
Here are some tips in rough order of importance for improving audio quality when videoconferencing. You can keep going down the list with diminishing returns of audio quality, stop whenever you and/or your team is happy with the quality you’re getting.
Get really close to the mic. It should be no more than an inch or two from your lips. Having a mic far away from your mouth makes everything else more difficult. A stand on a desk is better if it goes up to your face, having the mic at desk level will pick up lots of other noises and echoes.
Plug your computer in with Ethernet, or make sure you have really strong Wi-Fi. I always prefer to use a cable to remove a potential weak link.
Find a quiet room to talk in. If you can’t find a quiet room, or you are on a call with more than 3 participants, I’d recommend muting yourself when you’re not talking.
Find a room that doesn’t echo. Clapping your hands will help you spot echo quickly. You especially want to avoid sitting in any spot where you hear ringing after a clap. Blankets, couches, other soft surfaces help deaden echo.
Get a decent mic. You should be able to get something fine for $70-$200 USD. Your first choice is whether to use an XLR mic + USB audio interface or a USB mic. An XLR mic will last a long time, especially in office use. My mic is about 10 years old now and still running fine. I’d personally steer clear of mics with a built in USB connection. Because they combine active electronics with the mic, the electronics may break or become obsolete even when the mic is still good. However, they don’t require an extra audio interface which is one less moving part and they will still sound fine. Getting a USB mic wouldn’t be a ‘wrong’ decision, this comes down to your comfort with audio gear and how much stuff you want on your desk. Marco Arment has a good overview of mics. Microphones have different polar patterns which control where they pick sound up from. If you want to be able to play meeting audio through speakers instead of wearing headphones, then look for a mic with a cardioid polar pattern to only pickup sound from the front of the mic. Dynamic microphones are often in a cardiod pattern. Condenser microphones are often bidirectional and pick up equally well front and back, which will not work well when combined with speakers.
If you have an audio interface or USB mic, watch that you’ve set the input levels correctly. If the gain is set too high, you will get terrible sounding digital clipping. If your interface has lights, watch them while talking loudly and make sure they don’t turn red. You can also look at the input levels on your computer and make sure they aren’t hitting the very top of the input scale as this may also indicate clipping. When in doubt, set the gain levels a little lower; video conferencing software usually has some gain leveling built-in to match you with the other speakers.
Get a pop filter, this takes the plosives out of your speech, a mic picks these up much more than a person does.
If you don’t get a USB mic you’ll need a separate audio interface. https://marco.org/podcasting-microphones#interfaces has some suggestions. Anything over $80 USD will be fine. You only need a 1 input audio interface, but often 2 input interfaces will be about the same price. Look for one with a USB connection, you don’t need Thunderbolt for single channel recording. USB-C and/or USB 3.0 would be ideal for longevity, but they cost a little bit more, and USB 2.0 is just fine for a single recording channel.
Once you’ve got echo in your room under control, you can look at treating it with acoustic panels to “deaden” the room further. You can make these yourself, or purchase them from lots of places online. My office has a wooden floor, and drywall roof and walls which created lots of unpleasant echoes. The person I bought my foam panels off suggested I target ceiling/floor reflections first, I mounted four panels on my roof, and this was enough for me. Acoustic treatment is a whole science in itself and is very room dependendent, so do some research for your space.
If you’re not able to avoid echoey or noisy room environments, then you can look at processing your sound with noise removal software. I have used SoundSource (disclosure: a friend made it) and SPL De-Verb Plus to remove room noise on the other end of calls which turned them from painful to tolerable. I also tried Krisp but didn’t get great results from it. Your mileage may vary though.
A few weeks ago I got to attend and speak at Heart of Clojure. I met lots of online friends in person for the first time, and made some new ones too. I’ve thought a lot about how to describe it since then, and every time I come back to the word special.
Here were some things that I think made Heart of Clojure so special. If you’re running a conference, consider stealing some of these ideas.
There was an site which listed activities around the event. Anyone could host an activity and others could join. Having it on a website meant everyone knew what was happening around the conference. I felt like this helped Heart of Clojure feel very welcoming to all of the conference goers, especially those who were less outgoing.
One of the nights I joined an activity for an adventurous dinner. People got split into groups of 6-8 with a reservation at a restaraunt in Leuven. This broke up friend groups and introduced me to a bunch of new people. The restaurant we went to had their own brewery, and I got to try the flavours of a beer soup, beer stew, and beer crème brûlée!
Yulia Startsev’s talk To loosen up, to put together set the tone for the conference and captured the spirit of mixing technology and the humanities. Many of the talks and conversations referenced her talk throughout the rest of the event.
Leuven was a perfect town to have the conference. It was large enough to have a good nightlife, but small enough that you could walk between the conference venue, the bars, restauruants, and hotels in 10-15 minutes.
In the months before the conference, Jan Stępień paired up new speakers with more experienced speakers. This was my first conference talk, and I got paired up with Tiago Luchini. Tiago was a very experienced public speaker and gave me great feedback on my talk. Having to show someone my presentation helped me get it into shape well before the conference. I think Jan’s work here contributed to the high quality of the talks at the conference.
I’m not a vegeterian, but I really appreciated that all of the breakfast and lunches were vegeterian. I can imagine it would have made things very easy for vegeterians. The food was delicious and healthy tasting.
There was ample time left around the talks so that you didn’t need to choose between the ‘hallway track’ and the conference track. On the Saturday there was a siesta break between 12pm and 4pm. You could do one of the activities, take a nap, or just chat with others. I was a bit skeptical before the conference that this was going to be too long, but it worked out really well.
The lightning talks on Saturday broke up the afternoon and left space for serendipity to strike. Connie’s talk on waffles, Vim, and building her own language captured the audience.
On the Saturday evening after the conference ended, lots of people stayed behind to help pack down the auditorium. This made the job much easier.
There was a large contingent of people from Berlin, and they were all incredible. Berlin feels like it has a really vibrant community, one day I’d like to visit it.
Heart of Clojure was a very special event. I can’t imagine how much work it was for Arne Brasseur, Martin Klepsch, and all of the other helpers before, during, and after the event but it paid off. Heart of Clojure felt like a very polished event, not something being put on for the first time. Thanks to everyone who organised, helped out, spoke, sponsored, and attended the conference. I hope there is another Heart of Clojure in the future, if you get a chance to go, I highly recommend it.
A few years ago I came across a Leiningen project that defined profiles for :project/dev, :profiles/dev, :project/test, and :profiles/test. It took me a little bit of digging, but eventually I discovered what was happening. This is a convention that I think originated with James Reeves. I’m reposting my issue comment here, so it can be more accessible for searchers.
If you see profiles like this in a project.clj, here is what is happening:
Leiningen has a feature called Composite Profiles. On startup, if Leiningen sees a vector of keywords for a profile, it will lookup each keyword as a profile, and merge them together. For the :dev profile, Leiningen will merge the values of :project/dev and :profiles/dev.
If you want to add any custom settings or dependencies for your own use, you can place them into the :profiles/dev or :profiles/test in your ~/.lein/profiles.clj. If either of these are set in the user’s profiles.clj, they will override the empty :profiles/dev map specified in the project.clj. You need an empty map for :profiles/dev {} in the project.clj, because otherwise Leiningen will complain about a missing profile.
When I first started using Apple Music I signed up for a solo subscription. During the day, I would listen to music on my Mac, and my family would listen to music on the iPad. If we listened at the same time, we would get the error:
“Looks like you’re listening to music on another device.”
Eventually this error became too annoying and I looked at upgrading to a Family subscription. The Apple Music Family subscription allows up to six people that are part of the same Family Sharing group to play music at the same time.
It sounded like this was probably what I needed, but in all of the documentation I read, it wasn’t clear whether two devices that were logged into the same Apple ID would be able to play music at the same time. I talked with Apple’s support team and it still was unclear so I went ahead and upgraded to try it for myself.
After upgrading I tested playing on multiple devices, and it worked fine. If you have an Apple Music Family subscription, you’re able to play on multiple devices that are logged into the same Apple ID.
One of the features in Schema that I always appreciated was inline function schemas, using a schema.core/defn macro.
When spec was released it had many similarities to Schema, but one thing it didn’t have was a way of expressing specs inline with your function definition. Spec only supported defining function specs separately with fdef. This does have some advantages, it forces you to think carefully about changing your specs, and to be aware of possibly breaking consumers. While this is valuable, not all code is written to these constraints, and I found having fdef’s separate from the function definition had a number of downsides.
When writing Clojure, I noticed that I often resisted writing specs for functions. After thinking about it I realised that I didn’t want to duplicate information from the defn into the fdef. It’s not a huge deal, but it was enough to deter me from writing specs on code that was being heavily modified. This is a really useful time to have basic specs on your functions, so that you can catch refactorings gone wrong early.
I created defn-spec to increase the locality of the spec definitions, and to reduce the activation energy to start adding specs to your codebase. defn-spec copies the syntax (and implementation) of Schema’s defn macro. This has the advantage of adopting a proven design, familiarity for many Clojurists, and the ability to work with existing tooling that understands the Schema defn macro.
Benefits and tradeoffs
Like all things in life, defn-spec has benefits and tradeoffs:
Benefits
Makes it easy to incrementally spec your functions. It lowers the activation energy needed to add a spec, perhaps just for a single arg or return value.
Makes it easier to see more code on one screen.
Makes it harder for your specs to get out of sync with the function definition, as they are linked together. Depending on how you instrument your specs, it may be possible to go quite some time before you realise that your specs are broken.
Avoids repeating all argument names in the s/cat form.
Tradeoffs
For some specs, particularly complex :args specs with many branches, it may be simpler to define the fdef separately. defn-spec is designed for the 80-90% of Clojure functions that have simple argument lists and return types.
Making it easier to change specs means that it is easier to accidentally break your callers. If your fdef is defined separately, it forces you to think more about growing the spec.
This is similar to Orchestra’s defn-spec macro, but allows for optionally only speccing part of the function, and matches the well-known Schema defn syntax. Try them both out though, and see which one works best for you. defn-spec is still missing some features from the defn macro like destructuring, but they are on the roadmap to be added soon. I’m releasing this early to get feedback from other users.
defn-spec will follow Spec’s release cycle, and there will be a new set of namespaces and artifacts for spec-alpha2 and beyond. If you have features/bug reports, feel free to post them on GitHub.
Cognitect have recently released the results of their State of Clojure Survey for 2019. For thelastthree Clojure surveys, I have reviewed the free-form answers at the end of the survey and tried to summarise how the community is feeling. This year I’m repeating the exercise, keeping the same categories as before. If you’d like to see all of the comments, I’ve published them in rough groupings.
Some comments have been lightly edited for spelling, clarity, and brevity.
Update: Alex Miller has posted a few responses to some of the comments I highlighted below, and some suggestions for the next steps that people can take to help.
Error messages
Error messages have been a top complaint in Clojure for a long time, I think since the very first survey. CLJ-2373 introduced improvements to error messages in Clojure 1.10. A number of comments complained about error messages this year, but none of the complaints mentioned 1.10’s changes. Given that 1.10 was released on 17 December 2018, and the survey ran from 7-22 January 2019, it seems likely to me that many of the people complaining haven’t tried the latest improvement to Clojure’s error messages.
I appreciate all the work that’s been done in improving error messages. Thanks to Alex Miller and others!
Thanks for all of your hard work on Spec and the error messages in general (often caused by minor typos).
1.10 alleviates most of the problems I found with error messages. Since this was my single biggest concern I had with Clojure, I just want to say thank you for listening!
Just give me the Clojure filename and line where the exception happened, please. I beg of you.
Better error messages. Please!
Great to see the improved error messages in Clojure 1.10 - it was my number one issue in all previous Clojure surveys, but not this one :-)
Spec
Spec has been around for two and half years, but is still in alpha. A number of comments referenced Rich’s recent Maybe Not talk about coming changes to Spec. It was great to see Rich putting in to words some of the problems I had felt. It feels like the community is in a transitional phase with spec, where people are still not quite sure where things are going to land.
Clojure spec tooling would be nice! Because it’s hard to learn and visualize—and it’s not obvious how to integrate it with normal deftest tests. It has a massive amount of IDE integration potential though.
I’d really love spec to feel a little more complete. … I’ve got a lot of love for spec as a project and use it heavily at work, but I feel it’s shadowed at the moment by some ambiguity about what changes might be on the way.
Spec has great potential, but really bad UX. If you are new to Clojure it’s very hard to figure out what’s wrong. Spec made Clojure errors worse.
Spec should have better tool integration and more static analysis facilities.
Really excited to see the direction spec is heading, can’t wait to see the next iteration.
Thank you again, for all that you have given to us. Looking forward to the improvements in spec from “Maybe Not?”! I am certainly feeling the pain points they address, and currently work around them by generating heaps of variations on essentially the same spec per fdef.
I couldn’t put my finger on it, but it always felt a bit awkward to use specs and had to juggle the spec and what the spec meant at a particular point in time during the flow of my software. Thanks Mr. Hickey for the presentation during Conj, it finally makes sense! I’m looking forward to the next iteration of spec.
Docs
Documentation has continued to be a sore spot for beginners. API docstrings were often cited as being too hard to understand, instead people preferred to see examples at clojuredocs.org. The documentation at clojure.org has grown this year with a number of great guides added, particularly Programming at the REPL.
Concerning docs, I’m seldom able to really make use of the Clojure API docs. Since examples help me more to understand what a function does, I rely more often upon the examples provided by the community at clojuredocs. However, the quality of the examples vary and it would be helpful if there was a set of curated examples.
Clojure is great, but the main website, docs, examples and tutorials are not good. I would invest time in improving it as it has become a barrier for the language.
I wish we had a better documentation generator and a better culture of documentation. I really miss the documentation in the Haskell ecosystem (and five years ago, I never thought I would be saying that!). For example, the Haskell community generally expects detailed documentation of data structures, including detailed asymptotics, e.g. Data.Sequence. This complaint extends to many of Clojure’s included features, too. Headline features like STM have incredibly sparse documentation (about one printed page with the semantics only hinted at), so I’m left guessing at what I can actually do with these things.
The beginner support is exceptionally good. I think I would appreciate more intermediate to advanced support, in the form of books, courseware etc. to carry through to advanced Clojure.
I love Clojure. The docs are phenomenal. The improvements to the ClojureScript site and getting started and the CLI tools have been huge.
As a beginner I really struggle to understand the api docs. I tend to use third-party websites more often.
It would be great to know things like when to use a record vs a map. Intermediate guides would be nice. Also a guide on effective idiomatic Clojure like golang provides.
Please work on improving the ease of use. The doc strings for the core language are third rate, poorly written, cryptic and unhelpful. The error messages (although presented more nicely in 1.10) are still the same terrible error messages as they have been for years.
I’d ask for more examples in documentation and provide more tutorials like Scala School or Elixir School for Clojure, for newbies like me.
Beginner experience still needs more work. Please consult with first-time programmers and teachers who teach them.
Startup time
Startup time has been a perennial complaint, and there haven’t been any major changes here.
This long-standing issue of startup time + poor runtime performance (out of the box, without tinkering) + insanely high memory usage is becoming the major drawback of the language for me (and us). We are planning to move to a faster language (probably Rust or Go).
I would like to use Clojure (not CLJS) in AWS Lambdas for a side project but the painfully slow startup time prevents me from doing so. Improving startup times will do more than just improve developer happiness, it will make Clojure a much better tool for serverless computing.
Marketing/adoption/staffing
Clojure continues to grow in business adoption, but hiring developers has been one of the top three things preventing people from using Clojure for the last three years. I’ll add that one way that companies could address the mismatched supply and demand of Clojure developers is being more open to remote workers. Update: Alex Miller also suggested training people in Clojure, which I thought about when preparing the post, but forgot to include in the final copy.
A new option in the survey this year was “Convincing coworkers/companies/clients”; it was ranked number one in preventing people from adopting Clojure. In my opinion, this is a significant issue to be addressed if Clojure wants to grow its market share further. This probably needs more research to find out what the barriers are to convincing people. Elm is proof that exotic languages can be adopted widely, we should learn from what they’ve done.
We’d all benefit from wider Clojure take up. I worry about the view from the outside looking in. Why is it not attractive to more?
My biggest problem after one year of intense Clojure learning/coding is to find a Clojure job as remote (because there are zero Clojure jobs in Taiwan). Regardless of how good I am at writing functional code, finding a remote job (outside of the US) is hard.
Very passionate evangelist 5+ years — It’s surprising how hard it is to have some folks even consider Clojure. Rubyists are particularly baffled: “ok what do you mean REPL driven — we have pry!”, etc.
I co-founded a startup company and we picked Clojure as our primary language and are very happy with the decision. A nice note is that almost none of our hires knew Clojure before, they all learned it quickly and like it! I don’t see a lack of Clojure talent as a concern at all.
Clojure is thoughtfully designed and stable. It’s a dynamic, functional lisp that can actually be sold to the bosses (A sentence I never thought I would write).
Clojure has a great community and great technology but it would be super helpful in selling the language if there were more efforts put into marketing the language and perhaps having “blessed” solutions using Clojure from Cognitect; similar to what Lightbend does with Scala.
Thank you for all the hard work you all have been and continue to put into Clojure. We’ve built our entire new startup on Clojure, and was able to hire an amazing team within just a year by tapping into the local, tight-knit community.
Language
There were again more suggestions for improvements to existing language features, or development of new ones.
Please, take a look once again at Dunaj Project or similar ideas. I believe that the community still have many great ideas which could be adopted by Clojure and wouldn’t bring up any breaking changes. And we need better IDE for sure.
Please spend time improving clojure.test.
When I dream, I dream of seeing a slime-style backtrace in Clojure.
ClojureScript namespace definition <-> :require <-> macro & reader conditionals - and how these interact with each other is seriously non-trivial; especially so for macros you intend to use in Clojure too. Simplification there would be welcome, if possible.
Language Development Process
2018 had a lotofdiscussionsabout Clojure’s development process, and the relationship between the core team and the community. I was very curious to see how widely those feelings were reflected in the free-form comments from respondents. After compliments (~230), this was the most common comment type, with ~70 responses (positive, negative, and in-between) on Clojure’s language development process (out of 563 total comments). While many people had issues with the development process, there were also many people who were supportive of how Clojure is currently developed.
Grateful for the careful stewardship of Clojure and ClojureScript by Rich Hickey and Cognitect.
Clojure has strong opinions on how software should be built. It needs to keep those strong opinions as it’s these opinions that differentiate it and make it attractive. Keep up the great work!
Clojure is Amazing and you guys do a FANTASTIC job!!! Ignore the critics and keep doing things just like you have been. Thank you for this wonderful language!!
Closed core development and disappointment with spec have made made many friends leave Clojure/Script last year. Let’s make 2019 better.
I love Clojure, and an extremely grateful to Rich, Stu, Alex, etc for their continuing stewardship and work of this fantastic gift. With respect to recent kerfuffles about community involvement, I simultaneously understand and respect Rich’s position, and I am also concerned that some very respected Clojure developers who have contributed a lot (IMHO) feel disconnected and may be drifting away from the community.
About the governance of Clojure by Cognitect: thank you all! You’re doing it right.
Managing expectations in 2018 is about 10 years too late. Please give a signal that the core team recognises that, and also how much the community is worth to Clojure as a language before dismissing any critics to the (previously very unclear) contribution/lifecycle process.
I love the principled stand taken by the clojure/core team about silly arguments erupting in the “community”. I really appreciate that the core is so well thought out and there is a valid resistance to adding “stuff” in there.
I appreciate Alex’s insights into the Clojure development process, and what he’s working on. I also appreciate the all of the core team’s work that makes Clojure such a great language to use!
Thanks for recent improvements inline with community wishes. Still makes me sad we don’t have a bigger core team utilizing the skills of (some) of the community that could do things like e.g. actively improve core.async.
Clojure is a wonderful language; however, we are becoming worried about the future prospects of the language because of the way that Cognitect handles community feedback about the direction of the language.
2018 was the year that Rich finally explained how he felt about the open source community around Clojure. For many this was disappointing, but I’m glad that he finally explained how he felt, because at least it is now in the open and people can make decisions with full information. Five years ago, Clojure could afford to lose key contributors because the language was growing and new people were coming along. Clojure is now at a very different place and position, and it’s no longer growing at the same rate, or has the same mindshare. I’ve invested many years, and lots of time into the Clojure community. Up until this year, I thought I would have a long career with Clojure, but I am starting to look around at other language communities for ones that are a better fit for my values.
Even though I agreed with the substance of Rich Hickey’s “Clarity” gist, it is concerning to me that some of the more prolific library maintainers are leaving the community. I’m sure that had it been me, I would have responded even less graciously. Still, there’s an element of community management that goes beyond just being correct.
After using Clojure professionally for 7 years, I am hedging my bets by involving myself in the Rust community. While Rich’s comments may be lauded by the greater online community, it has had a large impact on my personal consideration on the state of Clojure. … But, as a member of the community, my concern is that the user base of Clojure cannot grow larger without this concept of ownership changing. This conflict, at the end of the day, is a conflict between what my hopes for Clojure were and the interests of Cognitect … But it seems that Clojure has remained what Rich originally intended it to be. … The conflict is simply that, for some reason, I had assumed my interests were in line with Cognitect’s. It has become clearer, year after year and especially this year, that this is not the case. It’s very likely this misunderstanding is a mistake on my part. Looking back, I see no reason that I should have assumed this other than it is generally the trajectory of most languages focus on growth. For this reason, I will likely exit this community as many others have. I don’t have any hard feelings or regret being part of it. I am very appreciative to all of the hard work that has gone into Clojure and regretful that I did not contribute nearly enough during my time writing it.
Community
People seem to mostly enjoy the Clojure community, but others have had negative experiences, often citing an elitist attitude from some community members. Losing the history from Clojurians Slack came up several times. I’m not sure if everyone is aware of the Clojurians Slack Log?
… the community has been aggressive/hostile from our experience.
Love the community. Everyone has been really helpful. I wish it was more diverse and wrote more stuff down. A lot seems to get shared face to face which is great, but makes a really high barrier to entry.
I’m a bit worried about the long-term viability [of Clojure]. I wouldn’t be so worried about the language and ecosystem not being friendly to newcomers, it’s clearly distilling many years of software practitioner’s experience and thus is opinionated.
The only bad/ugly part of the Clojure ecosystem is the use of (unsearchable)Slack as the main knowledge base.
I really enjoy this community. Despite the occasional war of words, generally it seems to be filled with helpful, deep thinking people who are fun to interact with. If we were known for one thing only, I’d like it to be that!
Thanks so much to the Cognitect team and everyone who helps to make the Clojure community better. I understand that people can be very vocal about the things they dislike and not so much when it comes to appreciate the things that work nicely. Being able to use Clojure professionally to build software that solves problems is really exciting and I’m grateful for it.
I suggest moving off of slack to a more accessible chat system. Losing history is a bad thing. Check out discord or matrix or gitter or mattermark or any other number of tools made for this purpose.
I’m concerned that there is too much of a disconnect between the core developers and the wider community. … I think it would be great if the core acknowledged some of the work of Clojurists Together. I think the new clj-commons organisation is a great idea and I’d like to see it better promoted and supported.
The tone in the community has changed. I think this is very sad.
The community and the prominent people around it are OUTSTANDING. It is really something to behold. This was my first motivation. I am staying for the good language and tools and etc. but the people behind it are definitely the main treasure here.
Libraries
People are still looking for a curated on-ramp into Clojure web development, a ‘Rails for Clojure’. There are a number of frameworks and templates here, but they don’t seem to be hitting the spot. I’m not sure whether that is because of limited mindshare/marketing, limited documentation, or the scope and quality of the frameworks. Having used Clojure for many years, I no longer find it difficult to find and assemble the pieces that I need for projects, but I definitely remember finding this being difficult when I started. Data Science and ML were places that people saw a niche for Clojure. Several people hoped for improvements on core.async, it still has a number of rough edges.
I would love to see a way to get open source tools, such as Cider, sustainable support. I’m worried about all you open source maintainers getting burned out and not creating these awesome tools. I don’t know what this looks like yet. [Ed. I’ve got just the program for you]
I know that Clojure promotes a build-it-yourself type of mentality, which is nice, but it would be great if it had standard implementations for things like webapps (auth, security, etc.) that were leading the way for alternatives to follow. It’s ok to be opinionated, if it works properly.
One huge opportunity that is still being missed is the field of data science — Clojure(+Clojurescript) has the potential to become the best platform for data research ever. In that field, more than others, it is important to be beginner-friendly.
Clojure needs a Rails-like “killer app framework” that provides novices a training-wheels included opinionated approach to building applications with Clojure. The roll your own / pick your poison approach to Clojure is a barrier to many.
core.async could use some love with closing long-open bugs and documentation on best-practices and how to build bigger systems with it. Or maybe it just needs to be officially deprecated? As an outside observer, it feels like it’s in limbo by the clojure-core team. Either way, we’re currently using Zach Tellman’s manifold abstractions a lot to great effect.
I much prefer the library over framework approach taken in the Clojure community, but I do miss the rapid development which RubyOnRails allows, i.e. conventions.
… I think a curated collection of recommended third party libraries would help a lot of teams.
Other compilation targets
People have been experimenting with the Graal compiler to produce static binaries without needing a JVM. At this point that seems like the strongest option for people wanting to use Clojure without the JVM. Better Android support was also requested.
Clojure doesn’t seem to have any plan for Android integration, this is pretty sad considering how much computing power is employed there, and how big of a showcase mobile development is :(
I think that the limitations of their respective host languages are holding Clojure and ClojureScript back.
Is there a Clojure project that compiles to WebAssembly?
So many people want a native Clojure. We need a native Clojure without JVM. Is there a plan about this?
The advent of GraalVM, and the resulting ability to build Clojure executables that startup up quickly has been a game changer. The slow startup time and need to use a JVM has been a real impediment to Clojure’s adoption as a general purpose language. It would be even better if someone would pursue a Clojure native x86_64 compiler which would generate native executables, but that is probably not going to happen. Ensuring that Clojure can run well with GraalVM is probably the most we can hope for, and there is plenty of work to be done there.
Difficulties in running on Android may end up forcing me to port all the existing desktop applications off of Clojure to pure Java/Kotlin solutions.
Targeting two of the most popular VMs is great, but being able to get native performance is always a wish, like LLVM.
Typing
For the past few years people have been less and less worried about types. I suspect this is mostly due to spec.
Love Clojure. But at two Java shops over four years, the lack of static typing (the ability to command-click identifiers in IntelliJ and navigate to their source, the ability to refactor) has prevented me from winning over colleagues. TBH it’s also reduced my own enthusiasm since the prospect of me working on our 1.6MM LoC project without static typing honestly scares me.
I’ve selected “no static typing” but matching Haskell is not my intent. Lack of static guarantees is what bothers me. If it were possible to reject programs that are guaranteed to fail, that would be an improvement. For example: (clojure.string/replace nil #"a" "b") or (clojure.string/replace "abc" #"a").
ClojureScript
Shadow CLJS was mentioned by many people as having been a great part of their workflow. For those not using Shadow, externs remain challenging.
shadow-cljs made me continue developing ClojureScript + started using it in backend services. It provides significantly better support for NPM libs than lein-cljsbuild that I’ve been using before.
Without shadow-cljs we wouldn’t have been able to use clojurescript because we found it impossible to get the npm packages we needed to work.
It’s not at a terrible state right now, but I’m at least slightly worried about more divergence between Clojure and ClojureScript in the future.
shadow-cljs does great job on compiling clojurescript and npm integration, but it more and more diverges from the standard compiler. I would love to see shadow-cljs merged into the mainstream clojurescript compiler. In general: I would like to have better, easier way to compile clojurescript without setting tons of flags.
Feel like cljs could make an amazing, even-better version of dash (maybe powered by Reagent/re-frame).
Although work has been done on externs, I find venturing into JavaScript interop fraught with peril. I would be very happy if interop would just work, same as Java interop. Most frustrating is that development builds work but releases don’t. Catching these problems early would seriously mitigate the issue. For now I’m using cljs-oops library, which works very well, but it feels kind of a hack. Overall I’m very content with ClojureScript though!
Setup/tooling
Clojure’s tooling has continued to improve. This year lots of work has been done adding plugins or expanding functionality of the new official Clojure tools. clj on Windows was asked for by many people. Maintainer burnout seems like a significant risk to the Clojure tooling ecosystem.
I’m happy to see that many have realised we need to find ways to better support the development of CIDER, a crucial part of Clojure development infrastructure. Seems like the lead dev has been burning out recently, and many of us depend on his (and a few others’) goodwill.
It has been a challenging first year in obtaining high productivity with the tools. Would be great to see a simple but capable tool set get some focus, e.g. Atom or VSCode.
I no longer want to work with CIDER or Cursive, rather, I’d like to work with REBL + a REPL as my IDE. I believe we can have a renaissance of Clojure tooling if prepl becomes demystified and/or we add a tap-repl, something like tap> to the Clojure repl. This would allow tooling to be added and removed dynamically.
Please port clj to Windows!
clj for windows please.
We need clj under Windows. Is it possible to release it?
Compliments
As always, the vast number of comments were compliments to Clojure and the core team.
Keep up the good work! 👍
Clojure is beautiful, functional and concise. It really rejuvenated my love for programming.
I absolutely love Clojure 💚
I really appreciate the well thought out design of Clojure. We feel that Clojure gives us distinct advantages while providing a language with minimum disruption.
5 years later and I still enjoy writing Clojure. Thanks to all those who make it extremely productive and fun to use!
I really appreciate all the work Cognitect and the community abroad put into making Clojure and the Clojure community a joy to work with. Keep up the good work!
It’s my 11th year with Clojure. What a journey. Thanks, Rich and everybody.
Conclusion
Clojure is a great language, and people are very enthusiastic about it. Its adoption in business continues to grow. There are a number of areas for improvement still, especially if the Clojure community wants to grow further. Not all of this work needs to go through Core though, particularly in areas of documentation and guides, libraries, and tooling.
I worry about the Clojure community losing key contributors. There are a few linchpins holding a lot of things together; if we lose them, it will be hard to come back from. If you don’t want to see this happen then please support the libraries and tools you rely on by contributing code, documentation, issue triage, or money.
Almost two years ago, there was a Github issue on reagent (a ClojureScript React wrapper), suggesting that Preact be added as a substitute for React. I wrote up a fairly long comment about why I didn’t think this was a great idea (at least not immediately). React’s recent announcement of the new hooks feature made me think about it again. I’ve republished it here with a few edits for context and time.
Introduction
In principle, I’m not opposed to the idea of Reagent using Preact. It has some cool features and I like that it is small (although in comparison to the total compiled size of a normal CLJS app it’s probably a wash). If Preact worked 100% out of the box with Reagent with no code changes required then I would have no issues with someone swapping out the React dependency for a Preact one and calling it a day. If there are only a few minor tweaks to Reagent required to pass the tests, then again I don’t really have any issues with that. I suspect that even if you have 100% of the tests passing, there will still be issues, as Reagent was built around React, and may not have tests that would cover the difference in behaviour between React and Preact.
Abstraction
It looks like it may not be a pure lift and shift to support Preact. If that’s the case then we run into a bigger issue: abstraction. Reagent was written and built around the ideas and in the context of React. There are assumptions (probably tens or hundreds) built around React’s API and possibly implementation details too.
Adding abstraction adds a large cost because you can no longer program against a concrete API and implementation, you now have to consider two. There are three numbers in computer science, 0, 1, and many. We would be moving from 1 to many, and that takes work.
An aside: recently at work, we were looking at moving a legacy system from only supporting dates in the past to also be able to support dates in the future. This should be straightforward right? We talked to the programmers responsible for it and they couldn’t guarantee that it would work, nor whether supporting future dates would be easy or hard. In the building of that (or any) system, hundreds of simplifying assumptions are made around the context that the system is going to be built in.
It is a very common pattern to have different backends and I don’t see any downsides to it.
I can’t think of a single example of a system with multiple backends that didn’t have any downsides to it, e.g. ORMs, HTML/CSS/JS, Java. There may be some, but they would be the exceptions that prove the rule. Everything has a cost, the question is whether there is a benefit that outweighs the cost. It is much harder to remove something from software than to add it, which is why we should be certain that the benefits outweigh the costs.
While Preact strives to be API-compatible with React, portions of the interface are intentionally not included. The most noteworthy of these is createClass() …
https://preactjs.com/guide/switching-to-preact#3-update-any-legacy-code
Reagent currently uses createClass. There are workaround options provided, but this is an example of some of the API differences between React and Preact which you need extra compatibility layers to support. Do we know if the compatibility layer works 100% correctly?
A possible future if Preact support is merged now
As a thought experiment, let’s assume that Preact is in Reagent with some kind of compatibility shim. Preact already has several performance optimisations that people can take advantage of:
customizable update batching, optional async rendering, DOM recycling and optimized event handling via Linked State. - (from Preact homepage)
Wouldn’t you want to be able to take advantage of those in your application? I certainly would. Now to do so, you may run into issues because the compatibility shim layer that was written was encoded around default assumptions of React, and they may not apply to Preact. Do we have to rework the shim layer, or lower level Reagent API stuff? Who is going to do that work? Who is going to review it and merge it?
Let’s consider the reverse. Perhaps in some new React version, Facebook comes out with a new API which is faster or better suited to Reagent’s style of rendering, so we want to switch to that [since writing this they came out with Hooks. These may be added to Preact also, but this isn’t certain and neither is the time-frame]. However that new model may not work with Preact. Again, we’re in a bit of a pickle: Preact users want to be carried along with Reagent and get the benefits of new Reagent work, but it may not be easy or possible to support the new API for them. Now what?
Consider everyday development on Reagent. Reagent’s source code is built around a very detailed understanding of React and is highly optimised. If Preact was supported too, then developers would probably need to gain an understanding of Preact too.
At the moment, Preact has one main contributor, it has been around for 1.5 years. React has many contributors. I’d estimate there are 100+ people with very deep knowledge of React. It’s been around (in public form) for 3.5 years. In general, the JavaScript community does not have a reputation for long-term support of projects. What happens if development slows/stops on Preact and the compatibility layer isn’t kept up to date? It is much harder to remove something than it is to add it. Who decides when/if to remove Preact from Reagent at a future date?
These are all hypotheticals, but I hope this demonstrates that the extra abstraction provided by supporting two VDOM layers doesn’t come for free. At the very least, it consumes extra brain cycles when testing and developing Reagent, extra support and documentation costs from users wanting to use one or the other, as well as extra indirection when running and debugging apps using Reagent.
The Innovators Dilemma
If you haven’t already, I highly recommend reading “The Innovators Dilemma” by Clayton Christensen. One of the key points he makes in that book is the difference between integrated and modular products and when to develop each kind.
CHRISTENSEN: When the functionality of a product or service overshoots what customers can use, it changes the way companies have to compete. When the product isn’t yet good enough, the way you compete is by making better products. In order to make better products, the architecture of the product has to be interdependent and proprietary in character.
In the early years of the mainframe computer, for example, you could not have existed as an independent contract manufacturer of mainframe computers, because the way they were made depended upon the art that was employed in the design. The way you designed them depended upon the art that you would employ in manufacturing. There were no rules of design for manufacturing.
Similarly, you could not have existed as an independent maker of logic circuitry or operating systems or core memory because the design of those subsystems was interdependent. The reason for the interdependence was that the product wasn’t good enough. In every product generation, the engineers were compelled by competition to fit the pieces of the system together in a more efficient way to wring the maximum performance possible out of the technology that was available at the time. This meant that you had to do everything in order to do anything. When the way you compete is to make better products, there is a big competitive advantage to being integrated.
…
In order to compete in that way, to be fast and flexible and responsive, the architecture of the product has to evolve toward modularity. Then, because the functionality is more than good enough, you can afford to have standard interfaces; you can trade off performance to get the advantages of speed and flexibility. These standard interfaces then enable independent providers of pieces of the system to thrive, and the industry comes to be dominated by a population of specialized firms rather than integrated companies.
I would argue that we are still very much at the point where the current VDOM libraries aren’t good enough yet. They aren’t yet ready to be commoditised, and the best option is to tightly integrate.
Options from here (with some conjecture)
Someone can make a PR to Reagent to add support for Preact. It will probably take a while to get merged because it is a significant change. Once it is merged and released, there will probably need to be several rounds of revisions before it is ready to go. Because Reagent moves relatively slowly, this will take a while.
Reagent also has a large number of production users, so new releases need to be well tested and stable. Adding Preact to the mix is going to slow this down further.
Someone can make a fork of Reagent (let’s say it’s called Preagent). You can run wild experimenting with what is the best way to use Preact in Preagent, take advantage of all of the great features Preact has, and have a much faster turnaround time for releasing and using it. You will be able to work out what is the right API and integration points for Preact because you have room to experiment with it, without the weight and responsibility of bringing the rest of the Reagent users along with you.
At some point in the future, you could review merging Preagent back into Reagent, given all that you now know. You would also have the weight of evidence on your side where you can demonstrate the benefits of Preact and can show how many users want Preact. This would let you make a much better case for including Preact, give you what you want in the meantime, and likely provide a higher quality integration in the future.
Alternatively, you may decide that Preagent is better served going its own way and integrating more closely with Preact. This is also a good option.
Abstraction is not free
The point I have been trying to drive through this post is that abstraction is not free. Over-abstraction is a common anti-pattern and it saps productivity. I had a friend who recently left a Clojure job and started a Java one. He quipped to me about how he’d forgotten what it was like to trace code through five layers of abstraction to get to the concrete implementation. As programmers, we’re trained to solve problems by adding layers of abstraction, but that isn’t always the best way to solve the problem.
Lastly, this isn’t a personal attack on you or your ideas. I’m all for innovation in ClojureScript webapps and I think that it is worth investigating Preact and how it could work in the ClojureScript ecosystem 😄. I’m not against Preact. I would consider using it if there was a measurable benefit. I’m just suggesting that the best way to go about this is probably not to integrate it into Reagent as the first step.
I sent the first newsletter for The REPL two years ago, and I have really enjoyed writing it. I’ve learnt a lot from what people have written and created to share with the Clojure community. Sometimes when I’m writing the newsletter I’ve thought “That’s fascinating, I’d love to hear more about the technical details of that”. Now I have an outlet for doing just that.
I first had the idea of producing a Clojure podcast around January 2018. I didn’t have the time for it then, but the idea kept swirling around the back of my brain. When Michael Drogalis’s Pyrostore was acquired, I wanted to hear more about the acquisition, so I contacted Michael to see if he wanted to talk about it. That became the first episode.
My goal for The REPL is to talk with people about the technical details of the projects they are working on. The Clojure community has a ton of interesting people doing really creative work with Clojure. I’ve really enjoyed talking with a bunch of people already, and I’m looking forward to talking to more of them in the future. You can find it therepl.net. It’s available on Apple Podcasts, RSS and all of the other common podcasting apps.
I was honored to recently be a guest on Screaming In The Cloud. I talked about Google Cloud Platform, some recent outages, comparing it with AWS, and a post I wrote on the Deps blog about Google Cloud Platform. It was a lot of fun, and Corey was a great interviewer; I learnt a lot from him on how to conduct an interesting interview.
Cognitect has recently released the results of their State of Clojure Survey for 2018. For the lasttwo Clojure survey’s, I have reviewed the free-form answers at the end of the survey and tried to summarise the zeitgeist of community feeling. I enjoy it each time, and so here’s this years analysis.
Some comments have been lightly edited for spelling, clarity, and brevity.
Error messages
Error messages have been one of the top complaints about Clojure since the surveys started, and this year they have gotten worse with the introduction of specs on core macros. I’m not the onlyone who has noticed this either. In 2015 Colin Fleming gave a talk on improving Clojure’s error messages with Grammars. In previous surveys there was hope that spec would be able to use this approach to improve error messages; Ben Brinckerhoff has recently shown a proof-of-concept that gives some similar (excellent) error messages in Expound.
My primary wish would be for more helpful error messages (e.g a la Elm) in Clojure and Clojurescript.
I was excited about spec initially; however, I find the error messages so large, noisy and unhelpful that I don’t want to use it. To put it another way: when there’s a spec failure, my heart sinks; knowing that the next few minutes are probably going to be frustrating and painful.
Spec was supposed to help with error messages, but it just made them worse. Stuff like (let [f/x 42] f/x) now emit insane error messages. But it looks like the Clojure dev team has moved on to improving command line tools now? Can we please just get error messages fixed before a ton of people get frustrated and jump ship? The current state of errors in Clojure is an embarrassment to the community.
Error messages is a big show stopper. I am trying to get Clojurescript adopted for web and react-native development withing our organization for few years, but I always see the same pattern: people get some error during project setup/configuration and get stuck forever because they have no idea what the error means at all and how get it fixed. Please please improve error reporting. :)
A big problem for new starters, an issue that has been singled out time and time again over the years, is poor error messages that lead to difficult debugging of code structure. That Clojure 1.9 has actually made these error messages harder to understand is a disaster. […] I hope I’m wrong and that I’ll understand the grand plan eventually.
Please please please please enhance the error messages. An incredible error reporting would make this language out of this world, and that would make the biggest difference.
Spec
Comments ranged from loving spec to disliking it. There were also requests for more guidance on how to apply spec to real-world use-cases and how to integrate it into whole programs.
OMG Spec is amazing. It’s dramatically changing the way I program. I’ve been starting new projects out by writing specs before nearly any other code. The result is that I am able to extend my hammock time to the keyboard, and think through my domain model with composable, high level descriptions implemented as specs. The fact that these high level descriptions then become so useful in so many different ways — generative testing, value conformation for data api parsing, more useful error messages, etc — is a stunning thing to behold. I’m excited to see how the fruits of these ideas permeate through our and other communities.
The presence of “spec” on some of those lists made me want to rate everything else “important” so I could show how unimportant it is. […]
spec is awesome, but its dogmatism leaves us needing tools like spec-tools, which makes working with spec clunky.
There needs to be more guidance, development and reliance on spec from both Cognitect and community.
Docs
Documentation in all forms has been another perrenial complaint but this is starting to improve. clojure.org continues to add more guides this year on Programming at the REPL and Higher Order Functions among other things. Martin Klepsch’s cljdoc holds a lot of promise to improve the state of third-party documentation, by automatically generating and publishing docs for published JARs.
We should take example of how Rust community/documenation is run: BIG focus on newcomers, features MUST be perfectly documented before being shipped. This serve advanced users as well. No offense, but Rich Hickey should understand what good documentation is, or delegate completely to people who do so that he can focus and making a even greater language (his strong point).
I feel the true weak points are confusing error messages, not enough friendly guides/tutorials for the early to beginner Clojurist, and a lack of solid CLJ/S example projects. These are key because it is difficult to know the right way without “blessed” examples.
Please provide a good set of tutorials on Javascript Interop such as how to translate a ES6 or higher JS file into ClojureScript.
The documentation is really awful - incomplete, vague, no examples, no semantic precision - not up to the standards of what one normally sees for programming language documentation.
Please improve the docstrings. They are the worst of any programming language I have ever used.
Really, the core documentation is not written for beginners or even intermediates
What I miss most is a book which teaches how to approach problems with Clojure. I mean primarily not “best practices” or “design patterns” or language independent algorithms. Nor a kind of language reference or few lines long mini examples, but complex, advanced examples with detailed explanations about the pros and cons of certain solutions applied/not applied.
Startup time
Startup time continues to be an issue for people. As serverless computing becomes more mainstream, this may end up pushing people from Clojure to ClojureScript, or elsewhere.
Startup time is really really really really really long
Startup time never used to be an issue for me, as my services were long running and persistent. Lately however, I’ve been using Docker for jobs. Startup time becomes an issue here. Continued improvements in total app startup time will pay dividends going forward.
Startup time regression. Clojure 1.9 startup time is 1/3 longer then Clojure 1.8 (1.6s vs 1.1s on my humble machine).
Marketing/adoption/staffing
There were lots of comments this year in this category. Even after 10 years, Clojure still unknown or seen as being extremely niche to many people, especially among business stakeholders. Hiring seems to be less of an issue than in years past. There is still a large contingent of programmers looking to work with Clojure in remote positions, although it’s hard to say whether this proportion is any higher than in other language communities.
I recently hired a programmer with no Clojure experience. She was writing lovely idiomatic Clojure within a month. The biggest difficulty was overcoming her initial fear of parentheses.
To use Clojure(Script) more it would help if there was marketing for it. It’s still largely unknown, and therefore hard to sell to other companies.
There is always an uphill battle in pushing Clojure as an enterprise solution, this seems to mostly occur to lack of marketing. Non-technical decision makers care about branding.
I’ve found that some folks really dislike lisps passionately. One of our vendors indicated that his customers and employees were highly critical of the use of Clojure as an extension point.
Unfortunately, this year my organization in ceasing its use of Clojure for production projects after many years of using it. Staffing and on-boarding external resources are the 2 main reasons for doing so.
Hard to convince managers at work to use Clojure because of the perception that its a fringe language
I wish it was easier to find a remote (non US-only) remote Clojure/Clojurescript job for people with no major Clojure experience :)
It’s very important to address the enterprise software community. We need formal tools, formal approaches, standard IDEs, SonarQube, Eclipse, Jenkins, e.t.c.
I believe “JavaScript converts” will be an important source of growth for Clojure moving forward.
We find Clojure a joy to use and hiring Clojure engineers or those excited and capable of learning Clojure has been a big advantage, contrary to common wisdom. Clojure positions tend to self select for strong candidates.
Language
Enterprise Java tends to lean heavily on reflection to pull components together (e.g., Spring, annotation-driven JUnit and JAX-RS). To incorporate Clojure into those environments, lots of component-specific wrappers are needed. It is easy to slip a little Java into a Clojure codebase, but this has made the reverse difficult.
I would love to see some more work on improving clojure.test. It works, but I wish there was less stateful macro magic and more flexibility with fixturing and test structure
I’m increasingly finding that I’m writing more cljc than clj or cljs but find it really frustrating that cljs seems to be a second citizen to clj. It would be much better if there was actually a core Clojure library/api that was actively supported by both, with clear variances between them.
I would love to see some work towards using JVM MethodHandle’s to obviate the need for the compiler’s direct linking. Seems we could have our var cake, and eat it too.
We have a relatively large codebase and 20+ build targets (fat jars/advance compilation js files). we’d like better AOT support and ability to incrementally compile this code base as most changes only affect a few targets. right now AOT compile is pretty black box and not very amenable to building tooling.
Language Development Process
This hasn’t changed in a long time, and I don’t see it changing in the future, but it has been persistently highlighted by many people as a big issue.
For long term success of Clojure/ClojureScript, it’s important for the Clojure core team & project to be more open and welcoming like other open source projects. The view from outside of the Cognitect bubble is that the language and community aren’t healthy at all, and I fear for it’s long run viability.
The core team seems to be extra cautious and move really slowly on the language and core libraries. And the roadmap for releasing new versions is really opaque to the community.
Cognitect has continued to be good technical stewards of Clojure, but poor stewards of the relationship between the core team and the rest of the community. Many (most?) senior Clojure programmers want nothing to do with contributing to Clojure or any projects owned by Core (like the site) because the process is designed to frustrate and prevent meaningful contributions.
Cognitect need to climb a few rungs down the ivory tower and engage with the community as if it were in fact made up of human beings trying to do their jobs.
I don’t know what direction the core team is taking Clojure, and the top issues (startup time, poor error messages, lack of involvement with the community and numerous improvements to Java interop in JIRA being ignored) have not changed significantly over the past 5 years. I never felt welcomed to contribute, and seeing how Skummet and other large efforts from the community were largely ignored was not encouraging.
More attention to JIRA issues/patches/contributions would be great. […] I painstakingly write a patch and jump through the JIRA hoops; but there it languishes. The dev team’s complete silence is not encouraging. Makes me feel more of an outsider, less of a part of the community.
Community
By and large, people found the community welcoming, and personally I have found it to be one of the best things about working with Clojure. However there is a persistent undercurrent of eliteness from some Clojure programmers which really puts people off the language.
I find Clojure a great language, but I haven’t noticed any increase in market (or mind) share. Quite the opposite, it seems to be going nowhere, possibly dying a slow death I hope I’m wrong, but, for now anyway, I’ve given up on it.
Thanks, Alex et al! I’m still extremely happy with Clojure, and appreciate the community. I worry a bit about the possibility of clj/s losing user share and dwindling, which I didn’t worry about a year ago. I hope like hell that that won’t happen, and try to do at least a bit toward preventing it.
That defn podcast is bloody marvelous
Being in the Clojure community since around 1.2, I loved the language then, and I like the direction, it’s been taking since. The community is great, very friendly and even its most controversial figures are usually polite, intelligent people. Heck, even the local troll from the mailing list is occasionally helpful, stays on point in his threads and sticks with the conversation, even after blowing stuff up.
clojure.org changes so infrequently (like once a year, if that?) that I’m concerned the language is not being enthusiastically supported and will soon fade away. It makes me nervous to invest time and money in a language that seems to have no vitality.
Clojure conferences lack diversity in both attendees, speakers, and programs. I hope the merging of Conj and West will improve that this year.
People who give to the Clojure community are role models. I’m serious about this. I’ve woken up and thought “what would Bruce Hauman do” more than once. I talk with my daughters about the Clojure community.
Cognitect is doing an awesome job, making programming fun again! Alex Miller is very helpful.
Effort community makes to promote diversity, particularly in terms of women is good, although there is a long way to go.
I am doing Clojure for some time now and what does not bring me joy is the fact that the message to other communities (other languages) is clear: “we are better and smarter”. […] I think we can do better without thinking that we are.
I didn’t want to tick the unpleasant communities cause it is not […] lately [however], […] I started noticing a kind of “elitism” especially from people that are frequent speakers or started working early with the language.
I have multiple colleagues, all great developers, all former Clojure developers. Whenever Clojure comes up they never fail to mention how alienating the “smarter than you” attitude sometimes present in the community can be.
I’m worried about the perceived shrinking of the Clojure community
I’ve gotten to know a few more female developers using Clojure in the past year, and have learned that some parts of the community are scaring/driving women away. This includes unpleasant interactions on Slack and working to avoid certain people at Conj.
Not to put too much stock in the oversimplified Gartner hype curve model, it does seem like Clojure is passing through what they call the trough of disillusionment.
Please consider supporting Clojurists Together, even launching initiatives for the community to help through it.
Libraries
More guidance on how to compose and select libraries remains a common common issue for people in this section, as well as improving the state of library documentation.
The plethora of web app (both client & server side) frameworks is both bad and good. Too many to choose from (no standard like rails) and also lots of options. Also, limited security options, especially w.r.t. use of session state, async handlers & server-less designs.
I frequently find myself frustrated by looking for a library that will help me do “X”. There are many choices out there, each of which does some subset of what one might want in that arena, but a different subset.
It is sad that I often find interesting Clojure or ClojureScript libraries that are abandoned and haven’t been updated for a long time
Need more quality open source projects for machine learning and data visualization libraries. Great to see the advances from conj but for a language that has a front end tool built in, there is no excuse for someone not to build a great visualization library for data science.
No de-facto web framework, Pedestal is in dire need of documentation, Schema is deprecated and Spec isn’t ready, New CLI tools in Clojure 9 competing with Leiningen/Boot. All of these complaints have something in common: it looks like the Clojure core team is actively hostile towards the larger community. All community-built tools/libraries are open to attack and replacement by the core team and there’s always fear of a major destabilizing revolution everyday.
Several widely used library started to stall… sustainability might become a concern… [Ed: hopefully Clojurists Together can help here]
Other targets
As in previous years, there are a smattering of people asking for alternative platforms to target other than JS and the JVM. LLVM, Go, and Web Assembly all had interest from people.
Is it posible to use Clojure with the Go ecosystem? That would be amazing
It is hard to watch Go get so much mind-share. Clojure on the JVM is great, but it would sure be nice if I could build a Clojure application that didn’t need the JVM.
I still long for a low-level oriented Clojure dialect to be able to produce small clj executables with low overhead and systems programming capacity, but one can dream!
webassembly support? and native support? (like a scala-native)
Typing
Requests for static typing were down from previous years. I attribute that mostly to spec gaining wider adoption and understanding.
Clojure needs to make a better case for why static typing is not necessary for maintainable, large, multi-contributor codebases and how the REPL dev process helps with that.
I still love Clojure as a whole more than any other. I’d love to address long-term maintenance. The difficulty of stepping back into a codebase after some time can be more challenging than with Haskell or PureScript. I wish the compiler could do more for me. Spec is huge though, really enjoying it. It still takes a good amount of discipline to use well, but I’m thankful for it.
I used Clojure for one work project because a coworker recommended it. Overall it was a good experience, but the lack of a type system made maintaining that particular project too difficult. Tools like Plumatic Schema and Typed Clojure helped in some ways but didn’t end up pulling their weight. Regardless, the Clojure service continues to run today and it’s better than the Ruby service it replaced.
ClojureScript
ClojureScript continues to improve, although getting the tooling setup remains a pain point, as does integrating NPM modules. There are recent improvements on that front though, so this may not be such an issue in 12 months time. I have heard lots of good things about shadow-cljs, it seems to be a strong tool to investigate for ClojureScript usage, especially if you work with a lot of NPM modules.
ClojureScript has had a serious push this year… ;)
For CLJS, we really need to get npm deps working. Right now it’s hard to get right, or at least there’s no up-to-date tutorials available..?
I find I spend more time than I should configuring my development environment so it works well. This is particularly bad in React Native, but cljs/figwheel can still be frustratingly complex to get right.
Lack of async/await support in CLJS is holding it back. The JS community has settled on promises and async/await as the primary asynchronous patterns, and although core.async is useful for certain things, shoehorning existing JS code into it is very kludgy.
Nothing has made a bigger improvement to my ClojureScript workflow in the past couple of years than shadow-cljs.
Intelligent externs inference, plus simple ^js or ^clj tagging for anything that’s missed
Virtually trouble free usage of other JavaScript modules /npm
Way faster cljs compile flow due to smart caching of intermediate artifacts
Self-hosted dependencies handled very well
NPM support in ClojureScript seems to be a little bit tricky and not working all the time for all the libs. But has been improved a lot in the past.
Probably my biggest frustration has been trying to use libraries from JS in a full-on ClojureScript front-end app.
Setup/tooling
Tooling on the Clojure side seems to be somewhat less of an issue this year, but setting up ClojureScript projects with all the things that you need remains an issue for many people.
Clojure is now mature and stable enough to start focusing in tooling. Many JVM tools are not Clojure specific and for people coming to Clojure from non-jvm languages diagnosing problems is a barrier (more with systems in production). From GC issues, to debugging exceptions, to attaching to a process in runtime to peek at some memory or patch something. It was pretty easy to do with gdb in linux, it should be much more easily done with Clojure.
Fast track to CLJS web dev for junior devs, in one piece in one place. There is not that much essential knowledge/tooling needed to get started that it can’t fit in a CreateReactApp-like solution. Too much content, too many sources, don’t help. No need to be exhaustive.
Git-based deps seem like a bad idea. clj build tool is otherwise cool, if a bit unnecessary given the prevalence and quality of leiningen, along with the stability afforded through maven central/clojars.
I think we should pay more attention to interactive programming. Like what Bret Victor proposes, beyond the REPL.
Making changes to libraries then having to jar them just to test is hard (checkouts never worked for me (or others) with cljs). I hope Alex Miller’s recent work will fix that.
Once Figwheel and Reagent and tests were set up, my experience as a developer was very positive - barring some bad times with advanced Closure optimization bugs. Really happy with ClojureScript myself, but it would be hard to convince a friend to start using it, because there is no clear or standard way to develop.
Compliments
As always, the survey was dominated by positive comments. Here were some of my favourites.
Clojure has ruined every other language for me. Every time I have to do something in another language, I always wish it was in clojure. It’s just so easy to get things done in clojure. I love the language and hope to use it forever.
Clojure’s still the language I find myself turning to most frequently when given a choice. It just feels natural, the toolkit simple and yet powerful enough to express pretty much anything at a whim.
I’ve been using Clojure professionally for at least 4 years and these have been the most productive years of my career. I’m looking forward to using spec more fully. I’m looking forward to seeing what improvements come to the language in the next few years and have no desire to leave. Thanks so much for this language, community, and way of working.
This. Is. The. Best. Language. Ever. High-level programming done right. Thank you Rich.
Conclusion
When I went to write this years summary, I re-read the previous posts to remind myself of where we were a few years ago. While some areas have improved, I was struck by how many of the comments could have been lifted from previous years survey results. There was a worrying trend in the community and adoption sections that people perceive Clojure to be shrinking. Perception doesn’t always match reality, and it may be (as a commenter noted) that Clojure is entering the trough of disillusionment phase of the Gartner Hype Cycle. However I think there are a lot of low-hanging fruit around that could significantly improve the experience and ecosystem around Clojure without changes to the core language.
When I develop ClojureScript projects, I almost always use Figwheel. It’s a great tool, but sometimes in development I would end up with stale files. This led to some very confusing debugging sessions. It only happened some of the time, and was always fixed after a hard refresh. I thought about just disabling the browser cache, but I didn’t like ignoring the issue. After seeing colleagues struggle with stale caching, I decided to figure out what was going on, and fix it once and for all.
Cache-Control rules everything around me
The first thing to do was to add a Cache-Control: no-cache header to all static file responses. Despite the name, no-cache tells the browser it can cache files, but must always validate them with the server before using them. If the browser’s cached version is up-to-date, a compliant HTTP server should return a 304 Not Modified response, otherwise it serves the new file.
If you don’t provide a caching header to an HTTP response, the browser can choose its own caching behaviour. The browser’s caching heuristics are much more aggressive than you want in development, and lead to the weird caching behaviour I was seeing.
I thought this had fixed the issue, but occasionally I would still notice stale files were being used. After looking closely at the compiled output files, I made a surprising discovery.
ClojureScript copies file modification times
ClojureScript (as of March 2018) copies the last-modified date of ClojureScript source files to the compiled JavaScript target files. This is so that the Clojure compiler can detect changes to source files. JavaScript from the Closure compiler (e.g. goog.base), gets a modification time that matches the time it was compiled.
Using a Last-Modified date header for caching is often useful, but was causing two problems here:
Closure compiled JavaScript doesn’t change from run to run, so caching based on the last modified date will cause it to be unnecessarily re-downloaded.
JavaScript compiled from ClojureScript that uses macros from other Clojure files will receive the modification date of the consuming ClojureScript file, even if the macro files have changed more recently and triggered a recompilation. This was leading to the second round of (very confusing) caching issues that I saw.
To avoid these issues, I recommend removing the Last-Modified header from the response when in development.
ETags
To knock both problems on the head once and for all (hopefully), I added a CRC32 checksum based ETag to Figwheel for static file responses. I packaged this up in a library ring-etag-middleware so that other projects could also use it.
Serve 304 Not Modified responses
At this point the browser will check with the server for every ClojureScript file, on every pageload. However, this causes all of the files to be downloaded each time, even if they haven’t changed. The last step is to add ring’sring.middleware.not-modified/wrap-not-modified middleware. This will return a “304 Not Modified” response if the ETag provided in the If-None-Match request header matches the ETag header in the response.
Summary
As best as I can tell, this has completely solved all of the odd caching issues that I was seeing, while still keeping the app snappy to load by reusing as much of the cache as possible. If you are serving ClojureScript files in development and not using Figwheel, I recommend you follow these three steps:
Set a Cache-Control: no-cache header
Add an ETag to your static file responses
Remove the Last-Modified header
Wrap your responses in ring.middleware.not-modified/wrap-not-modified or the equivalent in your Clojure web framework.
I was excited this morning when I checked Hacker News and saw an article from Cockroach Labs titled “Your Database Should Work Like a CDN”. I’m a big fan of Cockroach DB and enjoy their blog posts. I clicked on this one, but came away disapointed. I don’t think it fairly contrasted Cockroach with it’s competitors, and instead made some incomplete arguments instead of selling Cockroach on it’s merits.
In this article I will analyse sections of the post and add more context where I think some was left out.
Analysis
Availability
To maximize the value of their services, companies and their CTOs chase down the elusive Five Nines of uptime
Only companies with sophisticated operations teams can seriously set an SLA for five nines. It’s doable, but comes with a heavy lift if your service is doing anything non-trivial. It’s certainly not the default position among the industry as far as I can tell. The Google SRE book has a good section on this.
For many companies the cost of moving from a single region deployment to a multi-region one is too great and doesn’t provide enough benefits. Some services don’t actually need to be available to five nines, and if not done well, moving to multi-region deployments may make your system less fault-tolerant, not more.
This is particularly crucial for your customer’s data. If your data’s only located in a single region and it goes down, you are faced with a “non-zero RPO”, meaning you will simply lose all transactions committed after your last backup.
If a whole region dropped into a sinkhole in the earth then you would lose all transactions after your last backup. However that’s not a failure scenario that we tend to worry about (although maybe we should?). Every time a major cloud provider has had a zone/region go down, data was partially/wholly unavailable during the outage, but no committed data was lost once service was restored (that I know of?).
…and Bureaucracy?
This impending legislation requires that businesses receive explicit consent from EU users before storing or even processing their data outside the EU. If the user declines? Their data must always (and only) reside within the EU. If you’re caught not complying to GDPR? You’ll face fines of either 4% of annual global turnover or €20 Million, whichever is greater.
I’m not a GPDR expert, but this seems like an incomplete summary of the GPDR rules around processing, especially as the processing rule applies only to the processing of an EU user’s personal data (AFAICT), and conflates requiring consent for storing/processing personal data with storing the data outside of the EU. From article 6:
Processing shall be lawful only if and to the extent that at least one of the following applies:
a) the data subject has given consent to the processing of his or her personal data for one or more specific purposes;
b) processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract;
c-f) [Public/government interest exceptions)
If you have entered into a contract with the data subject, and that data is necessary for the contract, then I think you are fine (IANAL). As far as I can tell, transfers to Third Countries are OK as long as there are appropriate safeguards.
This is the part of my critique that I am least confident about, I’d welcome some links confirming/rebutting this.
When you take GDPR in the context of the existing Chinese and Russian data privacy laws––which require you to keep their citizen’s data housed within their countries […]
If you follow the links in the article, you see this for China:
Does the Cybersecurity Law require my company to keep certain data in China?
[…] To that end, the Cybersecurity Law requires “critical information infrastructure” providers to store “personal information” and “important data” within China unless their business requires them to store data overseas and they have passed a security assessment. At this point, it remains unclear what qualifies as “important data,” although its inclusion in the text of the law alongside “personal data” means that it likely refers to non-personal data. […]
“Critical Information Infrastructure” providers are defined a bit more narrowly, but the law still casts a fairly wide net. […] the law names information services, transportation, water resources, and public services, among other service providers, as examples.
and this for Russia:
3.6 Generally, to transfer personal data outside the Russian Federation, the operator will have to make sure, prior to such transfer, that the rights of personal data subjects will enjoy adequate and sufficient protection in the country of destination.
Some companies will need to store data on China/Russian servers, but the laws here are far narrower than “If you have any data from a Chinese or Russian person or company, you must store it in their country”.
Managed & Cloud Databases
Managed and Cloud databases often tout their survivability because they run in “multiple zones.” This often leads users to believe that a cloud database that runs in multiple availability zones can also be distributed across the globe.
It might be an assumption you would make if you have no experience with databases, but it’s not one that I’ve ever seen a software engineer make.
There are caveats to this, of course. For example, with Amazon RDS, you can create read-only replicas that cross regions, but this risks introducing anomalies because of asynchronous replication: and anomalies can equal millions of dollars in lost revenue or fines if you’re audited.
I’m not sure how asynchronous replication lag could result in failing an audit and incurring millions of dollars of fines. I spent a few minutes trying to come up with a scenario and couldn’t. Losing revenue from users also seems speculative, I’m not really clear how this would happen.
Designing a system to run across multiple regions with asynchronous replication is certainly not trivial, but people do it every day. If they were losing millions of dollars from it, they would probably stop.
In addition, this forces all writes to travel to the primary copy of your data. This means, for example, you have to choose between not complying with GDPR or placing your primary replica in the EU, providing poor experiences for non-EU users.
Again, I don’t think GPDR requires this.
NoSQL
For example, NoSQL databases suffer from split-brain during partitions (i.e. availability events), with data that is impossible to reconcile. When partitions heal, you might have to make ugly decisions: which version of your customer’s data to you choose to discard? If two partitions received updates, it’s a lose-lose situation.
This paints all NoSQL databases with a broad brush. While some NoSQL databases are (were?) notorious forlosingdata, that is not inherent to NoSQL databases.
Certainly if you are using an AP NoSQL database, you need to design your application to correctly handle conflicts, use CRDT’s, or make idempotent writes. Partitions do happen, and it’s not trivial to handle them correctly, but neither is it the case that you always need to discard your customer’s data.
Sharded Relational Databases
Sharded Relational databases come in many shapes and suffer from as many different types of ailments when deployed across regions: some sacrifice replication and availability for consistency, some do the opposite.
I assume the author is referring to systems like Citus. I don’t have enough experience with systems like this to judge the assertion, but this seems fair.
Conclusion
If you do need more reliability/availability than is possible from a single region, then Cockroach is a strong option to consider. I think a far better argument for CockroachDB and against NoSQL, Replicated SQL, and Sharded Relational DBs is minimising complexity and developer time. It is possible for developers to design their application for the nuances of each of these databases, but it’s certainly not easy or cheap, especially if you want it to be correct under failure. The reason Google created Spanner (the inspiration for Cockroach) was that developers found it hard to build reliable systems with weak consistency models.
[…] It is much easier to reason about and write software for a database that supports strong consistency than for a database that only supports row-level consistency, entity-level consistency, or has no consistency guarantees at all. - Life of Cloud Spanner Reads & Writes
CockroachDB provides consistent reads and writes, supports SQL, is able to be deployed multi-region, and in any datacenter. Those are a compelling set of features. If your application can handle the latency and performance tradeoffs that it makes (which are getting better all the time), then it will let you write software against a consistent datastore without spending as much time reasoning about hard consistency problems. Cockroach is a great product, and I think it stands well on it’s merits.
Since Terraform v0.10, Terraform providers are distributed separately from the Terraform binary. This lets them update at different paces, and allows a wider group of people to collaborate on the providers. This is mostly good, but it does introduce a new step for upgrading providers. It is slightly counterintuitive, but to upgrade your providers, run
I was building VM images for Google Cloud with Packer, and provisioning them with Ansible. Everything had been working in the morning, but in the afternoon one computer wasn’t working after I had upgraded Ansible with Homebrew. I was having a really tough time figuring out why Ansible and Packer were running fine on one computer, and not on the other. I was getting the following error:
googlecompute: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"127.0.0.1\". Make sure this host can be reached over ssh", "unreachable": true}
After checking the versions of Python, Ansible, and Packer on the two computers, I found one difference. On the computer that wasn’t working, when running ansible --version it had a config file listed:
I moved that file to another location so it wouldn’t be picked up by Ansible, and after rerunning Packer a second time, got this error:
googlecompute: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '[127.0.0.1]:62684' (RSA) to the list of known hosts.\r\nReceived disconnect from 127.0.0.1 port 62684:2: too many authentication failures\r\nAuthentication failed.\r\n", "unreachable": true}
I then used the extra_arguments option for the Ansible provisioner to pass [ "-vvvv" ] to Ansible. I ran this on both computers and diffed the output. I saw that the dynamically generated key was being successfully provided on the working computer (after one other local key). On the failing computer I had many SSH keys that were being tried before I could get to the dynamic key, and I was getting locked out.
SSH servers only allow you to attempt to authenticate a certain number of times (six by default). All of your loaded keys will be tried before the dynamically generated key provided to Ansible. If you have too many SSH keys loaded in your ssh-agent, the Ansible provisioner may fail authentication.
Running ssh-add -D unloaded all of the keys from my ssh-agent, and meant that the dynamic key Packer was generating was provided first.
I hope this is helpful to someone else, and saves you from hours of debugging!
Postscript
I was very confused by seeing that my computer was trying to connect to 127.0.0.1, instead of the Google Cloud Platform VM. My best guess is that Packer/Google Cloud SDK proxies the connection from my computer to the VM.
While working on Deps I wanted to detect the user’s time zone at signup, to localise the times that I presented to them. I hunted around and found a variety of libraries that offer time zone detection, like moment.js, and jstz. However I wasn’t keen on pulling in a dependency on one of these libraries if I could help it, so I kept looking. I also considered trying to detect the user’s time zone from their IP address using something like the Maxmind GeoIP database, and that probably would have been the method I settled on.
However in my research, just before I started with IP based detection, I found that there is a newish Internationalization API, which will give you the user’s system time zone.
All current browsers support the Internationalization API including IE11. However there is a slight wrinkle to keep in mind when using it: on some older browser versions that do support Internationalization (IE <=11, and Firefox <=51), the timeZone property is set to undefined rather than the user’s actual time zone. As best as I can tell, at the time of writing (July 2017) all current browsers except for IE11 will return the user’s actual timezone. If you do need to support older browser environments then you could look at moment.js. It uses the newer Internationalization API when it is available, and falls back to other heuristics if it isn’t. For my purposes, my browser statistics show almost no IE usage, so I chose to just use the newer method, while also allowing the user to manually update their time zone if they needed.
