The secret history behind the success of npm and Node

When JavaScript was first introduced, it was a programming toy that sort of worked, but worked differently across browsers. A few pioneers, though, saw the potential this language could have on the back-end as well, where it has transformed the application development landscape. We caught up with CJ Silverio, CTO of npm, who explains why npm and Node have grown incredibly popular. Not only can they build better back-ends but they’ve allowed for a new ecosystem of JavaScript frameworks with React on the front-end as well.

What were the factors that drove the success of NodeJS?

CJ Silverio: Around 2009, our founder Isaac Schlueter was excited about server-side JavaScript. About the same time, Ryan Dahl and Node.js were starting to make an interesting run into this area. The idea was to pull JavaScript out of the browsers and run it on computers outside of the browser context. Dahl wanted to write a fast Web server on JavaScript and ended up creating a useful general platform as a result. Schlueter got excited about this, and he realized was that there needed to be a way to share code.

Ruby has RubyGems, Python had Python eggs and Perl had CPAN. Schlueter started working on a package manager for Node. At the time, it was one of several competing package managers for JavaScript. But it happened to have the right feature set and also was written in Node.

So, for years as Node grew, npm became a way people published JavaScript to a shared repository, the npm registry, to build a program. It described a package using a JSON manifest for pulling the things you wanted from the npm registry as shared code you could use to build software.

What was different about npm compared to other JavaScript package managers?

Silverio: It introduced two innovations around semantic versioning and dependency management.

How did semantic versioning work?

Silverio: Packages all have versions. Semantic versioning is like a pinky swear among programmers about what they intend the version to contain. A package has three numbers. The last is the patch number. You pinky swear, “I think I’m fixing a bug.”

The middle version number implies, “I think I’m adding a feature.”

The first number is a major version update which asserts, “I’m breaking you.”

This is not a panacea, but this is just enough communication among programmers about what their intent is. You break people by accident all the time, but semantic versioning lets you signal it.

This turns out not to be a solution to everyone’s problems, but it is enough grease on the skids that it helps.

How did npm address dependency challenges?

Silverio: Dependency hell is a term people throw around. For example, you are trying to depend on one piece of software and something else you depend on also depends on a different version of the first piece of software. This is a much harder problem in Java, C, and C++. Anything that is statically compiled tends to have this problem, which basically mandates there will only be one version. For example, if I want to use OpenSSL, it’s not possible to have two things that require different versions of it in statically compiled languages.

Node includes a mechanism for working around this by nesting dependencies. It does not matter what version they want. Two modules can call different versions of a package, and it is fine they coexist.

How did these catalyze back-end development?

Silverio: These two things were useful enough for people to begin sharing packages on top of npm, and Node started to succeed. Schlueter was working on both npm and Node at the same time. As Node began succeeding, people started writing more JavaScript and sharing more of it as well.

And then the browser developers, who really write most of the JavaScript in world, found that Node would let them use the right tools in the languages they knew well to write websites better.

That is when npm and Node went kaboom.

JavaScript is a massively popular programming language, because it is in the browser. It is a tool that makes writing things in JavaScript easier and can share the JavaScript you were writing, so web developers started using npm a lot.

Our COO, Laurie Voss is into numbers and enjoys producing graphs of npm’s uptake and use. It is not exponential, but it looks exponential.

npm users downloaded 5 billion packages in the last seven days. Four years ago, it was fifty million in one week. There are a lot of developers that share a lot of JavaScript and have developed habits that assume npm is available as a resource. Anytime you’re developing, doing automated testing, running build environments, they will pull in new packages.

There were 50 billion Tarballs downloaded in the last week. The majority was probably continuous integration testing. When people are actually deploying a new app, they will download it once to make a new build.

How do bundling browser apps and compiling back-end apps compare?

Silverio: Webpack and Browserify are used to bundle packages together for browser apps in a way analogous to compilers for other kinds of apps. All browsers run JavaScript but in a different way than people tend to develop programs. Developers make little modules with lots of little files. When you deploy to a website, you have to shove everything into a single file. People use bundling tools to do this. A program called Browserify was one of the first to do this.

The one everyone uses today is webpack. Another is called Rollup. Bundlers do a function that parallels what compilers do. They take source code in one form that is convenient to developers and turn it into the form best suited for browsers to consume.

Browser makers are doing well right now in terms of compatibility among JavaScript implementations. Webpack does not have to make a build for Microsoft edge then for Google Chrome. This same compiled JavaScript will run on both of those. This is one reason that JavaScript development is flourishing. There really is one JavaScript, and one browser object model they are all using.

It is just that the Node servers are different than browsers. This is okay, because they are different things. You end up wanting to use the conveniences of Node when you are developing.

What is the secret horror of working with NodeJS?

Silverio: The secret horror of NodeJS being server-side JavaScript is that the two languages are the same, but the platforms are different. Running a Node program is almost the same as running JavaScript in the browser. But the environments are different. Node can read and write to a file system. For example, you can have native bindings to Node for a computer vision library. These are things that you cannot do on a browser.

This program might not run in the browser. It might run after you do some transformation to make all the nice things you get from Node available in the browser equivalent. That is what tools like webpack do.

How does npm make money from giving away free access to a package manager?

Silverio: We sell a version of our registry you can run on-premise, and soon we will announce this as an npm registry as a service. We also sell a security product which is another thing that matters to a lot of people. The downside of open source is that it also includes a lot of insecure libraries. npm is a place where people can share JavaScript code. The next question that I as a CTO ask is “where did you get that code from and is it safe to use?”

npm acquired ^Lift Security and these people have been experts behind Node security projects since its inception in 2013. This has grown our expertise on JavaScript security for finding and fixing JavaScript security and making decisions about whether what you are using is safe. We have a free version where every person using npm will get access, and when they install something using npm, it will tell them how to fix it.

Deeper analysis and reporting is available as a paid offering. It can also do more analysis over time across projects. For example, we could tell you when a particular vulnerability entered your application, and what your exposure is over time.

I’ve developed Node apps that update database rows and columns that ran for years. The fact that people are developing websites and development is constant.

How has the mix of front-end and back-end developers shifted?

Silverio; We have about 11 million developers using npm. Most of those are front-end browser developers using Node to do it. Many are using Node but sharing code that might have nothing to do with Node.js. For example, React is hosted on npm. If you want to get the latest version of React, you will run and install it using npm, Yarn or a CLI command that talks to the npm registry.

I refer to it as success as a catastrophe. You have a startup and now you have succeeded. npm has survived a catastrophe of success, because we have made really good use of our CDN.

Currently, npm usage is about 80:20 front-end to back-end developers. Two years ago, it was about 50:50. Node is critically important and is being widely used to write single page web apps.

npm being where it is and as usable and fast as it is has enabled people to do new methods of web development they could not do before. By allowing the plumbing to be taken for granted by developers, it has allowed people to build things on us that were not thinkable before.

When you got JavaScript by doing Google searches and going to a jQuery webpage, you could not build complex things from many small modules, because there was no easy way to find them and assemble them in a reasonable way. But tools like npm and Node make that not necessarily straightforward, but at least possible.

Semantic versioning is not perfect, but most of the time it works. People can just take the software with confidence and npm retrieves the components for them automatically.