A Diverse Drupal Community Needs Composer
The Drupal community has been trying to come to terms with modern build tools for a while.
We have Drupal core which is managed with Composer, but we also have contrib modules and some of them want to use Composer, too.
The canonical example is Address module. It requires an external PHP library, necessitating the use of Composer to add it to the codebase.
At least, that's the thought.
There are a lot of Drupal people who don't want to use Composer to build out their Drupal. These people have a lot of different reasons for why this might be, and that's fine. I really don't understand those reasons, but when trying to solve the 'hard problem' of Drupal Composer, we'll take it as a constraint that these people's needs matter.
Because they do matter.
But it's also true that the Composer-friendly people have needs, and they matter too.
It's a fact that Drupal's Composer stuff is incomplete. There are some internal consistency problems and there's no defined behavior for how a Composer-based Drupal core is supposed to work, so we can't test for that behavior and guarantee it.
That leaves the Composer-friendly people having to build elaborate work-arounds to suit their needs. In the Drupal world, such work-arounds turn into best practice quite quickly, which is a problem.
So if you accept these two, diametrically-opposed viewpoints as intractable and irreconcilable, then... It's intractably irreconcilable.
Another Way
Let's stop thinking in that way.
What we have is two user stories here. I think there's a few others that are missing, but I'm not going to list them because I'm about to show you how we can solve them.
I think these two user stories are not a dichotomy. I think that sometimes a Composer-friendly person might want to point and click to build a Drupal site. I think that a Composer-averse person might eventually learn how to use it. I think these two people are really the same person, when it comes right down to it.
That's why we should not talk about how to make Composer work for both camps, because no one really wants to or should have an opinion about Composer.
What's needed here is not a Composer policy. We need a Drupal codebase build policy.
We should make a build tool for Drupal. It will have a dependency on Composer, because one of its tasks will be a Composer phase, but it won't be Composer.
Existing Build Tools
Let's think about what build tools we already have for Drupal.
We have Composer as a build tool. You can use Composer to build out a Drupal codebase. This is the goal of drupal-project, and it's also what the DrupalCI testbot does to test core and contrib.
We also have Drush Make. Drush is a command-line tool that can take in a file with a list of dependencies and build a codebase. It used to have a Composer component, but I don't think it does any more, so when you build a codebase with Drush make, you still need to use Composer.
But the important thing about Drush Make and Composer is that they are not core. This means that, for instance, Drush can go out of sync with core compatibility, and can be broken for your use case. If you rely on Drush being available as a global executable, for instance, then you're in for trouble because they changed policy on that. This isn't to say it's a bad policy, only that it changed and thus doesn't guarantee a workflow.
Core should have a codebase build policy, and a tool which implements this policy.
DrupalBuild
I propose a command line Drupal build tool not unlike Composer, but within the core codebase. This way it will always be tested alongside core. It will be able to handle all the special cases for Drupal, and if you want a feature in it or find a bug, you can file an issue in core and it will be fixed in that context.
Until a better name comes along, we'll call this DrupalBuild.
DrupalBuild will be part of a subtree split, not unlike the Drupal Components. This will allow it to have two main modes of access:
- From the command line within an existing Drupal codebase, at something like
./vendor/bin/drupalbuild - As a PHAR-type executable.
Drupal.org will build the PHAR tool version. It will be downloadable from drupal.org in a way similar to Composer.
The instructions for using this tool will be like this:
- Go to the command line.
- curl the PHAR file to your machine.
- Or type
composer require --global drupal/drupalbuild. - Type something like
./drupalbuild.phar --project standard.
This tool will then figure out which other tools are needed, such as Composer (and maybe yarn? and maybe SASS? and maybe babel? and maybe others?) and then execute them.
I STILL HAVE TO LEARN COMMAND LINE STUFF?????
No. Hang on. Let me finish please. :-)
DrupalBuild Part 2: Buy-In Needed
DrupalBuild will be a tool that everyone wants to use. People who are now using a bespoke script will be able to use it. People who are looking at the command line for the very first time will be able to use it. Your site builds that are from two years ago that move all the files around? Forget those. You're using DrupalBuild now.
This is vitally important. The whole project succeeds or fails on whether command-line people will use this tool. If they don't, then no one will maintain it and then it will be about as useful as core/authorize.php. (Did you even know that file exists?)
If no one is using this tool, then the non-command-line-people will suffer. Their toolset (which I'll get to... be patient) will fall into disrepair and few will have the enthusiasm to work on it.
So: Design constraints for DrupalBuild include being useful for the command-line-happy user story, and also being tested alongside core for every commit.
But We Hate The Command Line
OK, OK, hang on... almost there.
DrupalBuild Part 3: Distributions And Out-Of-The-Box
DrupalBuild would give us an opportunity to think about how distributions work.
Currently, Drupal allows you to make an installation profile, which specifies a Drush Make file and some behavior which will build out your site in a specific way.
If you publish a profile to drupal.org in a certain way, we call it a distribution. The distribution is a ready-to-go Drupal-based application.
Now it just so happens that there is a thing called the Out-Of-The-Box initiative. This is a demo Drupal distribution that hasn't quite been finalized in terms of technology.
Well, we could use this nifty new build tool to specify our Out-Of-The-Box distribution. A user could say: ./drupalbuild.phar --profile outoftheboxdemo or something similar and be magically given a demo codebase.
Helloooooo? Command Line?
Thanks for being patient.
Graphical Build Tool That's A Desktop App
How's this for a first-time Drupal installation instructions list:
- Download a desktop app.
- Run the desktop app.
- See a screen that says "Do you want to build a Drupal? Or would you like to see the out-of-the-box demo?"
- Click on a button.
- Receive a tarball, or a codebase wherever the user specifies.
- Maybe even see the site live in the app.
I like that spec. That's an easy spec to like.
See that desktop app? Let's say it's built in Electron. Let's call it ElectronicDrupalBuild (EDB).
It's built with it's own copy of the DrupalBuild PHAR in it so you can use that instead of a local version. It can query drupal.org to get a list of available extensions and themes and profiles. It also has a Composer PHAR (and whatever other tools we decide) so that it can do the build without thinking too much about the command line.
You clicky-click on extensions you like and they show up in your codebase.
Want to know what this might look like? Run the Atom text editor and search for available extensions. Atom is built in Electron. Electron is an app framework, and you code for it in HTML and JavaScript.
So you build a codebase with ElectronicDrupalBuild. You can get whatever granularity you need from it: All the way from a dead-simple tarball to just a composer.json file.
Then you manage it, too. You point EDB at your site's composer.json file and root directory and it figures out the rest. It tells you whether there are module updates and things like that. It gives you an option to update your codebase, and then maybe an option to update your site.
It can do all this because it's using the command-line DrupalBuild tool that does all this stuff. The command-line tool is actually doing all the work, while the pretty Electron window makes everyone happy with UX.
Moving Forward With Your User Story
See how far away we moved from the two Composer-centric user stories? They were always a false dichotomy.
As the so-called 'beginning' user learns more and more, maybe they're OK with abandoning ElectronicDrupalBuild. They might migrate towards the command line. You know what? They still don't have to learn Composer. That's because they're going to learn a tool that's better suited to the task of building a Drupal site.
Since this is part of core, we could state that, for instance, if a contrib module needs a grunt command to build its SASS into CSS, there will be a way to specify it for DrupalBuild to take care of it.
Things like that.
As another example, we currently have a problem with managing JavaScript libraries, similar to the Composer problem. Could we centrally locate these libraries and make sure there aren't version conflicts between them? Perhaps, if we have a core-sanctioned build tool with enforceable rules, tested performance, and a motivation for documentation.
And That's It
The summary goes like this:
- Community buy-in is the most important 'hard problem' here.
- We can make a build tool that everyone uses and that has building core as its only priority.
- We can wrap this build tool with a nice pointy-clicky interface.
- This tool could enable other initiatives.
Thanks for reading this far.