A Modular Monolith in Laravel Lumen

• A Modular Monolith in Laravel Lumen
  • What’s not in Lumen
  • A Ghost in the Shell
  • MaaM: Module as a microservice
  • Packages all the way down

¶A Modular Monolith in Laravel Lumen

If you use Laravel, you’ve probably heard of Lumen. In case you haven’t: Lumen is the micro-framework to Laravel’s full-stack.

Lumen is meant to be fast. As far as PHP micro-frameworks go it’s probably not far off being the fastest. Compared to other micro-frameworks, it probably fares pretty well.

This is made possible thanks to its default prerequisites being a lot slimmer than Laravel’s and it differs under the hood in some fundamental ways. But developer experience still feels very much like Laravel, which is an important trick to pull off given the main framework’s popularity and ease of use.

Basically, if you’re familiar with Laravel, you should ‘get’ Lumen. Having said that, there does seem to be quite a few in the community who try Lumen and seem to really miss certain Laravel features, but I think they’ve maybe missed the point.

At Elvie, we decided last year to build the next version of our API in Lumen. As a trimmed version of the framework that we’ll be using for most of our web tools, it’s a good fit as it keeps much of our stack familiar.

Personally, I think Lumen really shines 😉 in its opinionated decisions in the context of building a stateless, viewless HTTP API.

Over the course of possibly a few articles, I want to share our approach and some of the key things we’ve learned about building this new API.

I should underscore all of this with a few details first though:

• We’re a small team, I’ve basically coded the main part of the API myself – some of this may not be suitable for larger teams, but I’d be interested to get some input from those of you on larger teams: do you foresee any challenges that may arise due to our implementation?
• Before me, our CTO coded the previous version of our API basically from scratch in his own framework. That framework doesn’t have quite the same popularity, documentation or accessibility as Laravel. It also lacks some of the things that make writing modern PHP applications enjoyable. So there’s some legacy in this move that has shaped our decision-making but that may not be applicable to yours.
• We have a fairly common (RESTful JSON API), but specific use-case (our workload isn’t super high). I hope you love our concept and I wish it would work for everyone, but please consider this when reacting… I’m not recommending everyone change everything just because it works for us/they like the concept. Caveat emptor
• I’m incredibly thankful for the immense prior work put in by others far smarter than me who have made this possible. Without their effort, guidance and example, this would have been much harder to pull off!
• If there are any parts you’d like me to explore in more detail, please let me know and I’ll prioritise writing them up separately when I have time.

¶What’s not in Lumen

Let’s get this out of the way first because these are some of the typical things folks want to do when they start building an API.

The first and most important part of our API that’s not actually built into/on it directly is the authentication.

We spent quite a bit of time thinking about this and decided to go whole hog and set up a completely separate OAuth server to be our master authentication service. This is built on top of Laravel Passport and it is a completely separate Laravel (yes, not Lumen 😱) application.

There are some key reasons for this, but the most important is that Lumen isn’t really intended to be used to run an OAuth server, so don’t try to do that. I mean, you can… but I don’t really know why anyone would. Lumen lacks too much to make a complete OAuth server without adding a bunch of stuff that it intentionally lacks… and that Laravel comes with out of the box.

This is because, by default, Lumen lacks Views and other goodies. But if you’re building a stateless, JSON API, views are not for you. If you need to output some HTML, you still can, but if you want Blade, for the love of all that is pure and sacred, use Laravel.

Edit 5th September 2018: Just this week we came across the need for Mailables and I have to admit I was surprised to find this missing from Lumen. But when I think about it, it does make a certain amount of sense. And actually it wasn’t too painful to get Mailables up and running in Lumen.

There’s probably a huge list of other gripes that people have with stuff that Lumen lacks. I actually don’t want to go into those because as it turns out, we haven’t hit any of them. I can only assume that this is because we’re using Lumen in the way it was intended.

If you need Laravel, use Laravel.

If you’re tempted to think that you’re on some sort of “I need more than Lumen but less than Laravel” middle ground, I strongly urge you to reexamine your life choices.

Aside of what a typical Lumen install doesn’t have by default, there are some other things that our Lumen instance intentionally lacks:

• routes
• models
• controllers
• migrations
• config
• interfaces
• traits
• basically any real business logic at all.

Wait… what!? How exactly are we using Lumen then?

Well, I like to think of Lumen as the shell.

Turtle power

¶A Ghost in the Shell

After we started building, Dan Manges wrote an interesting article on an approach he termed The Modular Monolith. I think we’re on the same path here, spiritually. I would really recommend you having a read of his article at some point if you haven’t already.

Ok so you’re probably wondering where our API is and if I can really justify writing about Lumen at all if this supposed Lumen app is basically empty?

Well of course all of our API is in Lumen, I’m not lying. But strictly speaking almost none of it physically exists as part of a single, concise Lumen installation – it exists as packages.

This was probably the most exciting idea in this whole project for me because I believe this particular design choice is incredibly powerful.

Why build things this way? And how can we do so with Lumen?

The ‘why’ comes down to how our old API was structured and how our new one will be developed and consumed. We had a few issues.

Our old API has this concept of modules that are grouped units of functionality centred around areas of concern and/or specific client applications.

Each module contains its own functionality but shares common parts of the outer API with other modules, such as authentication, validation, DAL, request/response handling and serialization.

Our first-party clients typically only access one of these at a time. So in a way it would be easier to think of them as separate applications.

The thing is, they’re not; they’re all part of a single application and, significantly, the same git repo — it’s a monolith! And perhaps worse: it doesn’t use Composer 😱

This presents some challenges when building on the API. Even as a team of just two devs, tagging and deploying can get kinda hairy. Any change to a single module is seen as a version bump for the entire application.

It also creates annoying brain work around deciding/debating whether a breaking change in one module should be reflected in a major version number bump across the entire platform.

It’s then impossible to use semantic versioning as an external indicator of the API’s version. So we basically have to have two versioning methods and there’s absolutely no parity between them.

This is quite jarring and a bit tricky to get your head around when client applications are looking at a version number that rarely changes while your internal version numbers are changing wildly. There’s just a lot of potential for weirdness there that I really don’t like.

Wouldn’t it be nicer if we could work on each module as a separate project?

“Yeh, definitely!”

Here’s how we settled on packages.

¶MaaM: Module as a microservice

This is possibly the go-to solution. And it’s obvious why — it fits so nicely with the idea of separate projects for each module, it ticks the popular opinion box, it feels like any new team members would just ‘get it’ and it would be super easy to extract our existing code out into discrete applications.

This is a common approach and verrrrry flexible. It kind of feels right, but at the same time also sort of wrong. It may solve some problems, but it brings us different ones.

For example, any change we want to make to the common features of our ‘API’ (here representing the whole collection of microservices) requires each service to be updated individually, basically repeating a bunch of changes in each one. So long, D.R.Y.

And you know for us it just seemed to add a whole bunch of cognitive load. We’re a small team. Our ideal right now is a single application. One deployment (or as close to one as we can get for as long as we can). One thing to update. Less room for error. Simpler development and testing.

• We don’t need separate services for all of these parts right now, at least not for the scale.
• Multiple services means a lot of extra copies of dependencies and opportunity for stuff to get into inconsistent states and possibly cause unexpected side-effects.
• It would add a lot of extra work at different stages of development and deployment which we don’t have time/resources for yet.

Source unknown

Riffing on that a bit, we figured we could make a package that represents the common parts and have each microservice import that package.

That’s not bad. It means any change to shared functionality only needs to be made once.

But even then, when we have multiple microservices out there, we’ve got to make sure they all use the same/compatible versions of the common package otherwise we might end up in a pickle.

The next logical step seemed even better: instead of full-blown micro-services, why not set up each module as a package?

This is a really simple concept that’s super powerful and flexible. But it’s by no means a new one…

Let’s have a quick diversion for a bit on how this is possible.

Laravel (and by extension Lumen) and many other frameworks include a concept for allowing external code (typically from third-parties) to hook into the framework and extend it.

In Laravel, these are Packages. For Symfony, it’s Bundles. Most others use the word Plugin. And I’m sure still others have come up with even more creative/obscure alternative names.

They all boil down to basically the same thing: a way for developers to build reusable libraries of code that other developers can (hopefully, ‘easily’) drop in to their applications to quickly add new functionality.

Usually there’s nothing stopping you from building your apps without any. But the complete opposite is also true too: some developers like to build their entire applications out of packages/bundles/plugins.

In the Symfony ecosystem, there’s been some change of heart on this: it used to be recommended to build your app this way, even if none of your Bundles were being reused. Now they’ve stopped making that recommendation and suggest that only your reusable code need be rolled up into Bundles.

You need to decide what’s right for your application, but in this case it felt like the right fit for us. So we decided to go all the way with this and build each module out as its own package. That way each package can:

• be its own repo, versioned and tagged separately;
• can be worked on, tested and deployed in isolation or in combination by multiple people in multiple ways more easily with less chance of breaking and ruining someone’s dinner;
• contain ONLY the code that relates to their specific areas of concern and nothing that isn’t relevant to them.

Laravel has really great support for packages. Lumen inherits a lot of this for free too. Using Composer, we can easily require our own packages, which in turn can have their own dependencies.

In production there’s less duplication and more consistency, but we still have a lot of flexibility.

Package-specific config, migrations, routes and more are all contained in the package and loaded dynamically using a Service Provider and the dependency injection container!

It’s super slick and it works really well. Plus it covers all our acronym bases: DRY, KISS, and yes even CRUD!

And the actual API application:

• can stay as one application — new developer onboarding is super easy;
• is the one true place to update all the things (we just do one composer update to update everything);
• still has the flexibility to become individual microservices super easily if we ever need to break off one or more modules and are ready to take on that challenge!

Ok, so we’re going to build out a package for each module. Decision made! w00t!

Taking a step back though, it was clear that most of our modules — now called Suites — would still share some common code. And if we were going to have really lightweight projects for development, we’d need a way to boot up some parts of our full Lumen application for testing and such.

¶Packages all the way down

It feels kind of natural at that point to jump back to the main Lumen app and start adding a bunch of base classes there and be done. But that poses a big problem: if all of your hooks between packages and the application exist in the application codebase, we’re really tightly coupling all the pieces making up this monolith-esque thing again, making all the effort of breaking our modules out a bit redundant.

This is when it became clear that we actually needed another package. This one would be shared by our Lumen shell and each Suite: our Support package.

Very much like Laravel’s own Illuminate packages, our Support package contains a bunch of base and abstract classes, interfaces, traits, events etc etc.

By making it a Composer requirement of each Suite and the API shell, we can make use of Composer’s built-in version management to work out safe deployment paths for production and identify issues with incompatible versions early on in development.

This allows us to keep our Composer dependencies for each package down to the absolute minimum required.

It also encourages us to version our packages properly and use appropriate version constraints in our composer.json files.

The Support package includes everything needed for testing against the Lumen stack so it’s insanely easy to get going on some meaningful work. There’s no arduous set up of a massive monolithic repo with multiple databases and too much config.

If you’ve got nothing on your machine but a basic dev environment (Git, Sublime, PHP, Composer) you can pull down a single Suite and get going — you probably don’t even need a database!

Having a Support package also has some other nice side-effects. It gives us a good place to store a bunch of other one-off and superfluous items that would otherwise sit in the Lumen app. This is a good way of looking out for ‘future you’ in case we find a way to reuse them or they come in handy for testing.

Thinking about and organising your packages carefully goes a long way to building and managing multiple, linked projects that much easier.

It also allows us to type-hint against our base classes and such from within Lumen and our Suites to give us greater guarantees around consistency and expected functionality. That’s pretty sexy.

I’ve got to be honest, I learned a lot from the Laravel codebase in this regard. It’s not just a good framework, it’s incredibly well written, made up of many discrete parts that play well together.

Whew! That feels like quite a lot to process. Perhaps we’re at a good point to draw a line under this for now. Why not have a breather away from whatever screen you’re looking at. Check on your dinner, play with the dog, or just go stare out the window and dream of Paris.

In case you do come back, if you’ve got any questions, comments, criticisms, or insights, I’d love to hear them. And if you’d like to read more about how we’re working with Laravel and Lumen at Elvie, I’ve got loads more to share, so please give this article a clap so I know you’re interested in reading more about this.