The running costs of eLife 2.0

We use Amazon Web Services (AWS) for hosting Continuum and the total monthly bill for running the live services and bandwidth is around $1,710.

This can be broken down as follows, and also includes pro-rata costs for reserved instances (i.e. we pay upfront for 44 instances to get a 40% reduction in cost, but share those out between running live services, development machines and data science activities).

You’re probably now wondering what you get for that outlay. Other than the flexibility and agility that running your own publishing platform provides, those computers manage to publish around 120 articles per month and that bandwidth helps serve 500,000 visitors per month too. We handle peaks in excess of 300 requests per minute on eLife, with an average of 200 – that equates to roughly 100 simultaneous visitors on the site. Our highest peak was just over 1,000 concurrent visitors (when the Homo naledi discovery was first published on eLife), so we have load tested the site to 10 times that amount and performance remained acceptable.

We over-provision the number of machines we use for two reasons:

1. Redundancy – we try to eliminate single points of failure in our live services. There are at least two machines running key services and all databases have a standby copy, ready to go, in a different part of Amazon’s data centre. This makes deployment of new features safer and more seamless as one machine can be updated while another continues to serve traffic. This is especially important for us as we deploy changes up to five times per day.
2. Low-touch scaling – we don’t want the journal website to degrade in performance when we have a spike in visitors, which is unpredictable. While we could build complex elastic scaling into the platform, we’ve opted for simplicity for now: additional machines. Since computers are cheaper than people (and don’t need to rest), this means midnight spikes are coped with and we can all sleep soundly.

As you’ll see in the more detailed section later, although we have spare capacity we still only take the smallest and least expensive types of machines from Amazon for most of the services. You can see how the response time only increases slightly when the number of requests increases:

With Continuum and eLife 2.0, we have taken care to justify each byte, meaning that every new library, stylesheet or piece of site furniture is evaluated as to whether the potential benefit is likely to outweigh the performance cost. This approach, coupled with the use of progressive enhancement, helps to reduce the amount of bandwidth needed to download the the pages and their assets. For example, today’s homepage downloads completely with around 1Mb of data used. Contrast this with other popular journals’ homepages that average over 1.7Mb, with one example downloading 11.2Mb for every fresh visitor to the site.

Most of the data transfer cost is in the use of CloudFront, Amazon’s content delivery network (CDN) service. CloudFront stores cached copies of our web pages and related files, including the figure and PDF assets, and we pay for the amount of data that is transferred to our visitors’ respective browsers. This is a common way of charging for CDN services, and is a cost most hosting options will have to bear. Following good practices to minimise the size of pages, images and related files therefore improves the user experience and saves on the cost of hosting too.

There are now five full-time developers working in the eLife Technology and Development team, along with myself, our Head of Product and Data Scientist. We don’t spend much time maintaining the system, though. Instead, we outsource that work to the digital consultancy that helped us initially build Continuum. We reserve 10 hours of time from their development team every month – but we rarely use it, as the system is stable and the quality-driven software practices we followed when building the system mean that actual software defects are rare and are quick to resolve if they do occur. There are often minor changes needed to the system, however, when new types of data are encountered, and this is the most common type of work conducted under the banner of ‘maintenance’.

Our developers do get involved when there are incidents, but spend most of their time improving or adding features to Continuum. We are happy to speak to anyone who wants their team to contribute code to Continuum, so please get in touch if you are interested. Incidents the team have encountered are usually a result of the evolution of the system with new features rather than major errors in the system.

I’ll not detail the cost of the software consultancy we work with here, since it will be very variable depending on location. If you need a comparison though, you should look for a software developer with skills in PHP or Python, and basic knowledge of AWS, with availability of between five and 10 hours a month depending on your volumes.

Since Continuum was launched, we have spoken to people interested in the platform for publishing content for a broad range of reasons: from departments in educational institutions wanting to self-publish their findings to continent-wide archives; and from small societies to the world’s largest publishers. This is why we have specifically architected Continuum to be scalable, and when we talk about scalability we mean scaling down just as much as scaling up. We’re also aware that not everyone will want all of the services Continuum can offer, so I’ll be as granular as possible in the detail of the ‘computer instance types’ and services we use so that you can consider your own use case.

For those who want a quick estimate, let’s explore some different scenarios.

The variables that will most influence the costs with Continuum are the number of visitors, number of articles published per month and the levels of resiliency you’d like to build into your hosting infrastructure. For eLife, these values are:

• Visitors: 500,000
• Articles published: 120
• Resilience level: good

Having more visitors will increase bandwidth usage in a proportionate way, and compute power will increase in a more stepped way. Publishing more content will also increase compute power needs, as well as the amount of storage/messaging needed.

I’ve calculated a few examples, based on proportionate growth with each of the variables and by estimating the compute power needed for each. I’ve assumed that bandwidth grows linearly with visitors and storage/messaging grows linearly with publications. Here is a summary of the variables and estimated costs for each potential use case:

You can see the data table and the values used for estimates in this simple calculation spreadsheet. Try copying the sheet to perform calculations for your own use case. Let us know if you create a real-world example for yourself – I’d be interested to publish updates to this model in the future.

One variable not included here is the people cost. Keeping any website running smoothly will require some sporadic effort, so you cannot remove that cost completely, even for a minimal setup, as it still needs to be available 24 hours a day. Similarly, a highly available, fault-tolerant system supporting millions of visitors is likely to need different skills or more people, so the variability is hard to predict. The cost should still be considered in your own evaluation.

As mentioned, we run eLife on infrastructure with resilience in mind. It can be run on far fewer compute instances, with databases that only have one copy (and regular backups for disaster recovery), and you could try and reduce bandwidth costs by using a cached CDN.

Running multiple services on a single instance is possible with the current architecture:

It would be best to segregate services based on their usage type, so we wouldn’t recommend combining two services that require heavy-disk access (like the Image Server and Journal CMS). You could also minimise the number of services you use by not having Recommendations or only using Journal CMS for basic list administration to minimise its footprint.

To explore the costs of running Continuum in greater detail, particularly for Heads of Technology, CTOs, CIOs and Lead Developers, the sections below cover the instance types and specific AWS services we use.

To get started, AWS provides a useful calculator to predict the costs of using its services, but you could also run Continuum on your own hardware or in other popular cloud providers like Google Cloud Platform or Microsoft Azure (which also provide calculators). Note that the Continuum scripts for automated deployment are currently designed for AWS, but we are working on using containers (with Docker) in the future that should ease deployment to any infrastructure.

Here is a table of the actual instances running eLife right now, grouped by instance size:

In addition to the compute instances, we also have instances powering databases using the AWS RDS service. This service takes some of the administration out of running a production database, such as hardware provisioning, database setup and patching and backups. It also provides a multi-availability zone copy of our database for the purposes of redundancy. We use Postgresql where possible for Continuum, but also support different databases for other applications including CRM, data science and reporting.

There are other services that contribute to the cost that are worthy of note.

Caching is handled by the CDN at the most external level, but we also have caching layers between services, most notably between the journal web app and the search service using Redis, provided by the AWS ElasticCache service.

Communication between services that use an event-driven architecture is handled through SQS and SNS. These, although at a negligible cost, could have an impact on high-throughput journals.

The Amazon Simple Workflow Service is a very cheap yet effective service that provides orchestration of jobs and tasks in the ‘eLife bot’, the part of the platform that does publishing tasks. It basically runs a number of small Python scripts and controls the flow, so could be replaced with more queues and listeners. For now, however, this works very well.

If you’d like to know more, or want more detail on how we host the infrastructure for Continuum, then please get in touch directly by emailing me at: p.shannon@elifesciences.org.

We’ll endeavour to make our data more accurate and timely by utilising cost tagging in AWS, so that we can expose figures directly from AWS Cost Explorer for those interested in future running costs as we scale. Unfortunately, you have to add cost tags in prospectively and we didn’t do that at launch, so will have to wait for more accurate numbers.

This blogpost is cross-posted on Medium.

For the latest in innovation, eLife Labs and new open-source tools, sign up for our technology and innovation newsletter. You can also follow @eLifeInnovation or Paul (@BlueReZZ) on Twitter.