Part I – Delivering Open Backend Services for the Public Sector

This is the first part of a four-part series of posts in which we aim to run you through a proof of concept that we have been working on at UKCloud to deliver backend services to support the public sector. If you are interested in understanding more about backend as a service it’s worth reading a previous post on the subject.

The four parts in this series are:

  1. Open Service Broker Dataservices setup
  2. Open Service Broker provisioning (IaaS)
  3. Open Service Broker provisioning (OpenShift)
  4. Dataservices features and roadmap

Part 1

Over the last couple of days, we have been working with a great company called Anynines. They are based in Germany and provide public cloud foundry services, hosted on AWS. They have been incredibly active in the community and over the years have built up some impressive engineering capabilities. Anynines have been helping us build out a proof of concept of their data services which we hope will support the development of applications in line with backend as a service on UKCloud’s platform.

If you are interested in why we believe that data services are so important, it would be worth reading a recent two-part blog post that we did on the subject, here.

The data services will be deployed on top of our public OpenStack cloud. These services will be exposed via the Open Service Broker Framework to prevent the kind of vendor lock-in the you get with proprietary cloud service providers. UKCloud wholeheartedly support this model.

In the post below we will cover infrastructure architecture and service initialisation, and in the next post we’ll discuss end-user provisioning both from IaaS and integrated with OpenShift.

Infrastructure Architecture

The Dataservices infrastructure can broadly be split into three logical sections.

Dataservices Initialisation infrastructure

BOSH is a project that unifies release engineering, deployment, and lifecycle management of small and large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs. It also performs monitoring, failure recovery, and software updates with zero-to-minimal downtime.

In this project we used BOSH as the deployment tool to manage the deployment and lifecycle of both the dataservices management infrastructure and the service instances that support customer workloads. More information on BOSH is available here.

BOSH has a concept of an inception VM, which is used to instantiate the full BOSH release. This is simply an OpenStack instance with the BOSH cli installed. This VM is then configured with the relevant IaaS details to be able to orchestrate provision via the OpenStack API’s.

With the BOSH cli and IaaS configuration we instruct BOSH to deploy BOSH “director” which will be used to manage not only the deployment of the management services, but also the deployment of the customer service instances.

Once the BOSH director is deployed, we target it and upload a stemcell image (which BOSH pushes to the tenancies OpenStack catalog), this is then used as the base image for all instances deployed via BOSH.

Dataservices Management infrastructure

We build the management services via the BOSH director.

DNS
DNS is required to dynamically handle DNS for the service endpoints. The Service Broker API will pass back the service endpoint details when the service instance is created so that the users can target their service from their application.

Database
A highly available Postgres cluster is provisioned to store persistent data about the dataservices.

Backup Manager
The backup manager co-ordinates the backup and restore of all customer service instances.

Data guard
Handles dynamic security to restrict customers so that they can only access their own service instances.

Service Instances infrastructure

Control plane
Each service (Postgres, Redis, Mongo etc) contains its own control plane consisting of:

  • An Open Service Broker API
  • A Service Provider Interface (abstraction layer for orchestration created by Anynines)

Each service control plane is then deployed via BOSH.

Customer service instances
Once the control plane is in place the service instances that host the customer dataservices can be deployed via a self-service request from the customer to the broker API. The service instance infrastructure will vary based on the services that are provisioned. The different deployment options are displayed as service options which the user specifies when they request a new service.

Using Postgres as an example, Anynines offers four service plans which can be viewed via the broker catalog:

curl –u username:password http://postgresql-service-broker.service.dc1.consul:3000/v2/catalog

At this point it is possible for the user to make a request for the new service referencing the plan which in-turn uses BOSH to deploy the dedicated customer service instances and deploy the software.

The user can then create a binding which will pass back endpoint details including DNS hostname, credentials and any specific details like database name. For example:

In the next part of this series (part 2) we’ll go into more detail about service provisioning, both with IaaS and with OpenShift.