Developing a tool that will allow small and medium scale organisations to get affordable access to the same deep processing power that enables big businesses to get the most from their data.
Working with a range of organisations across Europe provides a unique opportunity to tap into knowledge and expertise of specialists from across a range sectors and countries, for the benefit of the arts and cultural sector. The Audience Agency is able to provide a really strong and valued use case for the project through Audience Finder. Ultimately, our aim is that our contribution to COLA will enable us to make Audience Finder run more efficiently and effectively.
Leo Sharrock, Director of Data Platforms
What is the Project?
As things broadly stand in the world of data processing, an organisation usually has to pay for a fixed amount of server capacity (however many, however big, however powerful) at all times, whether it’s using it all in any given moment or not, or to schedule server up and down times in a fairly manual, planned way. In the brave new world of ‘dynamic processing’, however, it’s possible to provision (and so pay for) only the server capacity you need, only when you need it, according to a set of conditions that you yourself are in control of. Whether your big data processing task is working to a deadline, a target amount of data processing or a fixed budget, project COLA (Cloud Orchestration at the Level of Application) is about finding the most time-effective and cost-efficient way to get the job done within your driving parameters.
COLA is an EU funded multi-partner project, led by Westminster University, commissioned to produce a tool called MiCADO. It aims to improve cloud-based server efficiency and effectiveness for small and medium scale enterprises, so that they can process data flexibly, on a hitherto unaffordable scale. The Audience Agency is working with social-impact-minded technology firm, Outlandish, as one of 3 ‘use cases’ for the development of this – ultimately open source – MiCADO tool.
We’re proud that Audience Finder can play a key role in understanding how more efficiently provisioned servers can save arts (and other) organisations time and money.
The MiCADO tool operates on an intelligent combination of dictated rules and machine learning. On the one hand, you lay out a set of policy-based rules at the outset, dictating that you need the job done within, for example, a specific timeframe. MiCADO can then provision just the right server capacity to complete the job. Furthermore, the tool can then dynamically alter the provisioning rules based on how long the job is actually taking to complete. So, MiCADO would understand in real time how big the job is that you want done, how long it’s taking and whether that’s currently being met by the server provisioning that you have commissioned, or whether it needs to automatically commission more servers to get the job done in time. If your job is taking longer than anticipated, MiCADO provisions more servers, but if it’s going faster, you don’t need those extra servers and so you won’t end up using or paying for them.
Why is Audience Finder a good fit?
Audience Finder is one of many platforms that currently commission a fixed number of servers through Amazon Web Services – not, as we have established, the most effective possible approach. Outside of our normal business hours, we are paying for servers that are not doing any processing – either for us, or for anyone else who could be using them instead. Far more efficient would be to provision the servers so that they are only switched on at the times when we need them, with a degree of server power that would most efficiently and speedily deliver what needs to be done, and can be ramped up during the periods of most intense processing – think Audience Finder’s big weekly data cache, or Arts Council England reporting deadlines. Enter MiCADO.
Audience Finder is getting bigger. As our very significant data set has grown, it has come to require another level of technical infrastructure to support the same responsiveness, or to try and improve it.
The COLA project offers a cost-effective, sustainable way to support that growing volume, continuing to provide a cash-strapped arts sector with an essential tool, at less cost to the public purse. Concurrently, increasingly sophisticated tools, being developed by The Audience Agency itself, may enable us to automate bespoke deep data analysis, eliminating a hugely labour-intensive process and so opening up a level of insight previously only available to organisations large enough and wealthy enough to pay for our analysts and researchers to do that complex type of work.
An organisation might, for example, be interested in commissioning ‘elsewhere analysis’ – that is to say, wanting to know what other cultural and leisure activities their audiences engage in ‘elsewhere’, beyond their own venue. This new functionality would be able to auto-construct the appropriate data set, comparing an organisation’s own data with all other Audience Finder data relevant to their specific audience. We can then analyse which other genres your audience has attended, in which other regions, and exhibiting which behaviors and characteristics. Even when automated, though, the process would occur relatively infrequently, requiring vast server capacity but on an ad hoc basis, only coming into play at an organisation’s individual request – not a viable prospect in a world where you’re paying for all the servers all the time, regardless of use. An ultimate goal would be to build this kind of functionality into dashboards for our clients and consultants alike. That’s a few steps further down the line – but projects like MiCADO can shorten the strides.
A process called ‘containerisation’, facilitated by MiCADO, can make that an affordable reality, both for The Audience Agency and its users. Audience Finder, like many applications, is composed of various individual softwares, running independently of each other, and yet in tandem to deliver the overall platform. Using a platform called Docker, these softwares can be built together, forming a ‘containerised’ version of the full Audience Finder functionality that could then be instantly redeployed elsewhere in its entirety – be that in another organisation, market, country etc. Being able to containerise the platform and deploy it on MiCADO, almost instantly in it’s entirity, suddenly makes this installation and set-up service a much more economically viable proposition.
With the time and money saved by being able to run semi-automated analysis on efficiently provisioned servers, The Audience Agency will be able to supply complex analysis to small and medium scale organisations, in a way that they would never previously have been able to afford – and we think that’s a pretty exciting prospect.
Who else will benefit from COLA?
Beyond improving the level of audience analysis available to all arts organisations, however cash strapped, Project COLA has major implications for other sectors. Audience Finder itself has already been attracting interest from non-arts organisations, keen to develop similar platforms for better understanding their own users. Largescale national conference centres, for example, have many of the same audience insight needs, though far larger databases – often six or seven million customer records per venue, as opposed to a few hundred-thousand as for most arts organisations. This is a very different requirement, then, in terms of processing power and MiCADO’s efficiency would potentially be what makes it possible to affordably provide that service on this far greater scale.
A point of particular interest and pride for us, as a not-for-profit arts organisation, is that The Audience Agency – indeed, the cultural sector in general – is an unusual player on this EU commission funded playing field. The intention is that the three ‘use cases’ involved in developing the MiCADO tool (of which we are one) are sufficiently different in terms of policies that the end product will be transferable far beyond any of our specific remits.
Ultimately, this open-source, readily available tool will be marketed with a comprehensive front end into which any ‘non-developer’ ordinary user, could submit their own policies, to whatever end.
It’s a joy for us to work with such diverse ‘use case’ partners – they are, perhaps, much more what you might expect from an EU funded project. Sociedad Aragonesa de Gestion Agroambiental SL (SARGA), an affiliate of the government region of Aragon in Spain, is looking to improve the way that it communicates with residents through a social media analytics tools. Saker Solutions (SAKER) provides evacuation modelling simulations for disaster situations, running thousands of complex statistical models to establish the safest possible course of action in any given scenario. The cultural to the socially conscious, to the highly commercial. By sharing our knowledge, skills and tools with our COLA partners, we are able to build something far more robust together than we could alone and be part of a project that will continue to receive support and sustenance as our needs and ambitions grow. The COLA Project Consortium goes into more detail about the fascinating mix of expertise that is going into creating this tool.
At just over half-way through the project, it’s so far so good. Ahead of the scheduled completion date of June 2019, there are several technical hurdles for The Audience Agency and Outlandish still to clear, principally around our two bespoke tools currently under production: namely the AAS (Audience Analysitics Service) and ITO (Insight for Touring Organisations). We’re excited to keep working together to realise some of the benefits already generated through the project, so we can elevate Audience Finder to the more flexible technical infrastructure that will continue to support a rapid growth that shows no signs of slowing.