The Science Museum Group is using SnapLogic technology to power a range of data-led initiatives that are improving experiences for visitors, employees and researchers.
The Group, which runs five museums that welcomed 3.8 million visitors last year, holds 7.3 million items in its collections. As well as museums, the group also runs storage facilities. One of these facilities, Blythe House, is being sold, so the group is moving 320,000 small objects to its purpose-built National Collection Center (NCC) in an initiative called the One Collection project.
All items are to be photographed and packed by staff, with a huge effort to ensure items could be found easily post-move. This has IT implications, not least that staff needed a web front-end, so they could interact immediately with the objects in front of them and find records in a back-end system. Tom Saunders, Lead IT Software Developer at the Science Museum Group, explains:
There was a huge IT process behind the physical process to move all these items. We had to inventory everything – some of these things have been sitting on shelves since before I was born…We knew we were going to need an integration tool for this project and the IT department saw an opportunity.
By the end of this month, the group will have moved out of Blythe House. Most objects have been moved to the NCC, but the behind-the-scenes IT process attached to this shift took four years to complete using SnapLogic. Saunders recalls:
In those four years, we built 80 pipelines to integrate our information. We were moving information backwards and forwards between the web front ends and our other systems. So, if we move a container with a bunch of objects, we use the intelligence of SnapLogic to say that those objects have also moved to a new location.
Extending the remit
While most of the work behind the One Collection project is now complete, SnapLogic is being used across the organization. Saunders gives the example of photography –only a fraction of the group’s 7.3 million items are on display. One of the aims is to digitize all objects and link their photos to back-end description data:
We have our collections database, which is where we store everything that we know about everything we own, and we have our digital asset management system, which is where we store a lot of digital photos. Those two need to know each other's information and what's going on, so we’ve automated it with SnapLogic.
Another use case for SnapLogic is around e-commerce. After the group’s retail team adopted Shopify, Saunders’ development team built 33 pipelines across 17 tasks to integrate existing retail systems with the group’s CRM platform and warehouse and dispatch services. They also built pipelines for sales, returns and the introduction of new products to the store:
It was probably the most agile we've ever been. We started building as we found out what we were building, and then we went back and changed the things we'd already built. We turned it around pretty quickly.
When it comes to day-to-day activities, SnapLogic powers an operational dashboard. This dashboard presents live information about the group, such as ticketing and visitor counts. SnapLogic has also been used to pull data from PowerBI into a management information hub, which provides monthly reporting:
We’ve built 14 pipelines to do all of this and pull data from everywhere into 17 distinct datasets. It’s the senior management’s tool to help them monitor and maintain the health of the museum group and work out where to focus their efforts.
As well as large-scale projects, Saunders says SnapLogic is used to push data around the group for a range of other uses cases. On a monthly basis, he says the development team maintains 145 active pipelines and they’ve processed 37 million rows of data.
Pushing into new areas
Saunders says the Group has seen a huge amount of growth in its use of SnapLogic over the past six years:
It’s empowered us to be a team that says ‘yes’. We can prototype things very quickly.
Next on the agenda is an attempt to expand the management information hub towards self-service, where data is pulled into the system from enterprise sources. Another consideration is that the NCC will open to the public for research next year and that move will require support from the group’s development team:
We're going to have to wire up the customer experience of interacting with us through the website, and make sure an object is available for viewing and that we're moving it to the right places where people can get to it. So, there's going to be a lot of back-end connectivity that we’ll have to build in the next six months.
The Group is also exploring AI-based transcriptions of audio-based oral histories. It’s tough for researchers to sort through hundreds of hours of speech, so a project has started to automate transcriptions using AI and to use SnapLogic to move those files to AWS. There’s also consideration of digital preservation and how data assets, such as software and files, might be preserved in a similar way to physical assets. Saunders concludes:
Preserving the metadata is a massive headache, such as making sure the digital files haven’t degraded as they've been moved. We're going to be using SnapLogic a lot to move those files around, to check up on them, and to validate them.