By Jermaine Allen, MerchantCantos (EVCOM Member)
Many of us have experienced that moment of absolute dread when we’ve accidentally deleted documents we’ve been labouring over for hours. In this digital age, it’s become just as easy to do even worse with personal photos and films – priceless wedding albums or precious memories of children’s birthdays and family holidays can all be erased with the errant click of a mouse. As the panic prickles up the spine, we all find ourselves thinking exactly the same thing: ‘I hope my back-ups are working’.
In 2012, Pixar, the world famous animation studio, endured a back-up failure on an industrial scale. During the production of Toy Story 2, already years into its production, someone working on the film accidentally ran a command deleting content from the file system. A $90m movie literally began disappearing in front of the animators’ eyes. To make matters much worse, it was subsequently discovered that the back-up system had failed to work properly. The world was very close to forever losing the heart-warming tale of Buzz, Rex and Mr Potato Head rescuing Woody from a museum of toys… a cultural disaster on a par with the Mona Lisa going up in smoke, or the Leaning Tower of Pisa finally toppling over. Luckily, humanity was saved this misfortune thanks to an employee on maternity leave working on the film remotely, unconnected to the main network. This amazing stroke of good fortune allowed the studio to salvage the film and avoid an extremely expensive mistake.
I always like to keep this story front of mind, because it’s my responsibility to manage the film content of MerchantCantos. Every year we generate terabytes of data and the shift to 4K means these volumes are increasing dramatically. As a result, having a robust workflow and data protection plan is essential, which is why we have invested heavily in our global infrastructure – implementing LTO tape technology along with Cloud based back up and archiving solutions.
Here’s a quick history lesson for anyone below the age of 35. Before the dawn of full digital video production, around the early noughties, cameras still used physical tapes. Those tapes would be digitised into video files and edited into finished films. The films were then transcoded into multiple digital formats for online delivery. As a fail-safe, the finished films would then be mastered back onto tape. From a DR (Disaster Recovery) perspective it was brilliant; you would have your original camera footage on tape if your hard drive failed, plus two masters of your final film. Today, data cards have replaced tapes; once filming has finished, the footage from cards gets transferred for editing onto servers that contain multiple hard drives. The cards are then wiped for use on the next shoot. This all sounds quite straightforward, but unfortunately, hard-drives and servers aren’t always as fool proof as people believe. The potential for disk failures means it’s quite possible for any project to very quickly go the same way as Toy Story 2.
Nobody ever wants to have to tell a client that their all-important film has disappeared in a puff of smoke and that’s exactly why we’ve invested in our robust disaster recovery solutions. We’ve built multiple levels of redundancy into our workflow, protecting us against hardware failures, data corruption and accidental deletions. Moving to the Cloud has also given us the ability to access our content from anywhere in the world, at any time. Long gone are the days of overnight transfers of film rushes from international shoots.
Disaster recovery might not be the most glamorous element of film production – it doesn’t involve any prestigious locations or shoots with celebrities. And it’s not something our clients think about very much when they commission a project with MerchantCantos, but it’s of critical importance. Thanks to the investments we’ve made, just like everyone’s favourite toy cowboy, any data that gets lost has a friend in me.
As a DIrector of Photography the idea of one master copy of the camera masters has always left me queasy for all the reasons cited above. For this very reason shortly after the transition to non tape formats we started making “field-clones” of all camera masters right on set- it only costs for the storage but you are a real hero to your client if a drive corrupts- or some other mishap occurs. In the 10 or so years we have been doing this, we have saved the day three times, and a DIT’s job once.
MerchantCantos are very fortunate to have you – I trust they fully appreciate the fantastic work you are doing for them! This is a sobering tale that reflects due diligence and forward planning not only saves the day but also saves a company’s reputation – something taken for granted too often commercially. Well done for presenting this so eloquently.