TV News Studio for IBC 2019

Media IP Networks’ Rupert Kelly was appointed Technical Manager for the design and delivery of the TV News studio facilities in the core of the vast International Broadcast Conference, held annually at Amsterdam’s RAI exhibition complex. The TV News studio is used continuously during show opening hours for prerecorded interviews, live panel discussions, technology briefings and glitzy product launches.

Immediately on contract, Media IP Networks secured Jeff Laing as Chief Engineer to oversee the system build and train technical operators. By tradition, many junior operators for IBC are students from the UK’s leading broadcast training institutions, Jeff had a distinguished career break as a lecturer for BBC Academy.

In close partnership with Kev Connor and Roy Callow for IBC Resources and Anna Valley’s Jenny Bigrave and Steve Jones, Media IP Networks was directed by the IBC’s Matthew Tompkinson to develop the project to improve on 2018’s success with yet more cutting edge 2110 technogy from EVS and Grass Valley. This required a major rewrite and rationalisation for the documentation to define a plan that could be agreed by all sponsors. The schematic below captured all technical and operational possibilities and offered resilience, efficiency and performance.

Schematic showing wrap-around SMPTE2110 S-COREMaster orchestrator (in yellow) hosted on Arista7050 with Embyonix SFPs interfacing to HDSDI devices (blue) and backup Sirius matrix (green)

Previous years had seen the migration from a straightforward 5x camera HD SDI installation with discrete analogue and AES digital audio processing for a single studio floor up to a hybrid SMPE2110 / HDSDI design with several major manufacturers offering exhilerating new SMPTE2110 products and techniques, dedicated galleries and commercial wet hire. Several PTZ remote control cameras and a 2nd green screen set were added in 2018 and further upgraded with augmented reality (by Pixotope) and simultaneous recording in 2019.

As a main sponsor, EVS provided not only their legendary XT media servers, mostly configured for SMPTE2110, but also the ubiquitous Arista switches, their ‘S-core Master’ SDN control layer and upwards of 100 Emryonix HDSDI<->SMPTE2110 SFP+ converters. Fortunately EVS also brought several of their very best engineers to IBC, Jeff and I were well supported in all of the design and build stages in Liege (at EVS’ fabulous new facility – wow!) and on show during the rig and transmission days. Special thanks to Johan Engelen for his patient determination, good humour and brilliance.

IBC TV News’s very public studio floor.

Grass Valley Group provided the regular LXD86XF ‘World’ cameras channels with SMPTE2110 adaptors in their XCUs and K-Frame vision mixer crates populated with 2110 cards. GVG were also very supportive with access to their top engineer at the IBC, Geert de Neve, to iron out a few timing and sync errors when the 2110 system had been commissioned.

Additional HDSDI equipment such as a 128×128 Snell Sirius matrix, Evertz synchronizers and multiviewers, HD glue etc was provided by Gearhouse. Pixotope ingeniously turned the green screen, lit precisely by Biff, into a truely amazing Augemented Reality demonstration, popular as a stylised commercial product presentation facility.

Grass Valley provided both vision mixer and 5 full camera chains with full SMPTE2110 delivery paths which seamlessly integrated with the Embryonix HDSDI adaptors in the Arista switch frames under the S-COREMaster control palne..

An Andiamo MADI converter was used to interface with the sounds desk to feed AES into the ingest XTs. A Clearcom intercom system was deployed across all galleries, the floor and remotely to other IBC areas over the SM fibre-connected Baumann multiplexor.

TV News crew 2019. Special shout out to Neil McLeod and Gillian Kelly

Korea Olympics 2018 for NEP and Discovery


Media IP engineers fully designed, commissioned and maintained the resilient, scalable, high performance IP internetwork to connect all Discovery’s HD video, audio, communication and file-based resources within the International Broadcast Centre and to and across 13 games venues and Eurosport client sites for the Winter Olympics, PyeongChang, Korea in January and February 2018.

The IP network was fully operational and integrated with

  • Live HDSDI (200+ paths) and file-based broadcast content delivery, intercoms and equipment control across 4x Eurosport IBC studios and 13 venues.
  • Intercontinental Transmission links to Discovery’s Eurosport clients and libraries.
  • MediaGrid and venue-based ingest services provided by EVS.
  • IBC distribution and venue information services provided by OBS.

Venue-to-IBC IP infrastructure within Korea was provided by Olympic Broadcast Services (OBS) to all venues. Diverse managed trunks carried all broadcast, corporate ‘user’, streaming and control and coordination traffic.

Intercontinental transfer and transmission path services as well as WAN IP infrastructure for over 70+ HD feeds to European-based content presentation, transmission and archive facilities and 48 broadcast ‘markets’ was delviered by Globecast and tightly integrated with NEP’s LANs in the PyeongChang IBC.

Performance required:

  • In excess of 99.999% uptime (‘5x9s’)
  • Clearly prioritize IBC <-> Venue traffic types in the following QoS hierarchy:
    • IP system management (routing protocols etc)
    • Realtime high value streaming media traffic (2022-6/7, AoIP)
    • SSH/Telnet configuration and monitoring
    • Non-broadcast streaming (IPTV)
    • Equipment control (SDI Matrix, RCPs, SIP, GUI configurators)
    • Content file transfer (EVS Funnell)
    • Network statistics gathering (SNMP and Netflow)
    • Web browsing
    • Discovery ‘corporate staff user’ traffic
  • Be simple to monitor and troubleshoot.
  • Utilise all physical layer resources to provide BOTH resilience and load balancing.
  • Use protocols and methods that are proven and understood.
  • Integrate cleanly and efficiently with both the local LANs at each venue and the intercontinental WAN over to Discovery’s principal European transmission networks.

The IP architecture was based around proven and well understood collapsed core topology as below. Two powerful Arista 1750 L3 switches created the resilient switched core with full L3 routing across all the 90+ IBC LAN subnets although only 6 were directly connected to each and super-netting was used to avoid unwieldly routing tables. A further distribution layer of four 7150s downlink to the 13 connected venues, two each on both the A and B venue link sides as provided by Olympic Broadcast services. These 4 critical switches were interconnected in a grid topology using both link aggregation (as MLAG) and OSPFv3 equal path load balancing. No instability was detected by packet loss or path jitter during all of installation and operational use.

Final layer 3 IP internetwork design as built and operational for 5 weeks with 100pc uptime and informational syslog only.

Supernetting has widely been implemented at the core and distribution levels. A few static routes were entered to avoid more complex link state calculations of the core switches and avoid ‘rogue’ configs pulling down the core’s routing table.

OSPFv3 is an advanced link-state routing algorithm and when configured with short LSA timers and dynamic, near-equal path costs provided proportional load balancing as well as full, automatic failover resilience. This proven technique quietly doubled WAN capacity using fully implemented backup paths to effectively deliver 2x1Gbps (or 2x10Gbps) provision at L3 although resilience then fell back on effective QoS for critical traffic if a link is compromised and traffic levels were in excess of a single link.

All of the remote terminations (venues) at the end of each spoke followed the exact same design with custom changes made for IP addressing and additional services/subnets as required. All remote subnets were gatewayed through resilient, load balancing and QoS’ing Arista 7150 L3 switches with very similar and proven configurations supporting OSPF, MLAG and VARP for all remote services, whether implemented or not. So as well as direct access uplinks for the 34+ venue LAWO Rem4s codecs, they were the principal distribution to secondary access switches for non-2022 VLANs such as VSM, Ravenna, Ingest, Riedel (Cisco, Netgear). 

Layer 2 schematic for a sample venue, 1 of 13, showing interlocking horseshoe VLANs to avoid STP instability and enable equal path OSPF load balancing. Also the direct HDSDI codec VLANs in red, both LAWOs at each end on same subnet to permit SAP, to avoid L3 routing complications.

Across the whole estate was a strict condition of each and every subnet existing on a single VLAN with a single gateway. Per VLAN Spanning tree (PVST) is deployed in every VLAN with no STP performance degradation since each VLAN was limited to a maximum of 4 switches with the exception of corporate end-end traffic. Portfast was universally applied and no STP instability was seen across syslogs from any switch even with MLAG interlinks. There were no VLAN loops and Spanning Tree was NOT used to provide any failover mechanism. VLAN trunking was deployed throughout with central, simple supernetted L3 IP routing in the collapsed core providing a stable internetwork, working end-end at line speed for all services including EVS, Graphics and operational TV surfaces.

Snapshot from Solarwinds NMS in the MCR/NOC illustrating resilient 30+ device MLAG switching fabric across the IBC estate distributing all content and control within secure, monitoried VLANs.

Venue to venue traffic was fully implemented including 12+ broadcast HD camera feeds between the Ice Hockey venues as IP JPEG2000 streams over the protected XYZ/OBS infrastructure as required, notwithstanding security issues. Similarly, all broadcast IP sources at every venue were fully available at all IBC and Europe-wide transmission centres.

Snapshot of Solarwinds NMS monitoring in MCR and NOC for EVS connectivity end-end across estate and VMware RDP hosts within IBC ESX server.

Media IP networks commissioned a VMware ESXi6.7 server and populated with 10x Windows 10 VMs. A working, secure and reliable Windows10 VM with the VSM application was trialled and replicated 11 times to create the battery. Remote broadcasters in Europe and at the venues used regular MS Terminal Services clients (MS native, Mobexterm, VNC) to remotely control a strictly allocated and firewalled VM instance to select their IBC sources to their WAN links for Tx. Confidence was high that these VMs were reliable and effective. Apparently these VMs have been used to change sources on air from the broadcaster’s galleries in Europe

Snapshot of Solarwinds NMS in Media IP Networks’ NOC monitoring LAWO codecs at venues and in the IBC in realtime for bandwidth, jitter and packet loss.

Clever fun and racking a shiny floor in Dock10

Big studio shows in specially constructed television studios are always fun and have an ‘electricity’ all of their own. When Gameface’s pilot of Britains Brightest Family was commissioned into 3 annual series for ITV1, the best place to record the tournament of 37 x 1/2hr tests of mental agility, courage and knowledge was one of the larger studios at Mediacity’s Dock10.

Presented by Countdown’s Anne Hegarty, produced by Adam Adler , Directed by Deraldine Dowd, Ollie Bartlett and Richard Van’t Riet and lit by Gurdip Mahal (”17 & ’18) and Max Conwell (’19), Media IP’s Rupert Kelly configured and Vision Operated the 8 full studio cameras for all three series.

RIO Olympics for BBC. Streaming from Salford

With exclusive UK broadcast rights and complete access to every Olympic sport and event, BBC Sport streamed everything that ran, walked, jumped, was thrown, hit, kicked, stroked, shot or ridden.

As a shift Technical Producer and MCR operator, Media IP Networks’ Rupert Kelly kept the sport and the studio contributions all ‘on air’ for the 3 weeks it was all live, gripping the world.

From the state-of-the-art HD galleries in the BBC’s Quay House, Mediacity Salford, the internet was kept busy by the global delivery of every minute of every sport through the BBC’s streaming encoders.

Each sport was linked through the main galleries with as many as 10 sessions passing through the vision nd sound mixers at busy times in the first week.

Euro 2016 soccer tournament for ITV Sport.

ITV Sport pulled out all the stops to deliver comprehensive coverage of their pick of the Euros right across France.

As the Technical Producer travelling with the brilliant Leon Mann reporting and ITV Sport ‘s own Rachael Doe producing, Rupert Kelly from Media IP Networks brought dedicated live coverage of ITV’s pick from the 6yd line at several high profile Northern Irish and Scottish matches.

Using Live U, the truly portable H264 compression and aggregating 4G contribution link from the ENG positions pitchside, ITV Directors in the Paris studio (Arena OBs overlooking Notre Dame) had alternative shots of key events and players to the unpredictable host broadcaster’s coverage.

This gave ITV Sport’s presentation more depth and excitement for what was an exceptional tournanment, albeit spread out over all of France from Lille to Marsielle, from Bordeaux to Lyons.

Roger Pearce, Technical Director for ITV Sport and Euro2016 was truly delighted at the innovation and efficiency that Media IP’s Rupert Kelly and Mike Tope brought to the deployment of the LiveU for the Live transmissions and how the Production team used it with confidence and agility.

Glasgow’s Commonwealth Games

Media IP Networks was closely involved in the delivery of the transmission feeds from the Glasgow Commonwealth Games specifically the large open city centre-based cycling and running events from Glasgow Green.

Although the venue is spread across the city and out into the surrounding countryside the 32 contribution feeds for the sportive cycling and marathon from the motorbike and static cameras were all rendered to the OB control centre close to the start and finish.

From here Media IP Networks oversaw the delivery to the IBC for International distribution and to other local broadcasters from the event’s Technical Operations Centre.

Enter the Dragon at The Space Project

For the 2nd year the BBC’s Dragons Den series is being fully recorded at Stage 4 of Manchester’s innovative and efficient ‘Space Project’ close to Piccadilly and the city centre.

In a radical upgrade, the Production management have eschewed the laborious multiple single camera recorder format and taken the full blooded as-live outside broadcast option – with a twist.

Media IP Networks has been engaged to ensure all the HD camera, GoPro, 5.1 audio technology and mechanical functionality of the OB vehicle are guaranteed to the client, the BBC. Also we have integrated the truck’s IP infrastructure into the studios’ for monitoring, VoIP and Internet connectivity.

The large ‘cruiser class’ OB Unit 2 from Bedford-based facilities contractor Puls8 will be on station at the studios for the whole recording run of 6 weeks providing full audio and visual recording support to local Avid Airstream disk recorders.

In addition all the 12 cameras are being simultaneously recorded in Dock 10’s Salford facility on the other side of the City over a Reidel Fibernet loop.

Although this incurrs the additional expense of a dark fibre between the sites this is more than offset by the time saved in ingest at the large edit facility – all the material is ingested and reviewed continuously as-live with no need to transport and play tapes or cards.

Despite no failures of the innovative and efficient fibre connectivity all feeds are also recorded locally in the OB vehicle for safety. Dragons Den’s original content is too fragile to lose a second. Re-recordings for lost content are not an option.

Additionally the excellent lighting and vision crew have lit and matched all the sources – 9x x Sony HD1500 line cameras and 8x GoPro cameras. This to has rendered a large part of the previous workflow, specifically ‘the grade’ totally unnecessary, potentially creating a significant saving.

Broadcast Technical management for ITV’s Rugby World Cup

The 8th Rugby World Cup is in England this year with Media IP Networks present for some or all of the tournament at 4 of the 12 venues across England.

Rupert Kelly will be Venue Technical Manager for the hosts and domestic broadcaster, ITV Sport, with responsibility for the origination and delivery of the TV signal from the OB at the stadium through to the IBC.

Starting with the Japan v SA match in Brighton, Rupert moves up to Leeds for a string of games before moving back south for the big England v Australia game at Twickenham.

Then to hometown Manchester for the final stint at the Etihad for the last England match of the group stages.

It will be unsettling to see the tall, netless posts on City Football’s hallowed pitch . .