Korea Olympics 2018 for NEP and Discovery

Featured

Media IP engineers fully designed, commissioned and maintained the resilient, scalable, high performance IP internetwork to connect all Discovery’s HD video, audio, communication and file-based resources within the International Broadcast Centre and to and across 13 games venues and Eurosport client sites for the Winter Olympics, PyeongChang, Korea in January and February 2018.

The IP network was fully operational and integrated with

  • Live HDSDI (200+ paths) and file-based broadcast content delivery, intercoms and equipment control across 4x Eurosport IBC studios and 13 venues.
  • Intercontinental Transmission links to Discovery’s Eurosport clients and libraries.
  • MediaGrid and venue-based ingest services provided by EVS.
  • IBC distribution and venue information services provided by OBS.

Venue-to-IBC IP infrastructure within Korea was provided by Olympic Broadcast Services (OBS) to all venues. Diverse managed trunks carried all broadcast, corporate ‘user’, streaming and control and coordination traffic.

Intercontinental transfer and transmission path services as well as WAN IP infrastructure for over 70+ HD feeds to European-based content presentation, transmission and archive facilities and 48 broadcast ‘markets’ was delviered by Globecast and tightly integrated with NEP’s LANs in the PyeongChang IBC.

Performance required:

  • In excess of 99.999% uptime (‘5x9s’)
  • Clearly prioritize IBC <-> Venue traffic types in the following QoS hierarchy:
    • IP system management (routing protocols etc)
    • Realtime high value streaming media traffic (2022-6/7, AoIP)
    • SSH/Telnet configuration and monitoring
    • Non-broadcast streaming (IPTV)
    • Equipment control (SDI Matrix, RCPs, SIP, GUI configurators)
    • Content file transfer (EVS Funnell)
    • Network statistics gathering (SNMP and Netflow)
    • Web browsing
    • Discovery ‘corporate staff user’ traffic
  • Be simple to monitor and troubleshoot.
  • Utilise all physical layer resources to provide BOTH resilience and load balancing.
  • Use protocols and methods that are proven and understood.
  • Integrate cleanly and efficiently with both the local LANs at each venue and the intercontinental WAN over to Discovery’s principal European transmission networks.

The IP architecture was based around proven and well understood collapsed core topology as below. Two powerful Arista 1750 L3 switches created the resilient switched core with full L3 routing across all the 90+ IBC LAN subnets although only 6 were directly connected to each and super-netting was used to avoid unwieldly routing tables. A further distribution layer of four 7150s downlink to the 13 connected venues, two each on both the A and B venue link sides as provided by Olympic Broadcast services. These 4 critical switches were interconnected in a grid topology using both link aggregation (as MLAG) and OSPFv3 equal path load balancing. No instability was detected by packet loss or path jitter during all of installation and operational use.

Final layer 3 IP internetwork design as built and operational for 5 weeks with 100pc uptime and informational syslog only.

Supernetting has widely been implemented at the core and distribution levels. A few static routes were entered to avoid more complex link state calculations of the core switches and avoid ‘rogue’ configs pulling down the core’s routing table.

OSPFv3 is an advanced link-state routing algorithm and when configured with short LSA timers and dynamic, near-equal path costs provided proportional load balancing as well as full, automatic failover resilience. This proven technique quietly doubled WAN capacity using fully implemented backup paths to effectively deliver 2x1Gbps (or 2x10Gbps) provision at L3 although resilience then fell back on effective QoS for critical traffic if a link is compromised and traffic levels were in excess of a single link.

All of the remote terminations (venues) at the end of each spoke followed the exact same design with custom changes made for IP addressing and additional services/subnets as required. All remote subnets were gatewayed through resilient, load balancing and QoS’ing Arista 7150 L3 switches with very similar and proven configurations supporting OSPF, MLAG and VARP for all remote services, whether implemented or not. So as well as direct access uplinks for the 34+ venue LAWO Rem4s codecs, they were the principal distribution to secondary access switches for non-2022 VLANs such as VSM, Ravenna, Ingest, Riedel (Cisco, Netgear). 

Layer 2 schematic for a sample venue, 1 of 13, showing interlocking horseshoe VLANs to avoid STP instability and enable equal path OSPF load balancing. Also the direct HDSDI codec VLANs in red, both LAWOs at each end on same subnet to permit SAP, to avoid L3 routing complications.

Across the whole estate was a strict condition of each and every subnet existing on a single VLAN with a single gateway. Per VLAN Spanning tree (PVST) is deployed in every VLAN with no STP performance degradation since each VLAN was limited to a maximum of 4 switches with the exception of corporate end-end traffic. Portfast was universally applied and no STP instability was seen across syslogs from any switch even with MLAG interlinks. There were no VLAN loops and Spanning Tree was NOT used to provide any failover mechanism. VLAN trunking was deployed throughout with central, simple supernetted L3 IP routing in the collapsed core providing a stable internetwork, working end-end at line speed for all services including EVS, Graphics and operational TV surfaces.

Snapshot from Solarwinds NMS in the MCR/NOC illustrating resilient 30+ device MLAG switching fabric across the IBC estate distributing all content and control within secure, monitoried VLANs.

Venue to venue traffic was fully implemented including 12+ broadcast HD camera feeds between the Ice Hockey venues as IP JPEG2000 streams over the protected XYZ/OBS infrastructure as required, notwithstanding security issues. Similarly, all broadcast IP sources at every venue were fully available at all IBC and Europe-wide transmission centres.

Snapshot of Solarwinds NMS monitoring in MCR and NOC for EVS connectivity end-end across estate and VMware RDP hosts within IBC ESX server.

Media IP networks commissioned a VMware ESXi6.7 server and populated with 10x Windows 10 VMs. A working, secure and reliable Windows10 VM with the VSM application was trialled and replicated 11 times to create the battery. Remote broadcasters in Europe and at the venues used regular MS Terminal Services clients (MS native, Mobexterm, VNC) to remotely control a strictly allocated and firewalled VM instance to select their IBC sources to their WAN links for Tx. Confidence was high that these VMs were reliable and effective. Apparently these VMs have been used to change sources on air from the broadcaster’s galleries in Europe

Snapshot of Solarwinds NMS in Media IP Networks’ NOC monitoring LAWO codecs at venues and in the IBC in realtime for bandwidth, jitter and packet loss.

Network training delivered to IP matrix pioneers

Having been involved in technical IP network training since the introduction of the Cisco CCIE courses over 20 years ago, Media IP Networks are well-placed to deliver precise, customised IP Networks course to Arena Television in Redhill, Surrey.

Arena has made a significant and brave investment in new IP technology, replacing the core HDSDI router in each of their new fleet of carrier-class UHD trucks (OBX, OBY and OBZ) with Cisco Nexus 9K IP routers, IGMP and packetised, multicast TICO streams.

Arena’s engineers are tasked with the acceptance and configuration, operation and maintainance of the HDR UHD facilities in these £multimillion vehicles at major events and out on the road. This involves knowing and fixing IP network techniques and issues.

Media IP Networks’ Rupert Kelly reviewed the IP technolgy in use in OBX and will be delivering a bespoke 2 day instructor-led course including practical demonstrations and hands-on configuration of both physical and virtual Cisco Layer3 switches.

Update : Dafidd Rees, Arena’s Deputy Director of Operations said Media IP Networks’ courses were well recieved by the 2 classes of 12 engineers and was pleased they dovetailed in with Cisco’s own ‘handover’ presentation to his Engineers of multicast methods and their OBX topology.

A further day is scheduled for next year for the remaining engineers, mostly sound.

The powerpoint presentation and schematics for the course are available in the Resources section.

https://www.arena-uhd.com/

Automation technology refresh for ITV North.

After 5 years of continuous service the automated transmission system for the 12 northern regions of ITV1 and all of ITV2, ITV3 and ITV4 have been upgraded by Media IP Networks using Snell’s (now SAM) Morpheus product line. An emphasis has been placed on deployment of native ICE content engines and processors to replace 3rd party GVG K2s and Maestro mixers.

The new system now runs alongside the old with simultaneous transmission of content on both systems until the remaining issues with additional services like Audio Description, multiple audios, EPG notes etc are eliminated. At that time the old system will be decommissioned by Ericsson, the contracted network services provider. Content and commercials are replicated from the southern centre in Chiswick Park and run independently to provide resilience and local branding. Disaster recovery of the south for the north and vice versa is also provided by the expansive and scaleable real-time services running over high performance and resilient IP networks.

Media IP Networks worked under the guidance of Citrus Networks Ltd, based in London and working with SAM in the Ericsson facility in Chiswick Park.

The system has now been handed over to senior engineers in Ericsson in Leeds and they anticipate a migration to the new system in time for the live Rugby World Cup later this year across all ITV1 regions and ITV4. The commercials around live rugby are considered amongst the most lucrative for ITV, there is no margin for error in their delivery through the new Morpheus system in NTC.

“Have you ever had a dream, Neo, that you were so sure was real? What if you were unable to wake from that dream? How would you know the difference between the dream world and the real world?”

and

“This is your last chance. After this, there is no turning back. You take the blue pill – the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill – you stay in Wonderland and I show you how deep the rabbit-hole goes.”

City Football embraces HDTV for Fanshows and IPTV

Man City’s big plans for the new Academy Centre and its 17 new pitches and Elite training facility just across the new footbridge to the north of their home at the Etihad stadium have grown further still with the integration of their broadcast-standard TV facilities across the diverse departments in the new City Football enterprise.

Media IP Networks is involved in the review of the ambitious needs for the TV production teams working on the pre and post-match live fan show, the new digital platforms to fully embrace original live HD streaming media within the existing IPTV internet channel and in the Etihad and the ‘tactical’ video coverage of performance for the team managers form the 1st team down. This has brought us into consultation with the IT department to support the new fibre infrastructure running the IP networks and Man City’s own project managers with reference to budgets, buildings and procurement.

The principle achievement within this project for Media IP Networks so far has been the reconciliation of the needs and budgets of these three large TV productions within City Football – the matchday fanshow, the IPTV live streams and, to a lesser extent, the review coverage for the sports managers.

Since the revised and budget has now been agreed and a bid from Timeline Television accepted as the contracted integrators and operators, our involvement will now move to the delivery of the merged designs and trials of the IP networks and the new RF and CCTV cameras.

Media IP Networks will also be involved with Barcelona’s own MediaPro in the rebuild and relocation of the edit facilities to the new City Football Academy premises early in 2015.

And more ambitions ideas involve the link up with the other clubs in City Football, notably Melbourne, New York, Yokohama and AbuDahbi to create a 24hr City Football channel with original programming from across the world under the same brand.

Researching a transparent media network for MediacityUK

Although a keystone in the developers’ plans since the first draft back in 2007, a transparent, open, high speed and stable internetwork between producers, facilities and broadcasters in MediacityUK has been elusive up to now and is essential for Phase 2 over the next 10 years.

Market research by MediaIP Networks is underway to clarify business requirements and define sustainable services for a campus-wide media network. The results of this research will in turn enable Media IP Networks to confidently build partnerships with service providers already established in MediacityUK such as Metronet, BT Dock10 and Geo to deliver an integrated network infrastructure.

Studio / Datacentre ‘spline’ topology concept

Currently the research suggests that there is a significant demand for 1Gbps to offices and 10 Gbps to facilities for the storage, processing or replication of extremely media large files, principally in high definition but also in 4K. Similarly early responders have expressed interest in the network to ‘stream’ media in realtime for presentation or editing between facilities and to preview theatres.

Interestingly, several network technologies, old and new, are appearing consistently:

  • Spline planes supporting multilinked distribution switches in studios and edits.
  • FTP for file transfer in the guise of Signiant, Digidelivery et al
  • VLAN over VPNs for easy access to MS Windows shares and LAN apps
  • Traffic Shaping to protect VoIP or remote control protocols.
  • Internet access at minimum of 100Mbps to remote, off-campus facilites.
  • Multilevel firewalls to protect sensitive data whilst still permitting client and remote user access.
  • Integration with global managed networks beyond Mediacity such as Sohonet.

Further notes and interesting results from this research will posted here as appropriate.

Please visit us again and please add your own comments, suggestions and of course, any feedback!

Mediacity Pie Factory internet uplink

Mediacity data connectivity supplier, Dock10, has installed a full 100Mbps internet data uplink to post-production tenants in The Pie Factory.

Consultant Cisco engineers for Media IP Networks have since successfully commissioned a full-featured, ‘layer 2-only’ Virtual Private Network (VPN) over this link to connect remote media and office services to the editor suites, ingest and dubbing theatre in Mediacity’s University of Salford building.

This secure, reliable opensource solution allows full integration of Avid, Editshare and regular TCP traffic across diverse sites with VLAN’d Cisco infrastructure. It also fully integrates with the existing twenty+ PPP VPNs used by remote users and teleworkers to now present the whole IT estate to authenticated users.