Radio War Stories from Sochi

In the midst of the endless media blitz covering the 2014 Winter Olympics, Westwood One’s VP-Engineering, Mitch Glider, joined us in real time from Sochi, Russia. Mitch and his colleagues have assembled a long-form studio plus several reporters’ workstations.

Now the hard prep work pays off and engineering vigilance begins as hours and hours of Olympic radio coverage is fed back to New York for affiliate distribution. Mitch takes us through the studio build and the technologies employed to bring live commentary and reports from across the Olympic venues and ship the final products back home.

Watch the Video!

Read the Transcript!

Announcer: This Week in Radio Tech – War Stories, Episode 200 – is brought to you by the Telos Z/IP One IP audio codec, low-latency, high- quality stereo audio over private networks or the public Internet. The Telos Z/IP One is the best way to hear from there, on the web at telos-systems.com. On the eve of the 2014 Winter Olympics Westwood One's VP of Engineering, Mitch Glider, joined us live from Sochi, Russia. We find out about the studios built in just a few days and the technologies employed to bring live commentary and reports from across the Olympic venues. This Week in Radio Tech starts now.

Kirk: Hey, welcome in to This Week in Radio Tech. It's show number 200. Yay. We've made it a long ways. What is this? Four years' worth of This Year in Radio Techs, or just a little over four years. We took a short hiatus a couple of summers ago, and this is the show where we talk about radio technology, broadcasting audio, the things that broadcast radio engineers have to do the get the sound out to lots of folks, make it sound good and keep it all working. Sometimes we even go do difficult work out in the trenches. And that's what this show is all about.

This is show number 200. Every show that's divided by 10, we call it a War Stories episode. So today have we got a war story. It's from a war that's happening right now. So, hang on, I'm going to introduce our guest in just a minute. Our show is brought to you by the Telos Z/IP One. Telos happens to be my employer and I deeply appreciate their sponsorship of the show, and the Z/IP One is an IP audio codec. We'll hear a bit more about it later. Man, I'm just excited to be here and I know another guy is too. Chris Tobin has been with us ever since day one. Hey, Chris. Welcome in, glad you're here.

Chris: Yes, yes. It's interesting. I'm actually back now. There we go. Well, thank you for that introduction. I appreciate it, and it's good to see that Mitch is doing well in Russia as well, despite all the craziness that's going on over there. Can you guys hear me now?

Kirk: Yeah, I hear you great. Are you at an undisclosed location?

Chris: Oh, no. Not undisclosed. It's well publicized. I'm at DSIRF Systems. It's a systems integrative RF studios, transmitters, IP, you name it. I'm working on a project with them and it's several projects. One of them is a wireless-type project here. It's wireless video over IP cellular-it's pretty cool-and a couple other things involving several buildings in Manhattan as well.

Just behind me over my shoulder, and Mitch is probably seeing this video, he recognizes this. This is a traffic camera. Those of you in the audience who watch television and see traffic reports, these are the cameras. This company does the support and design for that. So for those in New York City, Philadelphia, and Boston, you have metro and shadow traffic. LA and Chicago it's the same thing.

Kirk: You know, with those traffic cameras... once in a while I get to do weather on a TV station in Nashville. 60 or 70 traffic cameras all around Nashville, and of course they'll zoom in on accidents and such. But I love it when TV stations do that thing where they watch maybe a split on the interstate or a merge and they look and see who tries to teach, who gets around, who takes the chances and almost catches an 18-wheeler or something. Do you ever see them do that on TV in New York?

Chris: Once in a while, yes. They'll do a highlight like that, but usually during inclement weather like this week with the snow and ice that we've had. There have been a few video shots where you saw a car sort of running off the road and getting back on and just hitting a van.

Kirk: Geez. All right. Hey, well, let's get right to our guest. Mitch Glider is with us from Sochi, Russia at the site of the 2014 Winter Olympics. Mitch, welcome in. I'm glad you're here.

Mitch: Well, thanks for having me. I totally appreciate the opportunity to be on the 200th show, and certainly always good to see Chris. I'm happy to be here. It's certainly the end of my day. It's about quarter-after-11:00 Sochi time.

Kirk: At night. It's bedtime.

Mitch: Well, that's a relative term, my friend.

Chris: That's so true.

Kirk: Well, we're going to jump right into this. I want to hear some of Mitch's frustrations and victories. Mitch, you have been posting on Facebook ever since you got there, you know, Sochi day 1, Sochi day 2, Sochi day 10, and you've got stories anywhere from the hotel room and hot and cold water, to dirty clothes, to ISDN, to IP connections.

First of all, I guess we ought to set it up and find out why you're there. You said you're using some NBC WiFi, but I don't think you're working for NBC. So tell me what brings Mitch Glider to Sochi besides the obvious Olympic reason.

Mitch: Okay. It's kind of a long, winding story, but I'm here with WestwoodOne Sports. I am the engineer in charge, along with Zack Akey who is my right-hand man and absolutely couldn't be doing any of this without him. Zack runs the WestwoodOne CBS facility in Washington, D.C. So, anyway, we have the Olympic rights from NBC for radio to do the Olympics and it just so happens that we also have other products that we do with them, like NBC Sports Radio Network and NBC News.

So it's kind of a separate deal. So while we're here, we are treated as, I would say, first cousins. We're in their space in the IBC, all of our telecom, and construction needs, and services are coordinated through NBC. Mike Eaby is the Senior Producer here representing WestwoodOne Sports and that's really how it goes. So we're kind of like cousins in this particular remote.

Kirk: All right. So you're there supporting the WestwoodOne radio networks. Now, WestwoodOne, that's had a couple of changing of hands at the moment. Is Cumulus still working out the deal to purchase WestwoodOne or is that a done deal?

Mitch: No, it's a done deal, and we are now under the Cumulus umbrella. It's in the early stages, so we don't want to spend a lot of time on that at this moment I would think. But needless to say the wheels are in motion and I'm here to do this particular job at this particular time and I have a great team back in New York that's carrying on without me, as Mr. Tobin knows, and I'm focused on the remote.

Kirk: Okay. So I got to visit you some months ago in New York in your office, and you described the preparation work that you and your team would be doing in New York, the staging, the construction, the testing of systems. Would you take us through a bit of that? You may have done that on a previous show, but I'd like to understand it again, what you did to get ready to go to Sochi so that when you unpacked everything it was very likely to work.

Mitch: That's a great question. Essentially what we did is we took the as-built documents that Zach Akey prepared for me while he was in London because we set a similar system out. I wasn't on that remote, but I engineered it from New York, and we evaluated that document, everything from [visio drawings] to layout, what we could do to improve it, to expanding equipment, things that we need to do differently, problems that they had with, let's say, the RG21 multi pair cables that we use for distribution, that type of thing. We then basically analyzed what equipment we needed to purchase and add to the existing system that we had, made the proper requests for the purchases, and it did go pretty well.

Eric Wiler, the Senior Vice President of Engineering and Operations, supported me with that and helped me get the funds that I needed. We worked with Howard Deneroff, who's the Senior Executive Producer. I might have his title wrong, but Mr. Deneroff is in charge of WestwoodOne Sports.

We staged everything in a large area, down to the last detail, monitors, computers, racks. We simulate things, we bring in ISDN lines back there, we simulate things to the best of our abilities, we do as much programming and prewiring as possible. Zack came up and visited, I believe, twice prior to the shipment to help go over the system with me and my technicians.

Essentially, right before it was ready to ship we did the kick the plug trick, and what that means is that without any warning or any regard for equipment, we pulled the twist lock out of the 220 plug out of the electrical socket and took everything down, and then plugged it back in and made sure that everything works.

I will just say that everything was fine and we had an experience where the tech power went down unexpectedly. We were not warned here, and because of that preparation, other than us worrying about whether we did... say the word, Zack, on the Cisco router.

Zach: Save run.

Mitch: Save run. We were doing a ton of Cisco work and Zack was saying, boy, I hope that Mr. Montera who did our Cisco work did a Save run. This way when it powered up... so everything went well. So that's really how the preparation went. It was a lot of just reviewing the existing documents, adding to them, and we do everything from [visio drawings] for the elevations and the layout, to very complex look up table-style spreadsheets for all the various wiring.

Kirk: I like that part about kicking the plug out. You've got to have some confidence that the gear that you're using maybe isn't performing a right cycle at that point or, if it is, the gear is designed to work around that. When you did that kick the plug out, what sort of thing maybe were you expecting to find not come up back up correct the next time? Or what actually did need some help coming back up, aside from the Cisco save run?

Mitch: Well, you know, when we did it in New York we knew that the save run had been performed. To be honest with you, with the equipment that I'm using, everything from a 32 KD, to Telos HX1s, to the Zips, to the Comrex Accesses, you know, I'm going to say except for the normal Windows barking at me when I started it back up that, you know, it wasn't happy with the way that we shut down the computers-that type of thing...

Kirk: Right.

Mitch: ...we had no failures. Therefore, when this real world incident happened here at the IBC, me and Zack looked at each other with our eyes popping out of our head and, to be honest with you, I had the utmost confidence that everything was going to be all right. But Mr. Akey over here was the gentleman working with one of our IT guys remotely and the first thing out of his mouth was hoping that things were saved properly, and the story was that he called up this gentleman and woke him up in the middle of the night and his answer to us was, "Well, I don't see a reason why it wouldn't." So we had to wait for about a half-an-hour to find out if he had a reason why he wouldn't have done that save, but everything came up great. Everything's been performing either perfectly, or in a lot of cases, exceeding our expectations.

Kirk: Cool. I've seen a few pictures of what you've been doing there in Sochi, and that was really the first clue I had as to the scope of your work. You got a few audio consoles and reporting work stations. Then you've got a couple big equipment racks full of gear, I guess for all the audio IO back home. Can you kind of take us through what either the rooms are or what the capabilities are?

Mitch: Sure. We have an engineering area that's approximately 10 feet by about 14 feet. Mr. Tobin knows exactly what I'm talking about because, yes, it's the same size as Salt Lake. That's where we keep a 32 KB audio router. That's where we keep various McCurdy meters and test sets to look at things. All four of the Telos Extreme ISDNs are there. The two [inaudible 12:48] are there. The two Comrex Accesses are there, you know, the transmission rack. Everything is broken out from that area with RJ 25 [pair CAT IV].

We use the same concept that the telecom and IT people use, multi-zoned CAT5, six LAN cables within, and then they all break out to essentially what are breakout boxes with 12 RJ 21s on them because we run two cables to each and we had them custom made by Whirlwind, not because they weren't available on the market. You could buy them, but they're plastic and they're cheesy and Whirlwind, for whoever knows the Whirlwind company, they make things bulletproof and stage-ready.

So we have a large long-form studio that has a traditional control room and a studio and we have an SAS SL16 in there and we have the typical turrets for on/off cough and headphone control and [sympty] in the studio itself. You know, there's some bells and whistles in there. We are using the ICM 32 intercom heads, enhanced intercom heads, and they do everything from key up and listen, for listening, for standard four-wire intercomming. We also have two small read-in type stations that are just really Dixon neutral mixers with a Comrex HX1 on them, thank you very much.

And then we have another, what we call, short-form studio which is the same setup. An interesting thing that I think everybody would be really intrigued by is we're using the enhanced multi panel soft heads by SAS and these are essentially single output controllers and XY controllers with volume controls and presets to control the outputs. And rather than carrying a lot of hardware to make these routing changes for the non-SAS consoles, we're using these multi panels quite extensively and we're extremely pleased with them. It's kind of the heart of how we route things around.

So that's really the idea, it's two read-in positions. It's a short-form studio that does updates. It's a combo studio, and then we have the long-form studio. We're airing out to 350 affiliates, including Sirius XM, and we have a wrap-up show every night. We have three two-minute updates an hour, 24 hours a day, as well as quite a bit of hockey play-by-play programming. Everybody can go to WestwoodOne.com, I believe, to get any kind of a scheduling and any kind of highlights.

Multiple Venues & Events = Multiple Audio Engineering Headaches

Kirk: Okay. I've got the idea now of your broadcast center with your work stations and the long form studio where there are several mic positions, audio console there of course, and the couple of racks worth of gear. By the way, if you want to see pictures of this go to Facebook and go to Mitch Glider. I don't guess you probably have to friend Mitch Glider to see the pictures. I don't think they're that private, so you can probably see them there. I've shared a few of those myself. So if you're on Facebook and you're my friend you can go there and see those pictures as well.

Now, Mitch, one thing I'm wondering about is, you know, the Olympics tend to all be... all these broadcast companies have a broadcast center but then they have a lot of remote gear out at the different venues for the different sports. You mentioned hockey play-by-play. Is hockey the only event where you've got connections out to or are there other venues in the area that you also are bringing live audio back from?

Mitch: That's a great question. There's two ways. For the venues, depending on what events WestwoodOne wants to cover with commentary positions, whether it's live or just to cover the tape, we get what's called a commentary position which is essentially a four wire that is brought back into my space, my engineering space, and is entered into the router like you would enter any other four wire system with intercom-key up and listen, talk down-but obviously all that audio is also available to come up on all the consoles, generating mixed minuses, so we have five wired venues.

I'm going to name the ones that I know and I bet you Zach could chime in for the ones that I don't remember. We have two hockey venues. We have the Alpine Skiing Center up in the mountains. We have the figure skating. He's giving it to me right here. Thank you, Zach. We have two hockeys, a figure skating, the long speed track for skating, and Alpine Ski. So those have commentary positions which means they're four wires, OBS, through the magic of fiber and a really great system, gets them back to me in copper.

Kirk: Oh, okay.

Mitch: Now, the interesting thing is that there are a lot of other venues that we do want to cover but at $18,000 of four wire one has to wonder: is it really worth it to get all the four wires? So what we've been doing is we've been ordering unequipped commentary positions and in this particular case we've been using the Axis Portable with a 4G air card that was provided to use by Cell Hire. I'd like to give a shout-out to Cell Hire because these people really know their business, were familiar with the Russian telecom system and provided me, with not only SIM cards for my iPhones that are working beautifully, but these 4G USB air cards.

So the typical setup, let's say, at the opening, closing ceremonies which I was actually present for the rehearsals the other day, we had a Comrex Access with the mixer expander, three headsets, and the connectivity goes back to an Access rack unit, which then we treat as a venue. So you can just imagine that we have the five hardwired four wires and then we consider the access one, we call it, a four wire and then we have another access that runs back to New York that we use for full-time, four wire communication between the engineering and the production.

Kirk: Got you.

Mitch: So that's really all. We have the hardwired venues, and then we have the accesses. Now, we have four accesses floating around between the coast and the mountains and it's up to Mike Eaby to decide which talent and which personality goes to either cover a live event or to do an onsite interview, any of those things. We have as many as three or four excellent producers back here that take in the tape. I use that expression-tape-thank you.

Kirk: Yeah.

Mitch: And set up clips for the short form or segments for the long for like that. So it's kind of like this moving machine of information coming in and one of the jobs we have here in engineering is Michael say, "Can you check in Alpine Ski just to make sure they're sounding good because we want to take them for a live hit on one of the updates but we want you to confirm the connection is good and your levels is good."

So I'll key him up on my intercom and say, "Hey, Joe. How you doing?" He'll say, "I'm good. How you hearing me?" "Great. How am I here?" "Okay. Great." I'll intercom into Mike Eaby and say, "Mike, Joe checked. He sounds great, and he's ready to go." Mike will pot him up on his board. He'll take over the mix minus. He'll IFB him down the line and queue and then away it goes, just like if you guys were doing a talk show and you got phoners coming. You put them on the offline BUS or you put them in queue. Same thing.

Chris: So, Mitch, hey, it's Chris.

Mitch: Yes.

Chris: With the Access as your commentators drop in, I'm curious, what's the delay like? You're using the local 4G. I guess it's GSM out there. How much of a delay did you encounter or is it minimal?

Mitch: Chris, it is so minimal that it's just exceeded our expectations and when Zach walks past me again I'm going to ask him what algorithm we're using on that Access. I don't have it off the top of my head, but I will tell you that these particular 4G cards... we're on the Megaphone carrier, okay? And we can't get it to perform badly.

Now, Mr. Phil Attenburgh, who I know a lot of you people watch [inaudible 21:28] business for a long time. He made a comment on my Facebook. He says, "Do your [inaudible 21:35] dance now, Mr. Glider, because when the 500,000 or a million people all fly in and they all want to get on their Megaphone carrier, let me know how you're making out." So we'll see, but as of right now the air cards are being used for two things: for the access and for laptops. So far they've performed amazing.

Chris: Oh, good. I was just curious because I know with a lot of carriers I've worked with the algorithm itself has minimal delay but then the carrier themselves introduce it on the IP side, so I was just curious.

Mitch: Chris, you, as being one of the pioneers in this, that this IP codec... use it now as opposed to 10 years from now, I'm tell you, you'd be giving me a Tobin grin ear to ear.

Chris: Well, yeah, we did some crazy stuff at Salt Lake City with the Olympics. I do remember that.

Mitch: One second. Zach, what is the algorithm that we're using on the access for the portables?

Zach: On the access?

Mitch: Yes.

Zach: For the portables, it varies.

Z/IP One Claims the Gold

Mitch: Okay. He'll get back to me. He'll give me an example. So that's the story with the accesses. Is this the time of the show where I make a joke? So let me tell you how the Telos Z/IP Portable worked.

Kirk: I didn't know we had a Z/IP Portable.

Mitch: Now, did you say that or did I say that? And who's show is this?

Kirk: So you're saying that you'd like a Z/IP Portable and it'd be nice if when it comes out it worked as well as a Comrex, right?

Mitch: I will tell you this. We're using the Z/IPs as what some people think is our secondary transmit path. It mirrors all four of our ISDN paths. We have four dedicated paths going to New York and we just call them long form, short form, play-by-play, and aux and it doesn't necessarily mean that's what's going down them. It just means after so many years of doing the Olympics between here and New York that certain things... we've all been on the same team for a long time, so if you can imagine, we're using the Z/IP One left and right as long form, short form and Z/IP Two as a backup for play-by-play and aux.

And all I can tell you is that we periodically test the ISDNs and we have New York send back 880 or whatever, and obviously whatever comes back on the ISDNs also comes back on the Z/IPs because it's mirrored coming and going. That's how we're using the Z/IPs, as a true backup. I will tell you that the Z/IPs beat the ISDN in L2 6448. The Z/IP is beating the ISDN. It's a FLAM, but it's Z/IP that's getting here first.

Kirk: Wow. We know that both technologies work and we know that ISDN can stay up for hours, days, weeks, and even months. People use ISDN as a full-time studio transmitter link-in in some cases. So ISDN is the kind of product that's nearing the end of the sunset of its life and IP is really coming into its own in terms of reliability and availability. It'll only get better, but in your situation you're getting more consistent and connectivity through the IP connection from Sochi to New York than you are with the ISDN from Sochi to New York. Is that what you're saying?

Mitch: That is correct. That's correct, and you know what? There's not enough time on this show for me to get into the ISDN situation here but for me, I'll make it brief. When we first got here, they came up perfectly and everything was great. I'll just say that since then they've been up, they've been down. You have to connect both p2p channels here with this particular switch configuration, and we had a situation where I really needed three ISDNs and if I didn't have a fourth it would be fine.

Now, I have four Zephyr Extremes and they're not new and we use them a lot, but they seemed to be in good shape, and some ISDN lines would work on some codecs sometimes. Now, I could take an ISDN in my hand and I could put it into ISDN one and I tried to make it work, have New York dial us, wouldn't work. I put it in the codec underneath, it would work. I'd put it in the next one, it wouldn't work.

I'm just going to end the story there with the ISDNs because it's just been an up and down thing. My management wants the ISDNs to be the primary and that is that for that. Three-second rule. Chris Tobin, one, two, three. Okay. Now, all I can tell you is based on our statistics that we've seen... now, as you know, the Z/IP has excellent statistical metrics in terms of how the unit is performing, as opposed to the audio metering, which not so good [inaudible 27:10].

Kirk: Okay.

Mitch: So my constructive criticism for you guys is work on the metering, but the statistics are great. I think Zach might've told me at one point that the Z/IP was up for approximately four or five days, something like that, without so much as a hiccup, and Zach was running recordings at an hour at a time to try to find a dropout and couldn't.

Yesterday we did a full technical rehearsal where we were feeding various studios down the line and Howard Deneroff was at the far end and anybody that knows Howard Deneroff knows that this gentleman has extremely good ears. He can hear the difference between L2 and L3. He has amazing ears. He was bouncing back and forth between the Z/IP and I heard him say on more than occasion, "It either sounds the same or I think it sounds better." So this is not a plug just because you guys are sponsoring this show. I want everybody to understand something: if it was up to me, the Z/IPs would be the primary and the ISDNs would be the backup. I am just thrilled to death.

Now, we do have a great Internet connection, we have a dedicated 5-meg DIA, and obviously back at the broadcast center we got a huge pipe. Zach did an amazing job of setting up external NATs, bringing them in, did it real clean. There's a lot of other ways to do things and I wanted to begin to get into them because I'm not at that level to explain things, but I will tell you that I know enough to know that they just sound great, and you know what? Tomorrow we're actually going to up the ante a little bit and we're going to try to push the sound quality and the algorithm a little bit and see what we can do, since it's the backup.

Kirk: Let me ask you. You bring up a good question. What algorithm and bit rate are you using now on the IP connection on the Z/IP Ones?

Mitch: Okay. On the Z/IP, Zach gave me some cheat notes here. We're using AAC 128 and AAC ELD ACLD.

Kirk: Okay. So here's a mixture of these for different purposes and AAC a 128 is perfectly adequate audio quality. It'd be interesting to try 256 and see how that works for you because AAC at 256, that's the same rate when you buy something from iTunes. That's what you get-AAC at 256. The low delay, you give up a little bit of quality. Can you hear it? I don't know. If you do low delay at a higher bit rate then it all works out to be about the same, and then the ELD, that's a codec that is made... that's enhanced low delay so it's quick but it also uses the spectral band replication and the top speed on that is 64 kilobits.

In fact, it's a pretty small range. I think it goes from 48 to 64 kilobits-that's the available range. My point is AAC ELD is great if you are really bandwidth limited at your location because it'll work great at 64 kilobits and it's low delay as well.

Mitch: We are not bandwidth limited so you will probably get a call from Mr. Akey tomorrow because I think that since Zach is the one that is really handling the management of the IP and of this remote, I think that you're obviously extremely knowledgeable at this and Zach wants to push the element, as I do. I am a little bit more conservative along the lines of it's working, now leave it alone, and Zach is like, "I'm not going to drive it off the road, what are you worried?" That's not a good imitation of him, by the way.

So I think we're going to give you a call tomorrow about that but really, to put it in a nutshell, that's performing excellently. The access between New York and Sochi is also doing very well, but in fairness, doesn't sound as good, but also we are keeping it in a very low mode for dual mono purposes. It's only for intercom but we do have to reboot in once in a while. Can't leave it up 24/7 without it sounding a little bit funny sometimes. But that's not a knock on them. It's excellent. We're very happy with both products and we do not see another remote like this in the future where the IP will not be the primary, and we'll bring an ISDN or two just in case we're having problems. That's Mr. Akey's opinion and that's my opinion.

Kirk: Cool. Hey, folks. You are watching or listening to This Week in Radio Tech. It's our 200th episode. We've been here for about four years. Chris Tobin is with us. Tom Ray and Chris Tarr are both unfortunately on other assignments right now. They couldn't be with us, but Mitch Glider is here. He is live in Sochi, Russia, the site of the 2014 Winter Olympics. Just about ready to start and they are having a rehearsal of the opening ceremonies today that Mitch was at. It's late at night there. It is almost midnight where Mitch Glider's at.

Our show is brought to you by my employer, and I appreciate this, Telos and the Telos Z/IP One. Mitch was talking about the Z/IP One earlier. This is the IP audio codec that, as we say, drops jaws, not audio. Well, okay. You do have to give it a good connection, but you can drop a few packets here and there and it still sounds just fine.

The Z/IP One is, as we said, an IP audio codec. What does a codec do, for those of you who may be new to this? You feed audio into it, analog or live wire, and coming soon an AES version... you feed audio into it and it codes the audio according to whatever algorithm you set it up for, MPEG Layer 2, or AAC, or AAC Low Delay. You can also have some very basic throwback algorithms like G722 and even G711, so it's compatible with SIP phone calls. Yes, you can actually make a SIP phone call to a Z/IP One.

Then there's also an optional codec if you want aptX. Well known in the broadcast world, aptX Enhanced is a codec that's optionally available for the Z/IP One. The Z/IP One also makes it easy to connect one place to another. For, let's say, less technical users you can go to front panel of the Z/IP One, look up a buddy on your buddy list or your contact list, and just push the connect button.

Through the benefits of the free-to-use Z/IP server you get NAT traversal and presence information. You don't have to know the IP address of the party that you're calling if they also have a Z/IP One. You just connect them together. You can change the coding algorithm and the bit rate on the fly. You don't have to disconnect, reconnect, reboot, none of that like you do with some other codecs. You can change those things on the fly if you find out, hey, we've got a great connection. I think I'll bump this up to AAC at 320 kilobits per second and get the best quality through that we can.

You also don't have that kind of option with ISDN. ISDN with a single BRI line, two barer channels, you're limited to 128 kilobits, and sometimes on international connections you're limited to 112 kilobits per second using AAC, or MP3, or Layer 2. So the possibilities with IP are quite a bit better. You've got a lot more options for providers and you've got more options for bit rates that you can use.

I want to take just a minute. We're going to ask Mitch to describe his setup just a bit more, but I've got a Z/IP One right here in my rack and, you know what? I'm wondering if I can screen share. I've tried this before and it doesn't always work, but let's give it a try and maybe I can't. Oh, well. Tell you what, we'll try it another time.

I wanted to show you how long this thing had been connected. This Z/IP One right here is connected to an office in Hong Kong, and we just got it connected. They just got a port forward done which seemed to be very helpful to the connection on their end, and it's been connected now for four-and-a-half hours. Four-and- a-half hours. There are a few packet drops and losses on the public Internet. We can't help that, but the AAC ELD algorithm puts up with about 5 percent packet loss and conceals those losses.

The folks at Fraunhofer say you either can't hear them or they're very tough to notice. I've been sitting here listening. In fact, they've got it looped back so I'm sending my audio to them and they're sending it right back to me and let's crank it up a little bit here. So this is what I'm sending and what I'm listening to. Hong Kong. This is going two directions, to Hong Kong and back. It's some hip bossa music there.

The point is there, it's easy, it works well, and besides making a connection using our Z/IP server, there's other possibilities too like SIP or NASIP, and also RTP, and symmetric RTP. You can send a stream to it and it'll automatically return a stream back if you know the IP address to send it to.

So those are a few advantages of the Telos Z/IP One IP codec. It does so much. Send contact closure to end to end. They're synchronized with the audio, and RS 232 data. That can go along as well and it's synchronized along with the audio. You send it in, it comes out the same time as the audio. If the audio is delayed two seconds for good buffering, well, the data's delayed two seconds as well.

Mitch, is there anything you'd like to add about the Z/IP One during this short commercial time before we get back into non- biased reporting?

Mitch: I would just like to say we really support your enthusiasm and I am out here. I am in a situation where ISDNs have been very problematic, like I mentioned earlier in the show, and I highly recommend this particular unit for not only the public Internet, but at WestwoodOne we use quite a few already on our private, what we call our Xmission MPLS network, which is a private cloud.

And I can tell you that, again, my boss, Mr. Eric Wiler, is extremely fond of these and he is an extremely picky individual when it comes to IP codecs. He's tried them all and had them all on this network, and I say his because he manages that MPLS network, and the number of Z/IPs that he's using and we're using is growing all the time.

I will say one other thing. When these get back from Sochi, guess where they're going? They're not sitting in a rack for two years. They're going right into a rack room somewhere in the WestwoodOne family.

Kirk: Oh, I thought maybe they're going to your house. You're going to enjoy some music from somewhere.

Mitch: Not yet.

Kirk: I got to tell you, over the Christmas holidays one of our tech support guys, Brian Jones, now, he lives up near the Portland, Oregon area. He's got a Z/IP One and Brian has the most amazing taste in music and he had a great Christmas music station going, and I could've streamed it on the Internet but I say, hey, why don't we stream it to my Z/IP One and I'll just pipe it through the house and listen to it that way.

We streamed from Portland, Oregon to Nashville, Tennessee for five weeks using linear PCM at 2.3 megabits per second over the public Internet, and my Internet may have gone down. I may have unplugged my router at some point, but these things just reconnected without asking to. Just "bam." So for five weeks I listened to a linear PCM stream from Portland, Oregon with great Christmas music from Brian Jones.

The stuff does work and the Internet's getting better. It's not perfect everywhere but it's getting better. Long commercial but thanks, everybody, for doing that and our show is brought to you by, as I said, the Telos Z/IP One IP audio codec. All right. Hey, Chris Tobin, I got a little message for you.

Chris: You do?

Kirk: You've done plenty of work with the folks at Music Cam and I had a nice experience earlier this week where we got a Z/IP One and a Suprema IP to talk to each other using MPEG Layer 2.

Chris: That's excellent.

Kirk: The people involved were just delighted with that operation. You want to speak to your experience with IP audio? Then we'll get back to Mitch.

Chris: Sure, sure. My experience, as Mitch knows, we did some experimenting way back in the day. I'm finding more and more that the IP audio is a great solution you can use for many things. As you found out, Kirk, with Brian Jones, if you have the proper link in between-I'll just call it that-you can do a lot. You know what? Two megabits or PCM uncompressed audio over the Internet, we'll call it, is not impossible. The trick is do you have the proper endpoints? In your case, you probably have an ISP provider who gives you the bandwidth in the proper way. Mitch is located at The International Broadcast Center with the folks at NBC who are most likely using, I'm sure, fiber across the Atlantic back to the States and they've parsed out the bandwidth they need, and they have to set it up so that the switches, whether it be Cisco or... who else is left out there? There's the bay networks and the others that are gone, but if you set up the switches properly, you manage with the packet control where you should, it'll be just like ISDN or a T1.

And then, I'll tell you, I've been doing more stuff with IP in the last six months than I have in the last six years and it's just wild what you can do. As Mitch is finding out, the ISDN it's limited. It's switch network, it's not a package switch, and... You know, circuit switch, that is. Things are changing, so I'm pretty excited.

The Future Looks Bright for AoIP

Kirk: I think we're going to see a day... well, we know the Internet has just gotten better and better. Heck, I used to be on dialup back in the mid-90s when I worked for Scott Studios. I lived in Dallas and dialup was the only option, and it's just gotten better, and better, and better. Here at my house I've had DSL, and I've had cable, and the cable has just gotten better and better.

So we're living in a world that's more and more dependent upon it and, yes, I know there's fits and starts. There are still providers out there who don't get it or there are consumers who don't get it either. Maybe they're using a bad router, a poor setup, or you've got seven bit torrent clients running on your network with literally thousands of people around the world hitting your router asking for a bit of a file from bit torrent. Please, don't do that if you want real-time audio and low jitter.

So these are all things we're learning about and I'm just delighted about where all this is headed. It's a good place. Shoot, TV networks are running 1080p video across the public Internet to back all video and it's a good thing, and we're going to a good place. All right. Mitch Glider. So for the next few days, Mitch, you're going to be involved daily, I'm sure, with trying to make sure problems don't happen and then really fast solving them when they do. Tell me about what you're looking forward to in the next few days as the Olympic Games get started there.

Mitch: Okay. So what's going to happen started tomorrow... really tomorrow will be the last day I think that me and Mr. Akey will probably come in together and leave together. I'm want to get back to it in a minute because Chris Tobin and I did my first Olympics back in 2002, but essentially what'll happen is I'll come in around 8:00 in the morning, Sochi time, and then Zach will probably cross over with me somewhere between 11:00, 12:00, 1:00, something like that and then he'll have to stay into the early hours to help Mr. Eaby record and edit the long-form show.

So, essentially, we go into fireman mode where we have to verify our connectivity, like I was telling you before, checking remotes, whether it's on an access or a four wire for the venue because a lot of these people will do live hits for these updates that are happening three times an hour. So it's going to be our job to make sure that that's all ready, and there's a lot of people working in our studios so there's, you know, my computer's not printing, or I can't get on the Internet, or just really day to day support that any broadcast engineer today has to wear in terms of the amount of hats that you wear when you work in a broadcast environment. So, essentially, that's what we are. We've become firemen. The place is done. It looks great. We take a lot of pride in making the place look a lot nicer than it really should, but that's just the way I was brought up as a remote engineer.

The one I wanted to mention is that when I did my first Olympics with Chris, that was when I first got into utilizing an intercom system along with an audio router to create a way to talk to venues and remotes prior to them going on the air by looping the four wires through an old McCurdy 9500.

Of course now I'm using a 32 KD with an ICM 32, which me and Chris take complete credit for the design and implementation [inaudible 44:46] for its [inaudible 44:47], if you're listening. We have a picture at the NAB where they got the best pic of me and Chris standing next to that ICM 32 because the endless ball-breaking, if you excuse the pun, so they made that product. So that's that whole backstory. You know, me and Chris together kind of developed the whole system that WestwoodOne uses right now.

So that's it. We're firemen, Kirk. We sit around and... But I will tell you, there is work to be done. We do a lot of as-built documents and documenting what we've changed, what we want to change for the next one, list of everything from tools that I forgot, to equipment, to very serious new designs for the IT network that Zach's been working on, and things like that. So that's really what we do with our time, is try to plan for the next one. Also, as a lot of remote engineers know, you're planning for your exit the minute you get here.

Kirk: What does that mean? You mentioned that to me when we talked the other night. What does that mean-to plan for the exit?

Mitch: Okay. When the final flame goes out, there is a mad rush, and in some cases there are restrictions on how long certain parts of the IDC can be inhabited or we have to get how quickly we have to get out. So what we do is we try to arrange our road cases and our [inaudible] and things like that and the way that we run our wires in such a way that the wires that are cut and run are kind of set up one way and the wires that we want to retrieve, like the multi-pair CAT5s with the RJ 21s, I don't want to leave them, you know?

Not only is it an expensive wire but it's a lot of labor to make it, so we start talking about it that night. Me and Zach will stay up all night, likely, and do 90 percent of the packing and get everything set up and then maybe come back and do a couple of quick things. We fly home the day after that, so we have one day to pack, and then we leave. Now, we set it up in two-and-a- half weeks, right? And we pack it up in 12 hours.

Chris: It's well choreographed, Kirk. Trust me. It's not something to take lightly.

Kirk: Chris Tobin, you were involved in the...

Chris: You plan out the cable layout in advance, you know, which ones you're going to leave, which ones you're going to take. If you add anything during the event, you know where to add it and not to and make sure you keep in mind what you're going to take and cut and run. I will say it's a daunting task to do everything in 12 hours and to meet that deadline. It's wild.

Kirk: Mitch, compare this year's Olympics... you've been to the Olympics before, right?

Mitch: Yes, I personally did 2002 with Chris, who was the engineer in charge for that particular remote, and then I was at 2004 and 2006, and then I stopped attending but still was in charge of the engineering since then. There was China, there was Vancouver, and there was London. So China was kind of not really the typical set up that we did, and then Vancouver was more like what we did in the previous at Vancouver, and then London, last year, where Zack Akey did this particular remote with Wally Tinket, who's one of the senior engineers at WestwoodOne. He's now retired.

So Zach had a lot of firsthand experience out in London. He's really the reason why the remote is what it is today, is because of his suggestions, and his hard work, and his skill set. His [inaudible 48:40] skill set is tremendously high. His IT skill set is extremely high, and his organizational skills. I just had to give a shout-out for Zack and make sure everybody knows out there in the community of remotes that there's a new, young engineer out there that is really taking things on by storm. He has the right combination of old school training along with the technology.

So hats off to Mr. Akey. He did London and now he's here with us, and we're also planning Rio. That is what else we do. We talked to some of the other NBC people. We can get some information about some of the logistics there already. They're having meetings. NBC has meetings at this Olympics to discuss Rio, so you're kind of always looking ahead, everywhere from thinking about the pack the day you walk in, to thinking about what's going to happen next time. We think about what equipment is going to stay in the rack, and who's going to try to take stuff away from us, and then we're going to have to fight for money when it's time to come back, so most of the times we're very protective of the rig, as I like to call it.

That's really the whole story. The two years go by so quickly. WestwoodOne does have the rights for the next two Olympics, just so you know.

Kirk: In our few remaining minutes, Mitch, I wonder if you and Chris might have a bit of a conversation, again, comparing your technical setup today and where you think it might go in the future with where it's been in the past. Obviously, IP audio is a player now and it probably wasn't so much four years and especially four years before that. Do you guys have any thought on where we're going with this kind of coverage technically?

Mitch: Well, I'll start, and I promise, Chris, I'll let you get a word in edgewise, I promise. Last time, we were primary a T1 back all with [inaudible 50:38] equipment and just an ISDN for backup, and we were using Rock 10 consoles, which are the old Logitech consoles, and we were using-I'm going to say-we were using the digicard system to play commercials that were prerecorded, and we built it on the spot because it was right after 9/11.

Me and Chris planned it and we took a bunch of equipment and a bunch of connectors and wires, and we built it right there. Now it's a whole different story in terms of how you prep for it, and I'll let Chris add his comments on that.

Chris: Okay. All right. Well... I'm sorry, I was doing two things at once as Mitch was handing it off to me. Well, Mitch is right. We did have the older technology. I'll call it analog and a little bit of digital here and there. I think future, moving forward, definitely IP. I think we're going to see more use of fiber interconnect between devices. I also would like to see, because I know I've done it and played with it and it's pretty cool to do, touchscreen technology to replace the desks or the consoles. That way you could really have yourself a flexible setup in a studio, or edit booth, or at a suite.

And I see moving forward just multi-casting. Like Mitch is saying, they're serving 300-plus affiliates. Well, wouldn't it be cool if right from the venue you multicast the report stream over an IP connection, out it goes somewhere. Maybe it becomes a MPLS network provided by the network for affiliates to be a part of. Rather than an affiliate having their own local ISP, they pay a service charge or a cost to be an affiliate to the network and have an MPLS circuit constructed in a building and now you've got a private virtual connection. All of a sudden the opportunities expand.

These are things that I've been talking to people about but a lot of folks think it's nuts. Then again, as Mitch pointed out, 10 years ago we talked about a few things that people said was nuts and today he's using a Comrex Access in a venue for doing stuff that people would've been like, no way. You can't make it happen.

Mitch: Hey, Chris. I think you've just come up with the topic for the next show is: what's Chris Tobin going to suggest when satellite distribution is either too expensive or no longer practical? Well, I think there lies at least a couple of shows because I know that your head and the way that you think forward is probably already there because that's a whole other conversation.

I agree with Chris a hundred percent, by the way. We've already talked about things like touchscreen surfaces and stuff like that as well. I will point out that we recently did the Grammy remote out at the Staples Center and Gary Clime and his Cumulus crew were married with the WestwoodOne radio road crew for the first time and they used iPads with an Axia system. I'm sure you're aware this.

Kirk: I actually designed it. Or I designed the idea, not the actual...

Mitch: Right, so the interesting concept... we loved that concept and Gary, I think, is very interested in my 25 pair CAT5E multi pair distribution system that I have now. He didn't say that to me personally but I know that he was in Tampa and he was checking it out. We've talked quite a bit and I'm hoping maybe one day to be able to work with that system using my cable and distribution. To Chris's point, they were using an iPad instead of a 367 and stuff like that. I don't really know the particulars because I wasn't on the remote but I just spoke to my personal friend, Anthony Vitziela, who's the engineer out on the West Coast who said that I would've just had a techincal... it would've been a technical dream. I just watered down my thoughts. Thank you, America.

So I'm very excited about the touchscreen situation, Chris. I agree with you. The fiber situation, I couldn't agree with you more. As a matter of fact, in London, Zach just passed me a little note to remind me, that the four wire commentary positions were delivered to Zach via fiber [inaudible 55:04] in a little four-space rack that sat on top of our rack.

The story goes that they handed him a [MATI] output or something along those lines and said here you go, and Zach was like, no, no, no, you're going to go get me some analog cards and I'm going to plug them into my router, so the story goes. But in this particular case we did not get fiber. We got some copper, but that's another story.

I agree with Chris all the way. It's all about IP distribution. I think there's going to be a way for an MPLS to kind of... I think there's going to be a way, and there is a way obviously, to VPN into an MPLS network for remotes and stuff like that, or to get a carrier to drop a big pipe in to an MPLS. The problem is it takes three months right now to get a big MPLS drop. They got to get that down to three weeks or two weeks, and then people are going to start using IP all the time. When they can start delivering quality IP lines the way they deliver an ISDN line, well, then we're going to be in a new world.

Kirk: Gentlemen, with that we're going to have to wrap it up. Chris Tobin, you got any very last comments to toss in for us?

Chris: No, no, no. Good. I think we've covered all bases. I don't want to extend anything more than what we've done. But a lot of questions have been raised with this netcast.

Kirk: Yeah, I guess.

Mitch: Chris, what you really should've said is, well, Mitch never lets me get a word in edgewise anyway so the hour's up, so that's it.

Chris: No, no, no. You're the guest. You're more than happy to say what you have to to get the point across and tell people to educate and inform. That's not a problem. I'm just here in case the connection should be lost because of crazy things in that part of the world, and then Kirk and I can step back and continue on with the conversation.

Mitch: And get a word in edgewise.

Chris: Not that I know anybody at NBC like the Baker Boys that could possibly take care of things that might be a problem on IP while you're talking with us. No, I'm just saying.

Mitch: It's like Dave Maza or anything like that, right?

Chris: Right, exactly. Sorry. Hey, Dave, it's Chris. How's it going? Listen, by the way, that circuit. Yeah, I know. Okay, thanks. Yeah, no problem.

Kirk: Lots of inside baseball here, folks. Hey, you've been watching This Week in Radio Tech, our 200th episode. Whoo-hoo. Four years. Who ever thought we'd be here for four years? Who ever thought that Chris Tobin, Chris Tarr, and Tom Ray would put up with me for that long-and Andrew? I send him a bottle of booze every now and then and he's happy, but if I didn't do that I don't know if I'd be here or not.

Thanks very much to Telos Systems and the Telos Z/IP One for sponsoring, now connected over five-and-a-half hours to Hong Kong. We just got the router set up right and it's pretty cool. I thank them very much, the folks at Telos, for sponsoring This Week in Radio Tech. Thanks, Mitch Glider, for checking in and being with us for an hour of your beauty rest time from Sochi. I appreciate you very much, sir. Thank you.

Mitch: It's my pleasure to be here again and I really enjoy this. I try to keep my head still and not blink as much, so you guys out there have to let me know how I did.

Kirk: You did great, and, Chris Tobin, you look great too, man. That place you're in there, whatever it is, a TV set, it's a little...

Chris: Well, it's an integration company. They do a lot of stuff. Like I said, it's video, it's audio, it's RF, satellite downlinks, satellite uplinks, microwave point to point. There's all kinds of stuff that goes on here. So it's pretty cool. There's several projects I'm involved in so I like it.

Kirk: Well, good deal. Stay warm there in New York. We just had a blast of cold and flurries and you might be getting it next. All right. Thanks very much to Andrew Zarian back at the GFQ headquarters in Queens, New York on the guys from Queens GFG network. Be sure you check out all of the fantastic podcasts, netcasts, on the GFQ network. No matter what your tastes are you will find something to love. I promise you. We'll see you next week for show number 201, next week on This Week in Radio Tech. Bye-bye, everybody.

Telos Alliance has led the audio industry’s innovation in Broadcast Audio, Digital Mixing & Mastering, Audio Processors & Compression, Broadcast Mixing Consoles, Audio Interfaces, AoIP & VoIP for over three decades. The Telos Alliance family of products include Telos® Systems, Omnia® Audio, Axia® Audio, Linear Acoustic®, 25-Seven® Systems, Minnetonka™ Audio and Jünger Audio. Covering all ranges of Audio Applications for Radio & Television from Telos Infinity IP Intercom Systems, Jünger Audio AIXpressor Audio Processor, Omnia 11 Radio Processors, Axia Networked Quasar Broadcast Mixing Consoles and Linear Acoustic AMS Audio Quality Loudness Monitoring and 25-Seven TVC-15 Watermark Analyzer & Monitor. Telos Alliance offers audio solutions for any and every Radio, Television, Live Events, Podcast & Live Streaming Studio With Telos Alliance “Broadcast Without Limits.”