You are here: Home > Publications > RIPE Labs > Andy Mindnich > Deploying IPv6 at IBM

Deploying IPv6 at IBM

Andy Mindnich — 25 Feb 2019
How do you embrace IPv6 in a global enterprise, with tens of thousands of network devices, delivery teams all around the globe and no real shortage of IPv4 space? In this article, I will describe how we introduced IPv6 inside IBM’s own network over the past few years, and look at the challenges we encountered as well as what lies on the road ahead. As enterprises in general seem to lag behind in adopting IPv6 as compared to ISPs or carriers, this article is aimed at helping those who still need to build a case within their own company.

Introduction

Across the Internet ecosystem, people are busy doing things with IPv6. Mobile carriers have it, broadband carriers do it, content providers offer it, operating systems use it, the Cloud wants it, and IoT needs it. But all too often, and especially when we look at enterprises, what’s keeping the people pushing for IPv6 busiest, is convincing key stakeholders that they should invest in their own deployment.

Seven Steps Towards Your Rollout

1. Convince your decision makers

When conversations about IPv6 start taking place in an enterprise, it’s vital that you make the case and have your message resonate with decision makers. One of the ways we did this at IBM was to point to product lifetimes. When you tell your finance team that a product you’re about to install, which works fine without IPv6 right now, will inevitably need to be ripped out long before its expiry date once IPv6 becomes a necessity, you’re not only showing them how they can influence rollout costs through timing, you’re also giving them a good reason to get started fast - ideally now.

2. Nominate a technical champion & find an executive sponsor

You should nominate a technical IPv6 champion. Ideally, this is a person with a technical mindset and who is passionate about IPv6. He or she will be the one to start technical conversations with other senior architects in your organisation. This will help them understand what IPv6 means for their respective area. At a certain size, this technical role can't be done as a 'side-job', in addition to a busy daily schedule. So either allow this person sufficient time, or better still - dedicate a person to this role. Remember, your promise to finance was that by doing this, you will save money in the long run. This might help fund the additional headcount needed.

If you have an established structure of ACBs (architecture control boards), ensure that IPv6 is added as something that should always be asked about during the review process. 

In addition to the technical role, nominate one person from your executive team to be the 'IPv6 sponsor'. That person doesn't necessarily need to be an IPv6 expert. However, having one at this level will help to have questions about IPv6 raised again and again, when new projects are launched, funding plans created, etc. 

3. Get seed funding

You should aim to get some seed funding for IPv6. You don’t need to secure a multi-million dollar budget at the start. You can start small and grow over time. But don’t wait!

Once you have some funding, I would advise you  not waste a lot of your time looking at legacy services. If you focus on new services, as we did in our case, you might be able to dovetail on their development and leverage their roll-out effort. Try to certify every new service being introduced as being IPv6 ready. Also, the willingness of these teams to look at IPv6 is likely to be much higher than of those managing an established service or application. Overall, this focus on new services is likely to yield quicker results than trying to fight the difficult cases.

Once you get a better understanding of your detailed IPv6 architecture and the features you need, start updating your purchasing guidelines for equipment, so that your company only procures devices which will offer the features you need in the future. To start with, you could read this document: ripe-554, we used an approach analogous to it. The key is to identify the particular features you will need at different levels of your network hierarchy. You won't need everything in every device! And don't trust the vendor if all they have to show is a single green check-mark saying 'IPv6-ready' ;-)

4. Be an evangelist!

In my opinion, and this is possibly underestimated by the technically-minded: be an evangelist!
As important as it is to have the technical conversations with the tech team, it’s no less important to reach out to as many groups as possible in your organisation. Talk to the hosting staff, to the front-end load-balancer team, to your client OS folks, building mgmt, endpoint mgmt and security, etc. Basically, talk to as many groups as you can. Keep talking about IPv6 and spread the word. If nothing else, at this early stage of the game, it could be as little as getting your company more prepared for IPv6.

5. Find role models

Whether you’re a corporate enterprise or a multinational, it's good to go out and look at examples of organisations like yours, who've already gone towards IPv6. That said, many of the examples you'll come across will be targeted towards providers and carriers. But there are others out there who’ve done it: talk to your peers, other enterprises, and corporations of the same size, in the same area or industry. Attend conferences, where you can meet like-minded people. Some of them may not have shared their approach in public, but are willing to engage in a private exchange of knowledge between the technical teams of their company and yours. You can learn from their achievements and their mistakes, though of course, in the end, you need to make your own decision. There is no right or wrong.

6. Make an IPv6 addressing plan

This is a big one. Once you have made the decision to move to IPv6, you need to develop an IPv6 addressing plan. Setting up a good addressing plan that fits your organisation, the structure and topology of your network, and your future plans is key.

One important item for us, as a multinational, was to decide where to get our IPv6 allocation from. Do you just use one prefix from one of the Regional Internet Registries (RIRs) or do you approach each of them and get separate prefixes per region? There's no single answer to this question, as the best approach depends on the specifics of your network. In our case, we decided to get just one prefix for our internal network. However, internally, we split the allocation into parts dedicated to specific regions so that, in worst case, we would only have to renumber a region and not the entire network.

Then, once you have received the IPv6 prefix allocated to your organisation, you need to think about the structure of your network and what hierarchy you want to build in, your backbone, and your current network topology (which is likely going to change). You also need to look at how much you want to base it on, say, geography or on the proximity of sites (which might be a more stable parameter).

Although you maybe have a pretty good idea of what your network and business might look like in five years, it's usually difficult to go much beyond that. Therefore you need to come up with a plan that will remain stable enough to last, but which is also flexible enough that you can accommodate the unexpected.

To summarise, I can only recommend that you take your time to make a proper address plan early on.

7. Go native!

We are still heavily dependent on IPv4 these days, so the only large-scale way to deploy IPv6 right now, for us, is to go with dual stack. But this can only be a transition architecture, because it means we're running two IP stacks in parallel, which requires extra maintenance.

I recommend that you deploy native IPv6 wherever you can so as to avoid tunnelling. The beauty of dual stack is that you can still use your existing network management tooling. Your endpoints will choose the best way to connect – over IPv4 or IPv6. This option gives people some comfort that there is still IPv4 to rely upon. However, the additional burden of two stacks is something to consider in the long run. We want to get away from it eventually and get back to maintaining just a single stack in our network: IPv6. We all know that IPv4 will probably be around for decades in some parts of the network. But I recommend building it then as a service encapsulated on top of IPv6, rather than running it natively on your backbone forever.

Spreading IPv6 Around the Globe

Once you've taken these initial steps, it's time to roll out. We operate about 40,000 routers and switches and another 20,000 access points for our offices around the globe. We have very stringent standards of how we roll out things at IBM sites and people sometimes ask me if there is a dedicated IPv6 team. No, there isn’t. You basically need to pull together the experts from each network discipline: the campus architect, the wireless architect, the data centre architect. They really need to understand what IPv6 means in their specific environment. After we made some architectural decisions, we tested and certified the solutions in our lab before starting to roll them out to a few pilot sites. When these were successful, we applied these tested configurations into production with the regular delivery teams in a larger environment.

The positive side-effect of identifying sites in every country around the globe means that you then get a much larger team involved, rather than just your senior architects. All these delivery and service management teams now need to familiarize themselves with IPv6. There is no longer an excuse, because it has become a 'normal' part of the standard configurations. They will, in return, very likely come back to you with questions or findings from the operational aspect, with things you may not have thought of when developing the architecture. So it will start some very good discussions. 

With the thorough testing we had done beforehand, we were able to make this IPv6 roll-out a very smooth one and had very few production issues. You know it's a compliment, and a testament to your technical teams, when your global operational lead (the person usually involved in troubleshooting and fire-fighting problems) tells you: I really haven't heard much about IPv6. 

Achievements

The things we've achieved to support IPv6 in the past few years are:

  • We have native wide-area transport for traditional MPLS or sites over Internet VPN
  • We offer IPv6 for LAN and Wireless LAN, for end-users and into the data centre
  • We offer IP services such as DNS, DHCP and route it outbound on Internet egress gateways
  • We updated our asset inventory, so that we could do IPv6 rollout tracking

So far, we have covered more than 100 sites around the globe, the footprint of over 100,000 IBM staff. The majority of them are actively using IPv6. As soon as you offer IPv6 to clients or endpoints, they will start picking it up and you will see IPv6 production traffic in your network. The majority of our production traffic at this point is destined to IPv6-capable websites and services on the Internet. In parallel we are starting to enable internal services to also be offered over IPv6.

Talking about Europe specifically, we had started with a lab site in the UK, where we have many software developers. They were eager to participate from the beginning and more than once I heard the comment "Oh, finally! Someone has woken up to deploy IPv6." After this successful pilot, we expanded to a site in France, because it met all our technical pre-requisites and was scheduled to undergo some network changes anyhow. We quickly followed with several other countries without facing any problems. Today, there is no country left across Europe where we have not done at least one site. It has become business-as-usual to leverage existing project lifecycles, such as an equipment replacement, to drive IPv6 along with it as a standard service. 

This was the whole promise in the beginning,  to embed IPv6 into the regular project lifecycle, rather than funding a dedicated roll-out project. 

Technical Difficulties on the Way

Bugs

It turned out to be difficult to properly plan the amount of time needed to develop a solution, because of a number of bugs we ran into. You need to test, test and test again. During our lab tests, we came across numerous bugs and worked with our vendors to have them fixed (before we rolled out). 

One of the things we learned is that vendors treat bug reports very differently. Some of them are quite responsive and provide solutions, others say “It's a feature, not a bug”. The quality of resolution depends a lot on how seriously the vendor treats IPv6 in their products.

Monitoring

I said above that “Dual stack is nice because you can fall back on IPv4,” but it also has a downside because it can lead to not noticing when something is wrong with IPv6. The end-users might not even notice IPv6 failing because IPv4 is still working. Even if everything works fine in the beginning, you really need to have a good monitoring system in place, so you can detect any problems with either of the protocol stacks.

Lack of support for enterprises

One of the big issues I noticed when deploying IPv6 at an enterprise is some lack of standards support from the rest of the industry. A lot of transition technologies and other IETF work are geared primarily towards carriers and ISPs, and are not necessarily applicable to enterprises.

Multihoming

If you want to do something like hybrid WAN, where you have some larger consolidated egress gateways in a region, but you also want to do a local breakout in your sales offices, you probably need to think about some form of a translation. Just using a firewall routed egress with preserving the source address is not possible.

In IPv4, you use policy-based routing and both the local breakout and the central part will have different IPv4 NAT pools. So, by nature the return path will be symmetrical and you'll come back in through the same firewall as you left. That's not the case if you want to preserve the IPv6 source address.

There is ongoing work to see how one can use multiple source addresses based on the service you want to reach. But it is still under discussion. It's not something you will find in products today.

Conclusion and Upcoming Activities

In summary, although enterprises are often still bringing up the rear when it comes to IPv6 deployment, there are plenty of reasons to take the lead, and plenty of routes we can use to get there. Be an evangelist, speak up, go to as many people as you can, do internal briefings. Also, as you do begin to make headway, remember to be an role-model. Tell other technical communities how you got there and make them aware of the benefits IPv6 can bring to them in their environment. 

This would not have been possible without many helping hands. For their special support across Europe (and elsewhere), I'd like to thank Ana Espinosa, Norbert Becker and Lonnie Taylor. It's a pleasure to run this program with you!

 

 

 

 

 

3 Comments

Mejan Mollah says:
01 Mar, 2019 04:25 AM
Great article...Thank you team members....
JMA says:
19 Mar, 2019 02:05 PM
Fun article, as WHOIS tells us, it took a few years to get there:

17 years after the IBM Zurich Research allocation (and working connectivity through SWITCH):
```
inet6num: 2001:620:20::/48
netname: CH-IBM-RESEARCH-IPV6
descr: IBM Research - Zurich
created: 2002-08-10T20:38:14Z
```

14 years after the ARIN allocation :
```
NetRange: 2001:49C0:: - 2001:49C0:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
CIDR: 2001:49C0::/32
NetName: IBM-IPV6-01
Organization: IBM (IBM-1)
RegDate: 2005-11-02
```
Still not live/seen in BGP....
Andy says:
19 Mar, 2019 02:43 PM
You looked for the wrong prefix, this is an old one, not in use ;-)
But you are right in saying that it takes a looong time to change something that big.
Oh and if I'm not entirely wrong, thanks for helping push this in Research!
Add comment

You can add a comment by filling out the form below. Comments are moderated so they won't appear immediately. If you have a RIPE NCC Access account, we would like you to log in.