Detailed network design post about the network edge, Dual ISPs, and BGP.
Ready to live on the edge? In the last design post we talked about remote access VPN, but in this 4th installment of the network design scenario series we will take a detailed look at designs for the network edge. What is the network edge? The network edge is where your network and outside networks connect. In the enterprise world this is your path out to the internet, in the provider world this is generally where you connect to upstream providers or peers. We will cover high and low level designs, different types of topologies such as SMB, enterprise and service provider (SP), look at the building blocks, redundancy options and other considerations.
I've seen a lot of posts over the years on r/networking about questions of what do I do when I have Dual ISPs? or How can I get redundancy with 2 internet providers? How do I fail over? I feel this is probably one of the less understood areas of networks for a lot of folks, perhaps because Border Gateway Protocol (BGP) is very integral to it. Some might struggle with how to perform fail over properly or there are just critical details that are often overlooked. I intend this post to be a detailed guide (as much as possible) to try and provide everything you need to understand and help design your network edge.
I try to explain some things for someone who isn't as familiar, but some topics are too deep to explain or cover in depth, so there are some assumptions on the knowledge level like hardware sizing, cabling, routing/switching, firewalls. This is for educational purposes only.
Basic edge requirements:
Securely connect the internal network to other outside networks
Provide a secure DMZ delineation between internal and external networks
Optional: Provide a secure path for outside networks to your edge services
Optional: Provide redundancy and resiliency for internet/peering
Edge design challenges:
Routing with multiple providers or paths
External to internal routing and internal to external routing
Firewalls, asymmetric routing and NAT
Connectivity options
Budget
Topics:
The Foundation - ISP, Diversity, IP addressing, Redundancy and DMZ
Branch High Level Design
SMB Business High Level Design
Enterprise High Level Design
Service Provider High Level Design
Low level Edge example and details
Low level DMZ example and details
Closing
Laying the Foundation
Before we get into some designs and options we should first talk about the foundation of the edge. Before you get to the first building block its always good to review the business goals, why do you need network edge, what goals does management have for the business or location - will it grow? is there mission critical applications or users this is supporting? How much downtime can be afforded? Getting some of those questions answered will help determine the requirements for the build.
Usually you will have a firewall(s) (I hope!), possibly router(s) and various types of switches, depending on your goals and network size of course. There is also the other side or other provider's equipment - how will the circuit connect from them to you?
Circuits
That is the first building block, who is your provider? There are so many options these days but generally if you're an enterprise you need one to connect to the internet, and well, if you're a provider you have to connect to someone else to reach their network. What type of service level agreement will there be for the circuit? or do you need mission critical 99.99% uptime? because that would be more costly than just a broadband type SLA and that broadband circuit could take longer to repair in the case of an unexpected outage.
How will they hand off to you - will it be a copper RJ45 connection or a fiber hand off? Consider if the minimum point of entry (MPOE) into the building is 500ft away from your telco closet or your primary firewall then you will need to extend from there into your data closet, however copper will not go that far, therefore you would need a fiber hand off.
Where will you put the circuit? if you're an SMB you can probably plug it right into your firewall, however if you're an enterprise or this is like an IXP you will probably have it go into a switch in the DMZ or a border router.
In that case also what if the other provider installs their demarcation equipment in that same MPOE room, is there backup power in there? Because during a power outage if your main telco MDF has backup power but that equipment does not, then you probably won't have a connection! So it's important to consider the provider or peer's equipment and placement during your initial design.
Usually if the circuit is in a data center (DC) it will have backup power from the data center host and same goes for cross connects. Cross connects in the DC are usually single mode fiber (SMF) and layer 2 connections. Remember to consider the cross connect monthly cost in your total cost if operating in a DC or similar.
Diversity
When you have multiple ISP connections coming into a location or for the same network it will be good to scrutinize how they are built on the provider's side. You probably will want diversity between these connections so a failure isn't shared between them. Like if both fibers are on the same aerial telephone pole and a car takes out the pole then both connections will be down. Sometimes it will be difficult to build into the single building from a different path, so the connections might come into the building in the same room but beyond that perhaps diversity can be built in.
You'll want to get the local wiring service centers and Central office (CO) or point of presence (POP) where the circuits go and eventually terminate, so you can work with your providers to ensure the redundant connections you have go to different equipment and buildings and hopefully can use different fiber beyond the "man hole" or "street" if 2 circuits are on a single building (that is usually a term for on the edge of the property).
Even with 2 locations in different geographies it could be possible for different providers to have their circuits use the same POPs or data centers, so it's important to have this built into your design when using multiple connections on the edge. Above we can see a fiber map example with POPs shown, maps like these can help look at diversity.
When in data centers it will be important to ensure your cross connects utilize different equipment as even with different internet providers that could be a single point of failure. Work with your DC team or possibly IXP team (for some SPs) to help ensure proper redundancy. Speaking of data centers, they can sometimes offer an "IP blend" service where they will facilitate the edge connection and then they peer with multiple providers on different fibers/devices for you, this can be a good option in certain scenarios for already built diversity.
IP Addressing
The next building block is your edge IP address scheme. A good thing to do if you are a larger outfit or will have dual connections is to obtain provider independent IP addressing from your regional internet registry like ARIN. Therefore you can announce yourself via BGP to the world with IP addressing that your organization owns, however this could be costly if looking to obtain IPv4 /24 or /23 etc. these days. As for IPv6 blocks, they should be handing those out like hot cakes.
In my opinion its easier and gives you more options to get the fail over you want with your own address space. You would also need to obtain your own BGP Autonomous system number (ASN) as well if taking this path.
If you are a smaller enterprise or this is a smaller remote location then usually you can obtain a /29 or /30 address blocks for the site, however in this scenario you usually cannot use BGP and will have to use another fail over mechanism for that.
Also if building a DMZ, will you utilize public IP address or private IP addressing? If you're able to get like a /48 IPv6 or /23 IPv4 blocks then you probably will have plenty of addresses and options for routing between multiple locations and running services on the edge, but if not then you will have to utilize private addressing, which would have some limitation if running NAT on the edge, you will see that in the high level design section. This brings us to the next block -
BGP and Fail over
Speaking of announcements, how will the outside networks or internet know how to reach you? BGP will be the standard way to do it. The first question for the builder is where will you announce your addresses from? If building a DMZ will it be the firewall or the border router? If from your edge router, will the provider know the next hop inside the DMZ to reach your networks or NAT subnets? or if you are only announcing from your edge routers and performing NAT on your firewall, how will that work?
When utilizing two provider connections you will usually pre-pend your ASN in BGP on the backup connection so you don't receive return traffic from the internet on your secondary connection. This is because AS path is utilized in the BGP route calculations and having a longer AS path means the route will be less preferred by peers. You can also advertise shorter prefixes on the backup connection if you have the option as those will be lesser preferred. I like that because both routes are in the global table so you have better resilience and faster pick up for backup scenarios.
Then for outbound traffic you will utilize local preference from the provider received routes in order to ensure your traffic leaves out the right connection. High local preference is preferred over lower and is applied on your side. Weight often comes up but that is only for the configured router and not recommended. Both pre-pending and local preference are applied with a route-map, we will touch on that later.
Sometimes though BGP is not an option, then you will likely have to use static routing between you and the provider. And then the provider will announce your assigned IP prefix to the internet. The good thing about using BGP is you have a lot of options and knobs for control, with static routing you will be more constrained and have the lesser of the options when it comes to having multiple connections, but it'd been done and can be accomplished.
In the provider network it will be good to know if this connection needs preference or if you are receiving those routes from other peers. E.g. you want to prefer certain prefixes from your new peering neighbor, however for other parts of your network those prefixes are still preferred via another transit neighbor. Also what routes will you be announcing, will you be aggregating or not? Could that mean your peer receives more preferred announcements elsewhere if aggregating? Often BGP communities will be applied in order to carry edge information throughout the network, something to consider if you aren't doing that now.
For both scenarios it will be important to see what the neighbor's BGP policies are to help craft your configuration and design. We will talk about some of the BGP configurations later in the post.
DMZ and Edge Design options
The last building block is really up to the designer and the various factors as discussed like the requirements and what type of network is being built. The larger network with more connections and resiliency requirements will typically have a set of dedicated DMZ switches for the various router, firewall and ISP connections. Although if budget is an issue these switches can also be switches used for another capacity like a service leaf in a spine/leaf design or a core switch in a tiered design and simply utilize an isolated VLAN for the edge functionality.
It's important to size your hardware based on your current needs, but also for future expansion. You won't want a router limited to a certain bandwidth level when you might need more 2 years later, or a firewall that can't handle possible user session growth over the course of a new years. Alternatively if you are building for 1Gb now but might need 10Gb in the future, will the device line card support that? Always take a minute to think about future needs when building.
High Level Design
Next we will look at the various high level edge connection examples. From one connection to having multiple connections.
High Level Topology 1 - Single homed
Here we have the bare bones basic single-homed setup which would likely be a branch office or a very small business's office; with only 1 connection present. There's the switch and firewall to the internet utilizing static routing with a copper hand-off direct into the edge firewall. You'd probably have DHCP or a small static IP block the provider would give you to connect to the internet. You don't have any redundancy hardware or connection wise here. However it is simple! and it meets our basic requirement of secure connectivity to the internet.
High Level Topology 2 - Dual-Homed
Now, lets say a company had the same setup as above but the business has grown a bit now and maybe there was a bad internet outage that caused some extensive down time, so management wants to get better internet redundancy.
You might see a setup like this, which is how I assume a lot of SMB networks are setup. I also know a lot of people constantly ask about IP SLA or ping monitoring for this type of edge network.
Notice there is a Fiber connection now for the primary internet to AT&T and we've added another backup connection from Comcast. Consider when ordering for your primary perhaps having a better medium and uptime agreement. Fiber networks are -generally- newer and have better equipment running them and are preferred over coax broadband or fixed wireless networks. Fiber circuits like DIA usually have better SLAs and dedicated guarantees, but are usually more costly.
This network would probably have 2 blocks of provider assigned IP blocks and just performing NAT out each connection based on where traffic is exiting, that would facilitate the outside routing back to the network. If IPv6 was in the mix you'd probably want provider independent space but we won't dive into that.
The ping method of monitoring a connection for fail over should be discouraged when possible in my opinion (but often recommended and listed in guides as the go-to), still as you can see there are limited options here.
I often think about how if 8.8.8.8 went offline how many outages and connection switches there would be as there's probably hundreds of thousands of devices depending on the uptime of that google DNS IP address to measure connectivity.
As you can image though utilizing internet endpoints like DNS servers to gauge the health of your connection works, it doesn't mean that's the best way. For example what if there is unexpected maintenance on those servers and they go down? or you get throttled due to a timer that is set too aggressively. You'll get a false positive fail over (which will happen).
If you are able to verify connectivity using endpoints you own, like HQ or DC edge network devices of other locations, or cloud load balancer IPs etc., that would be better. Look at asking your provider as some might offer destinations to ping or measure against.
Moreover, you don't necessarily just want to ping the gateway IP you are assigned either, because if that is directly connected to your edge you won't be able to detect an upstream failure. This is why BGP is so great at the network edge because if there is an upstream failure like a circuit down, the advertised routes inbound and outbound for that connection will be withdrawn (due to no BGP connection upstream) thereby causing your network to divert to the backup circuit. Although you would want to monitor the interface state of your WAN if your device has that capability.
When utilizing the ping measurement you'll have to get crafty on how you steer your internal traffic. You'd probably have to use like object monitoring to withdraw the primary static default and then have the secondary one take over, although some devices have the route updates built in to their IP monitoring.
Consider your NAT rules as well, its easier if both your connections are on a single firewall or HA pair. Keep in mind if you don't have DMZ switches but two connection on HA firewalls, you'd have to perform an HA fail over to get to use the secondary connection.
This becomes difficult and is why you also usually can only have the ping method on a single device when dual homed. Having two border routers in front of your firewalls with IP SLA monitoring and object tracking for default routing is a sub-optimal design.
This section was on a 'dual-homed' topology where we have two connections on our single endpoint providing connection redundancy, next we will look at the next tier of a single connection but to dual points.
High Level Topology 3 - Dual Single-Homed
Now we're talking right? We got an actual DMZ, edge routers, and using BGP! We have two edge routers that each have a connection to a diverse provider. Perhaps this is an enterprise that has their edge inside a data center. We can see there is a DC provider Digital Realty IP blend for the primary connection, and a large provider Lumen offering our backup connection. This design provides full physical and logical redundancy.
We can see we are using external BGP (eBGP) to peer with the providers and announce our subnets. Reasonably we have multiple blocks within like a /47 IPv6 and /23 IPv4 where we are able to advertise out each connection to control the traffic. Then we configure iBGP (internal) to facilitate the communication and routing between the firewalls and the border routers. This methodology is the preferred way for communication at the network edge.
As with PING monitoring we have less flexibility compared to BGP. Notice in the diagram we are using the BGP attribute local preference to mark inbound routes which gives the outbound traffic a way to decide which outgoing path to go on. Furthermore, we are deciding which IP prefixes to be advertised out which connection via route filters to thereby influence inbound traffic flowing back from the internet.
It's important to control the inbound and outbound routing using BGP attributes and filtering as you could run into asymmetric routing due to traffic leaving one connection and returning on the over. This could be a problem for firewalls when dual-homed. However, I like the 2 router design to control the perimeter and have the firewalls behind them with an outside interfaces zone, this eliminates asymmetric routing issues from a firewall perspective.
Recognize another challenge we listed at the beginning - NAT (v6 idealists can skip ahead :) . I recommend to NAT on the firewalls as they usually have more horsepower to do this and likewise offer more flexibility vs. running NAT on your border routers.
In addition, consider another challenge - Budget. What if you're not running any outside services and servers or only need a few ports to connect your firewalls, ISP, and routers? That probably doesn't justify a whole other pair of high-end switches for the DMZ. Therefore you could utilize existing switches with separate VLANs only for the outside connections to connect the devices and only have those few ports with that isolated VLAN.
Additionally I like having border routers because they usually have better buffers and options for QoS/traffic shaping, which is where its good to perform. Plus they can usually have all the routing options you need versus a firewall which could be more limited on the BGP side. You could do without the routers and have the circuits terminate into the DMZ switches and carry the isolated VLAN to the firewalls if budget is a problem (or maybe you are running like Arista switches at the edge?) - you decide.
It's important though to have a clear delineation between internal and external networks, so sometimes just that added security (perhaps for compliance) for DMZ switches and/or routers is justified.
High Level Topology 4 - Dual Multi-Homed
For the final High level topology where we are looking at dual connections on dual routers aka Dual Multi-Homed we will look at a service provider type network.
Dissecting this topology a bit we can see how a service provider network might be divided into difference sections, such as Core, aggregation and of course EDGE. The edge plays a key role for the service provider, as larger SPs make money by providing transit or upstream connectivity for smaller SPs, and connect their business and residential customers to the outside world.
They further require more redundancy due to the amount of users or bandwidth capacity they cover. When you support many enterprises there is potentially million of dollars worth of traffic traveling over your circuits, so having redundancy with multiple edge connections is important.
The term active/passive is often replaced with active/active connectivity in dual multi-homed designs. Where there are multiple connections that are all utilized at once depending on where the traffic is coming from. You could have active/passive at each boundary it just really depends on the network and if having monthly costs on unused circuits is worth it.
There will often be a multitude of paths in and out which means the traffic has a lot of different routing options. There is hot and cold potato routing concepts which determine often the way traffic will be handled. Its imperative for the dual multi-homed network operator to understand the different routing paths within the existing network and how the inbound/outbound traffic flows will be effected when adding new circuits or handling failure scenarios. The behavior should be deterministic and predictable which can be achieved with proper BGP configuration (and manners).
For example if you have a circuit or line card fail in an edge POP in a certain metro, will that effect all traffic in that region, or will the traffic flows migrate to a different path out to the internet? it's good to gauge failure domains at the edge. Of course there is always some single homed customers or areas where customers are without protection.
Continuing, looking at the above diagram we see Azure and Lumen connections shown on each side of the diagram which could be completely different geographic regions. It would be good for the provider to have multiple paths not only for redundancy but for performance. Sometimes its beneficial to peer directly with some cloud providers or other SPs to get better latency/performance to their specific networks, rather than transiting another carriers which could be more expensive. Therefore well constructed edges can too increase capacity and performance.
Looking back at what we listed for edge network challenges, a lot of the same problems face service providers as do enterprises, but often in different ways. Like SPs don't always have to worry about NAT or asymmetric routing, since they accept the hot potato routing behavior and the customer equipment NATs. However, like enterprises the SP must delineate between different parts of their network for better control. Either way services are increasingly being brought closer to the border of the network so having high performance edge POPs is becoming more important for the service provider.
Low Level Design
For the low level designs we will focus on the enterprise oriented dual single-homed setup as that will probably apply to most of the readers here. Its provides just enough to cover multiple scenarios and is a proven design, but it isn't too complex
The Edge - Layer 3
Before reading I want you to look closely at all the information on this diagram which is focusing on the final edge between our routers and the neighboring provider's equipment. We see some information for the DMZ and firewalls for context, but most of the information pertains to the routers. This low level could really be applicable to a SP or an enterprise because it has a lot of the same items each administrator would use.
You can see the circuit IDs of the primary and backup paths, cross connect info, interfaces/ports, support info, hardware location info like the building, rack and patch panel, and device name. We see a lot of layer 3 configuration items called out like what are the Route-map names including what prefixes they are covering, the Access-lists for filtering, IP addressing and BGP ASNs. We further see some notes on primary/backup paths and how the BGP attributes are influencing that.
These are all important items that would be covered in the edge design and what I would expect to see in low level documentation on a network diagram. Any other ideas? Let me know.
Routing
We see some information at the bottom from the firewalls like they're using iBGP to the routers, but you could optionally use OSPF. We do need iBGP between the 2 border routers so they exchange the local preference information (only iBGP does) and will fail over to the backup path when needed and that our prefixes are still advertised to the internet.
Think about if the edge routers receive the default route and have a local preference applied (along with the outbound advertisements), then they will know which route to take outbound, but if the firewall isn't using BGP to get those attributes then it won't know. Therefore if using OSPF you would need to control the default route advertisement in OSPF from the edge routers to control the firewalls routing behavior - like redistribution metric or E1/E2, conditional default info originate etc. That is why I usually would recommend BGP from the firewalls to the routers to have BGP capabilities.
One challenge from the first section was internal to external routing, if the firewalls are running BGP to the edge, I would guess you're probably running OSPF inside, so the firewalls would originate the default 0.0.0.0/0 or ::/0 into the inside network based on if they have the default from the border routers. However if there is other places in that OSPF area or domain a default is being injected (like from another sister DC), there could be an administrative distance issue on those firewalls since iBGP received the default with an AD of 200, but the OSPF default is an AD of 110 or lower than iBGP and therefore chosen.
In that scenario you'd have to block the default from the OSPF RIB to FIB so only the iBGP default is present in the FIB. Or you could change the OSPF AD on the firewall which is more dirty in my opinion (pick your poison).
Some internal networks will be pure BGP so this wouldn't be a problem, the routes would be carried in BGP throughout the network.
On the flip side the advantage for an enterprise of using OSPF from the perimeter routers to the firewalls and beyond is that the default originated from the routes will be propagated into the inside network, therefore the previous example problem will not be encountered. This makes it easier for the multi DC/internet exit type network.
OSPF would be for area 0 of course which is the current meta these days. You could run an area or even a NSSA stub which allows default injection, but I would steer away as you'd need to block the default from the FIB again on the internet routers (assuming the NSSA default is injected from the ABR of area 0) which is less than desirable.
As mentioned before it would probably be better to use your public v4 IP space in the DMZ for routing purposes, and if using public NAT pools on the firewalls it will be easier for reachability, as you could just advertise those subnets via iBGP to the edge routers; with OSPF you'd likely have to redistribute a static of the NAT pools.
I usually recommend for the edge routers to be set with Next-Hop Self in the BGP configuration so the upstream peer won't have a next-hop issue with those NAT subnets or other downstream connected prefixes for SPs. For v6 IP space I would recommend to carve out some DMZ networks that would only be used outside, so its easy to identify those areas of the network.
For larger edge POPs, if not interested in running default routes only or if you have inbound services in the DMZ, doing something like partial or full route-tables from each of your providers is an option to utilize multiple circuits at once. This would be an active/active instead of having active/passive setups. You could let traffic flow in and out based on what routes are preferred from which provider on your routers. Keep in mind your hardware will need to be spec'd accordingly.
This maximizes ROI of your circuits and technically can be setup as N+N for routers and internet circuits for your edge. You'd get more session capacity and bandwidth from a network perspective, aside from LBs and firewalls which also helps from a capacity perspective. Still you'll want to make sure your traffic flows are predictable and adjust the routing attributes accordingly.
I show a possible example of this kind of topology above, notice the different call outs which might be more in-line with service provider configurations. Items like BGP communities being amended or removed based on direction to allow better advertisement engineering outbound and within the domain, or running IS-IS instead of OSPF as an IGP. There might be route distinguishers for MPLS present or multi-VRF information. Despite the differences there are still some important network edge configurations regardless of topology, like ACLs, route-maps, IP addresses, circuit IDs and CIRs.
One side note as well, once you advertise your prefixes to the internet you'll want to check various BGP looking glasses to see if the internet can see you; along with checking your ASN pre-pends if applicable.
If you're using provider assigned space, something small like a /28 then you should probably use their looking glass, as anything longer than a /24 will be aggregated beyond the ISP's network.
BGP and Route Filters
If you are taking the BGP route but aren't familiar with prefix lists and route-maps you should definitely do some reading. It's kind of like subnetting at first, a bit confusing but after some practice its easy. You could just use prefix lists for route filters in BGP, but the advantage of using route-maps is you can apply or match BGP attributes like the ones mentioned before.
Just to reiterate you primarily use short/long prefixes filtering and ASN pre-pending for outbound advertisements to notify the internet how to reach you on your primary/backup pathways, and then you use BGP local preference to influence the inbound advertisements to decide the outbound flows for your inside network.
For service providers you can also match or assign BGP communities which are vital to traffic signaling on the network. It's important the SP admin is filtering out prefixes that shouldn't be announced and as a safety mechanism, so a bad configuration change doesn't incorrectly steer traffic and black hole it!
Essentially you configure different prefixes lists with subnets to match and then assign those to different sequences in the route-map. So for the prefix list to match the default route it would be a 0.0.0.0/0 and then for your outbound advertisements you'd list your /24 or /47 etc. because if you are receiving other routes and advertise them back you will possibly become a transit AS. Therefore only advertise your own prefixes so you don't receive traffic not destined fro you (ops accidental DoS).
See a cisco syntax below for example, but the same concepts applies to most vendors.
Default only
v6
RP/0/RSP0/CPU0:router(config)# ipv6 prefix-list default_only_v6
RP/0/RSP0/CPU0:router(config-ipv6_pfx)# permit ::/0
RP/0/RSP0/CPU0:router(config-ipv6_pfx)# deny ::/0 le 128v4
Router(config)#ip prefix-list default_only_v4 seq 1 permit 0.0.0.0/0
Router(config)#ip prefix-list default_only_v4 seq 10 deny 0.0.0.0/0 le 32for your prefix(es)
v6
RP/0/RSP0/CPU0:router(config)# ipv6 prefix-list public_only_v6
RP/0/RSP0/CPU0:router(config-ipv6_pfx)# permit 2001:db8:3330::/47
RP/0/RSP0/CPU0:router(config-ipv6_pfx)# deny ::/0 le 128v4
Router(config)#ip prefix-list public_only_v4 seq 1 permit 3.3.3.0/23
Router(config)#ip prefix-list public_only_v4 seq 10 deny 0.0.0.0/0 le 32
Look at adding remarks to help document the purpose of the prefix list. For SP admins you would likely have the customer or peer prefixes instead of the default only prefix in inbound filtering, this is important to avoid propagating bad route advertisements. Use show commands to verify your configuration.
Next you need to match those prefix lists in route-maps. We'll use the v4 example.
Primary router for primary path, no pre-pend and preferred local preference
Router(config)#route-map BGP_ALLOW_IN_v4 permit 10
Router(config)#description for inbound BGP received routes
Router(config)# match ip address prefix-list default_only_v4
Router(config)#set local-preference 110
!remember we need to set the local preference preferred for router 1
Router(config)#route-map BGP_ALLOW_IN_v4 deny 50
!
Router(config)#route-map BGP_ALLOW_OUT_v4 permit 10
Router(config)#description for outbound BGP advertisements
Router(config)#match ip address prefix-list public_only_v4
!we only want to advertise OUR prefix(es) only, nothing else
Router(config)#set community 3356:110
!setting BGP community based on provider information for local prefernce
Router(config)#route-map BGP_ALLOW_OUT_v4 deny 50
!
end
Secondary router for backup path, we need to pre-pend and have lower local preference
Router(config)#route-map BGP_ALLOW_IN_v4 permit 10
Router(config)#description for inbound BGP received routes
Router(config)# match ip address prefix-list default_only_v4
Router(config)#set local-preference 90
!remember we need to set the local preference less for router 2
Router(config)#route-map BGP_ALLOW_IN_v4 deny 50
!
Router(config)#route-map BGP_ALLOW_OUT_v4 permit 10
Router(config)#description for outbound BGP advertisements
Router(config)#match ip address prefix-list public_only_v4
!we only want to advertise OUR prefix(es) only, nothing else
Router(config)#set as-path prepend 65000 65000
!this will add 2 ASNs to the path thereby making it less preferred
Router(config)#route-map BGP_ALLOW_OUT_v4 deny 50
!
end
What is good about this is that the route-map is what is applied in the configuration, so editing the prefix list is simpler if things are structured well.
Next we tie it all together in our BGP configuration, Cisco syntax below for router 1, but similar concepts are also handled the same with with other vendors. I leave out the iBGP configuration to router 2 and the firewalls from the low level design, only looking at the ISP neighbor and am using the old BGP syntax for example purposes only.
Router(config)#ip bgp-community new-format
Router(config)#router bgp 65000
!for big routers don't forget nsf or nsr etc.
Router(config)#bgp router-id 3.3.3.1
Router(config)#bgp log-neighbor-changes
Router(config)#network 3.3.3.0 mask 255.255.252.0
!if you have that route present or a static null0 then can use network cmd
!next the neighbor
Router(config)#neighbor 1.2.3.6 remote-as 3356
Router(config)#neighbor 1.2.3.6 description primary DIA
Router(config)#neighbor 1.2.3.6 password pasw0rd123
Router(config)#neighbor 1.2.3.6 soft-reconfiguration inbound
Router(config)#neighbor 1.2.3.6 send-community both
Router(config)#neighbor 1.2.3.6 next-hop-self
Router(config)#neighbor 1.2.3.6 route-map BGP_ALLOW_IN_v4 in
Router(config)#neighbor 1.2.3.6 route-map BGP_ALLOW_OUT_v4 out
!
end
Use show commands to verify your configuration. There's other knobs like route-policies and communities that can be used to aggregate and more stream line configurations and filtering but that is beyond the depth of this post.
More links about prefix lists and route-maps are listed at the end of the post.
Edge Filtering
If you recall from the diagram there is a call out for an access-control list (ACL) called EDGE_FILTER_INBOUND. One advantage of using perimeter routers before the firewall and presumable DMZ devices is that you can filter attack and BOGON traffic as its called before it hits the firewall or rest of the network. This can be a good screening process where you can block some large adversary subnets, unused subnets, and also ports for common attack traffic like DDOS amplification which can save resources on the firewall or other devices.
I list some articles to research at the end of the post. You'll want to research a bit on this as to what to contain in this ACL, and remember its placed inbound on your outside interface, don't forget to allow exiting traffic back in!
All SPs should also block BOGONs as a BCP along with their own prefixes inbound usually on peering connections to avoid spoofing or loops. Also common DDoS amplification ports (in my opinion), so the edge ACL can apply to both providers and enterprises. If you're an SP admin and aren't doing this but are considering it, let me know what you come up with.
v4 Cisco syntax ACL example
Router(config)#ip access-list extended EDGE_FILTER_INBOUND_V4
remark ingress filtering on border router to dmz
permit ip host 1.2.3.5 host 1.2.3.6
permit ip host 1.2.3.6 host 1.2.3.5
permit icmp any 3.3.3.1 0.0.0.0
permit icmp any 3.3.3.2 0.0.0.0
deny ip any 3.3.3.192 0.0.0.248
deny pim any any
deny udp any any eq 1433
deny udp any any eq 3389
deny tcp any any eq 3389
deny tcp any any eq 1433
deny udp any any eq 10074
deny udp any any eq 445
deny tcp any any eq 445
deny udp any any eq 4444
deny tcp any any eq 4444
deny udp any any eq 27960
deny udp any any eq 11211
deny tcp any any eq 11211
deny udp any any eq 3702
deny tcp any any eq 3702
deny ip 0.0.0.0 0.255.255.255 any
deny ip 10.0.0.0 0.255.255.255 any
deny ip 100.64.0.0 0.63.255.255 any
deny ip 127.0.0.0 0.255.255.255 any
deny ip 169.254.0.0 0.0.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.168.0.0 0.0.255.255 any
deny ip 198.18.0.0 0.1.255.255 any
deny ip 224.0.0.0 15.255.255.255 any
deny ip 240.0.0.0 7.255.255.255 any
deny ip 248.0.0.0 7.255.255.255 any
deny ip host 255.255.255.255 any
deny ip 3.3.3.0 0.0.0.252 any
!you could have a permit IP any any or you can have it with these and then a deny IP any any, depends on the goals and SP vs. enterprise use cases
permit udp any any eq 30000 50000
permit udp any any eq 4500
permit udp any any eq 500
permit tcp any any established
deny ip any any
The DMZ - Layer 2 and 3
The last section we will look at for the low level design section is the DMZ. This is where the border routers will connect to the edge firewalls or load balancers that provide protection to the internal network.
We can see more detail on the switches and firewall versus our other edge Layer 3 diagram from the previous section as far as what interfaces are involved, specific IP addresses, spanning-tree, and port-channel information.
Also notice the specific call outs to reference other diagrams for information. Perhaps you'd have a detailed diagram for the VPC of the 2 switches, also a detailed diagram of the core network to the firewalls before it reaches the DMZ showcasing those interfaces and OSPF like we talked about earlier.
We can see the DMZ VLANs for connectivity and what IP addresses are being used for what purposes. We can see for the routers there is a an internal management via a VRF. This is a common best practice for the outside zone in that to keep your outside traffic totally segmented logically from a layer 2 (VLAN) and Layer 3 (VRF) perspective. You'd then build this down to your firewalls where you'd have a special zone for outside untrusted management. For you SPs you probably have multiple VRFs for customers and different services but I know some of you don't have VRF segmentation for management as well!
We see that the DMZ switches are logically connected via VPC for high availability (you could also use stacking, or chassis with separate line cards), this allows for LACP port-channels to be used between the 2 routers and HA firewalls to the switches, the advantage of that is there is more resiliency if an interface or optic fails. Plus with port-channels you can remove or add interfaces without impacting anything which is good for maintenance.
Speaking of maintenance, having an edge setup like this makes its easy for maintenance, since you can steer traffic with the 2 perimeter routers based on which one is getting rebooted or changed.
We can see what specific ports are listed here so when we go to connect all the devices or need to troubleshoot something we have a detailed as-built document. Moreover, the diagram shows the detail we need to prove there is a clear delineation between the inside and outside networks.
I like having inside management and outside management loop backs, these will often be the BGP or OSPF router IDs and can be endpoint IPs to test connectivity and such.
Maybe to make this diagram a little better I might list more firewall related info like NAT subnets, zones, make/model etc. and more VLAN information if there were DMZ services running in this outside network. We could also show standard interface configurations for those outside interfaces - this would also be important for when DMZ switches are doing double duty also as functionally inside network switches.
Conclusion
To close out - we covered many aspects of the network edge from connecting to your service provider (or other providers), having adequate diversity, to the challenges of routing in and out of the network. We looked at different high level designs being single or dual homed, using BGP or PING for path control and covered low level looks at the DMZ and Layer 3 Routers at the perimeter. We covered detailed configurations from prefix lists to access-control traffic filters; along with the importance of filtering for BGP and applying attributes for route control.
For both service providers and enterprises the need to connect to outside networks and for those networks to connect back will be important in the future as it is now. We want to route, filter, and design our network edge's properly for our business's needs and I hope this post helps you do that. thank you
Would you like to know more?
