Hello Internet,
I was reading about flowspec the other day. It has been around for a while now however it was on the news just recently due to the Internet outage caused by L3.
Apparently, a rule send by a flowspec controller was misconfigured and took down several BGP peers which in return took other BGP peers to L3 and you already know what happened.
Thus I decided to show a lab that implements flow spec for both IPV4 and IPV6.
Here we go.
The lab was originally done by Decoding Packets, you can see the original here.
Unfortunately, that lab uses a Juniper device as a router/controller and also an ASR9000 virtual appliance as a flowspec controller, which I have neither.
So I decided to modify it with the appliances my EVE lab simulator can use.
Motivation
I will try to motivate the use of BGP flow specification (flowspec for short).
The idea is to have some sort of centralized traffic policy management.
- Access Control List or Firewall rules
- Policy-Based Routing rules
- Traffic rate-limiting rules
- QoS remarking rules
All of the above can be made at each device on your network however if you have multiple points of entry not uncommon for a large ISP, touching all of your devices becomes problematic.
In its basic form flowspec allows you to rapidly deploy and propagate filtering and policing functionality among a large number of BGP peer routers to mitigate the effects of a distributed denial-of-service (DDoS) attack over your network.
It sends a flow that contains information regarding what to block, including networks, ports, or services. It does this by using the existing BGP infrastructure and thus enables you to send rules to basically any device on your network that is BGP accessible.
Topology
Figure 1. shows the topology in use.
A few points about the topology.
- As mentioned above the original lab was made by Decoding Packets. The lab was modified because their lab used some devices I did not have.
- Their lab used a Juniper virtual appliance as a peer and also as a flowspec controller.
- They also used an ASR9000 (or their virtual equivalence) as a flowspec controller.
- I did not have images for those devices so I decided to use ExaBGP as a controller and use Cisco images that I had.
- In addition, their lab used 6PE and 6VPE. This allows running IPV6 on top of an IPV4 infrastructure. I decided to run a native IPV6 scheme all the way.
- The EVE virtual environment is used. Word of caution unless you have enough resources it will be a hog for resources and will be sluggish when using it.
- The devices used are:
- Cisco CSR 1000v virtual appliances
- ExaBGP application running on a Fedora 30 server.
- Of course, you can use any other virtual setup of your choice like for example GNS3. The only thing you need is the IOS images that can support flowspec and the ability to run ExaBGP on a server that can interact with your virtual setup.
A word regarding ExaBGP. ExaBGP is an open-source application that runs on several flavors of Linux. It implements the RFC for flowspec however it can do much more. Since it can talk to BGP peers, it can announce routes, withdraw routes among other things and because it talks directly to BGP peers it allows you to manipulate your BGP peers routing policy. You can learn more about Exabgp here.
The topology uses both IPV4 and IPV6. While you can make a bit more complex I decided to make it simpler.
Good_Guy and Bad_Guy are two external sites that peer to the network via INET7. The rest of the devices are in AS 1 except for the customer routers CE8 CE9 and CE10.
CE9 and CE10 are on their own VRF however for this lab I do not have any flowspec rules, that will be the topic for another blog.
The idea here is to block Bad_Guy for accessing CE8 while Good_Guy would still be able to connect.
Devices
The devices are configured using the addresses shown in figure 1.
The configurations are here. The configuration for ExaBGP is discussed later. The next section outlines the configuration of BGP for the topology.
Setup
Configure the routers with AS 1 to peer among each other. Use OSPF and LDP to bring connectivity for IPV4. This allows us to use the loopback interfaces for peering between routers.
Do the same with IPV6, there is no LDP for IPV6 so peer using the configurations for each interface. Remember the Good_Guy and Bad_Guy are external so they have their own AS numbers. The same for the customer routers (their own AS numbers).
Make P5 a route reflector client to all others in AS 1.
After making sure you have connectivity on IPV4 and IPV6 (for example, ping from Good_Guy to CE8 using the loopback as source) move onto configuring the flowspec address family between R2 and P5.
Make P5 a route reflector client with R2 for flowspec.
After connectivity is achieved and peering is established on both IP and flowspec address family then you are ready to configure the ExaBGP server.
ExaBGP
Install Fedora server 30 as a virtual machine in the EVE environment (or whatever environment you are using, i.e. GNS3).
Using DNF to install ExaBAGP. Caveat the version of ExaBGP from the Fedora repo has an issue and won’t start.
To fix the issue use PIP and install Exabgp this way (follow the instructions on the ExaBGP wiki for this if you do not how to use PIP). It will download the correct dependencies and overwrite the repo version.
Add IPV4 and IPV6 addresses and make sure the server retain them after a reboot. For IPV6 we are using the fact that the server will obtain an IPV6 address from P5 via SLAC. make a note of it you will need it later to peer to the server.
FlowSpec
Now that we have all the pieces in place we can start the configuration of ExaBGP.
Create a file and named it: exabgp.conf, add the following to it and save it under /etc/exabgp.
[root@exabgp exabgp]# more exabgp.conf
neighbor 10.1.1.25 { # Remote neighbor to peer with
router-id 10.1.1.26; # Our local router-id
local-address 10.1.1.26; # Our local update-source
local-as 1; # Our local AS
peer-as 1; # Peer's AS
flow {
route IPV4-TELNET {
match {
source 201.201.201.201/32;
destination 8.8.8.0/24;
destination-port 23;
protocol tcp;
}
then {
discard;
}
}
}
}
neighbor 2001:DB8:2:1::1 { # Remote neighbor to peer with
router-id 10.1.1.26; # Our local router-id
local-address 2001:db8:2:1:250:1ff:fe00:500; # Our local update-source
local-as 1; # Our local AS
peer-as 1; # Peer's AS
flow {
route IPV6-TELNET {
match {
source 2001:201::201/128;
destination 2001:8::0/64;
destination-port 23;
protocol tcp;
}
then {
discard;
}
}
}
}
The statements are self-explanatory. You can take a look at man pages for exabgp.conf for more information.
Basically, you are peering with router P5. Unlike Cisco in which you need to define the address families for each BGP session, ExaBGP understands that it needs to peer via IPV4, IPV6, and then via flowspec.
Here I am blocking telnet from the loopback interface of Bad_Guy going to CE8. Since we want to drop the traffic we just discard it. You can rate limit it if you wish.
Testing
How do we test, well we look to see if the flows are getting to R2 and also we would ping.
Caveats:
- The flows only get installed on ingress routers. This means that P5 while having the flow will not block traffic. This is done on R2 which is the ingress router. Neat is it not.
- Flowspec needs to be enabled in addition to BGP peering on Cisco routers. This is not necessary for example on Juniper devices. You can spot the necessary statements in the configurations of R2 and P5.
On P5 you should see the BGP peers it has for IPV4:
P5#sh ip bgp ipv4 unicast summary ..... Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 2.2.2.2 4 1 3196 3216 10 0 0 2d00h 3 4.4.4.4 4 1 3188 3195 10 0 0 2d00h 4 6.6.6.6 4 1 3189 3198 10 0 0 2d00h 1 10.1.1.26 4 1 1394 1544 10 0 0 23:09:55 0
You should also see something similar on router R2.
The IPV6 output will be the same except that it will have IPV6 addresses instead.
Now check for the flowspec being advertised to P5:
P5#sh ip bgp ipv4 flowspec
.....
Network Next Hop Metric LocPrf Weight Path
*>i Dest:8.8.8.0/24,Source:201.201.201.201/32,Proto:=6,DPort:=23
0.0.0.0 100 0 i
Here “*>i” means it is an internal route (remember BGP treats this as a route being advertised).
For IPV6 you will see:
P5#sh ip bgp ipv6 flowspec
.....
Network Next Hop Metric LocPrf Weight Path
*>i Dest:2001:8::/0-64,Source:2001:201::201/0-128,NH:=6,DPort:=23
:: 100 0 i
As you can see ExaBGP is sending flowspec advertisements to P5.
You should see the same on R2:
R2#sh ip bgp ipv4 flowspec
.....
Network Next Hop Metric LocPrf Weight Path
*>i Dest:8.8.8.0/24,Source:201.201.201.201/32,Proto:=6,DPort:=23
0.0.0.0 100 0 i
.....
R2#sh ip bgp ipv6 flowspec
.....
Network Next Hop Metric LocPrf Weight Path
*>i Dest:2001:8::/0-64,Source:2001:201::201/0-128,NH:=6,DPort:=23
:: 100 0 i
Now on router Bad_Guy try connecting using Telnet with the source interface of loopback 0 to any of the loopback addresses on CE8, you should not be able to connect. You would be able to ping though.
G201# telnet 8.8.8.8 /source-interface loopback 0 Trying 8.8.8.8 ... % Connection timed out; remote host not responding
The same for R2 using IPV6. On the other hand, if you connect from Good_Guy it will succeed.
You can check that R2 is blocking the traffic by using the flowspec command. This is left for the reader.
Summary
As you can see, BGP flowspec is a very powerful tool. What we covered here is just a very simple example of what it can do.
Couple with ExaBGP you automatically update or withdraw flows.
For example, you could have flows sent to a flow collector like the one by SolarWinds or Riverbed or Cisco.
You can then trigger based on the flow information an alert if a DDoS is happening. You have the source, destination, and ports or protocols.
You can then send that to the ExaBGP server (using the API defined) which in turn could add the new rule using whatever tools you wish. Modify the exaBGP config, restart the process, and voila you have dynamically updated your policy.
Many other possibilities exist, I would recommend reading more here.
Until we meet again.
Ciao.