In this blog I will describe how to setup and configure BGP between Citrix SD-WAN appliance and rest of the network infrastructure when deploying Citrix SD-WAN in gateway deployment. Also I am going to show how to create custom rule to match specific flow and steer such traffic over virtual path

I am going to use below topology to setup BGP for SD-WAN in in-line deployment and test some its functionalities.

Let’s configure BGP at DC site first, for Branch side process will be exactly the same. To configure dynamic BGP peering go to Configuration->Connections->BGP enable BGP process, provide BGP router-id and AS number. Make sure that Advertise Citrix SD-WAN Routes option is tick, otherwise routes which are learned on the Citrix appliance won’t be advertised to other BGP peers.

Next step is to configure BGP peers Configuration->Connections->BGP -Neighbors. In this example I have three BGP peering for each link configured on the appliance MPLS, LAN and Internet respectively. In addition, policy for MPLS and Internet BGP peer was configured. Policy is similar to creating a route map and ACL to match certain routes and configuring BGP attributes for that neighbor. You can specify the direction to indicate if this policy is applied for incoming or outgoing routes.

The default policy is to accept all routes. You can match routes based on different BGP attributes.

If you look closer at MPLS peer configuration, local DC prefix 172.16.1.0/24 is allow to be advertised to the peer and everything else is deny. For the incoming updated from the MPLS router default policy is used which accept all prefixes.

In terms of LAN BGP peering only default policy is applied which accept all received prefixes and advertise everything towards LAN segment.

If you look at the Internet peering configuration you can see similar configuration for the outgoing policy where only local subnet is advertise. That is required to make sure that DC prefixes can reach the Internet and can be NAT-ed on the firewall. For the incoming policy only default route is allow and everything else is deny.

Following print-screen is for BGP Branch configuration.

When BGP configuration is ready it can be push to the Citrix appliances. First save the config and export it to the Change Management inbox and then press Stage Appliances to apply the config.

So lets verify what routes were learn and advertised through the BGP between Citrix appliances and neighbor peers. You can run following command show_stats routes on the Citrix appliance to verify received routes form your BGP peers. Looking at the routing information on the DC Citrix appliance it seems like we do not get all required routes, there is no entry for the Branch LAN segment 172.16.8.0/24 and default route for the Internet access.

Lets check the Citrix appliance if there is no any audit error on both DC and Branch sites. There are two audit errors which needs to be solved. The error says that dynamic route will not be imported into Virtual LAN route table since no route learning filter exist to include routes. So what exactly it means.

According to the documentation Import Filters is responsible for how route learning takes place. It can only be applied on dynamically learnt routes advertised by neighbouring routers. Import filters are not applied on routes learnt over Virtual Path. Technically speaking it means that without the Import Filters dynamic routes learned via dynamic protocol like OSPF or BGP won’t be insert into the routing table on the appliance. and not advertised to the Citrix SD-WAN Appliances at other Sites. To configure Import Filters go to Configuration->Connections->BGP ->Import Filters. Lets look at some option how kit can be configured.

Below print-screen shows DC site Import Filter configuration, in that specific example default route and 172.16.7.0/24 prefix is allowed.

For the Branch site I used different approach where default route is allow and any prefix length equal to /24.

Below output is from Citrix appliance after new configuration was pushed and shows that Branch site subnet 172.17.8.0/24 was learn through the Overlay network.

Depends on your requirements you can use different approach and use different filters as required to achieve end-to-end connectivity.

Now lets a make a ping between both locations.

So there are end-to-end connectivity between DC site and Branch site LAN segments but how to find out which virtual path the ICMP actually took. To find it out go to Monitoring->Flow where you can get all the information about current flows traversing specific appliance and which path has been selected as the best one. I am going to get these information from DC site box. Looking at the output below you can see some interesting information. First of all you can see that flows statistics are bi-directional. Secondly you can see which path is prefer for such flow, in this example preferred path is via DC-INTERNET-LINK->BRANCH-01-INTERNET-LINK which is shown under Path column.

In addition, there are some other characteristic of this flow such as App Rule ID, DPI information, Direction, Rule ID etc.

Lets simulate some extra latency on the internet and run an ICMP again between both locations. Before I am going to introduce some latency for the Overlay network over the Internet, lets look how healthy both link are. As you can see both links look good.

I introduced 50ms latency on the Internet and as you can see highlighted value shows 51ms. Despite extra 50ms latency for that virtual path, this path is still marked as ‘GOOD’.

I am going to repeat the ping between both sites and then check the flow statistics, to see if there is any difference. Right now the preferred path for ICMP flow is Virtual path over MPLS network due to that latency over the Internet path exceed the threshold of 50ms.

Now assume that something happen with MPLS network and latency increase up to 80ms. When ICMP ping is run again it shows RTT around 110ms.

The highlighted value below indicate that MPLS path has 80ms one way delay.

Looking at the flow information we can see that traffic fall back to the internet link as primary path for the ICMP flow.

As demonstrate above the SD-WAN takes appreciate action based on different metrics like latency, packet lost and jitter to provide intelligent load balancing behind the scene. I did not configure anything special on the Citrix appliance to provide such functionality. That’s the buty of SD-WAN solution that can provide real time monitoring for business critical application and take an action to react to any issue for the underlay network. By default there are some predefined rules for common application, they can be find under Connections->Virtual->Patyhs->Rules. The print-screen below shows these default rules. You cannot change any setting for these default rules however you can create new one or clone existing one and setup according to your application requirements.

Let’s say you have specific subnets you to send over Internet as primary path and fail-over to other available virtual paths when an issue is detected for underlay network. For the test purpose I am going to create custom rule Connections->Virtual Paths->Rules which will match any bi-directional traffic between 172.16.7.0/24 and 172.16.8.0/24. For the transmit mode I picked up persistent mode and preferred path over the Internet transport network. The persistent mode means that traffic for the flow will remain on the same path. This only changes when the path is not available.

I changed some other values for that custom rule like header compression, packet aggregation and persistent value to shows you that actually that class is used when ICMP ping between both these subnets is executed.

There are plenty others available options under LAN to WAN which can be used to change settings like packet drop queue, RED etc.

Let’s push the new config and run ICMP test again to see if our traffic will match new created rule. As you can see below the ICMP traffic is listed under flow statistic and you can see that compression is on. In addition, traffic originated from 172.16.7.1 to 172.16.8.1 hits the Rule ID 0 and return traffic hits the Rule ID 70. Also please note that virtual path for that flow is over DC-INTERNET-LINK->BRANCH-01-INTERNET-LINK and always will use that link as primary path unless characteristic of that link will not match pre-configured thresholds.

To find out what are these rules you can run following command on the CLI show_stats rules to display all rules. As shown below Rule ID 0 match custom rule which was created early.

The Rule ID 70 match return traffic on the DC Citrix appliance.

As you can see BGP configuration and basic rules settings are pretty self explanatory and if someone has understanding of routing protocols can easily test such deployment at home using EVE-NG. There are plenty other features available which I did not describe in this blog like Application Route which is a flexible way to steer your application traffic or build-in firewall which can be use at branch sites to provide security between different zones.