This week I spent quite some time troubleshooting an issue with a wireless access point that was no longer registering with its central controller. The cause of the issue came out of an unexpected corner and i wanted to share the background with you all.
The setup is simple, we use a Motorola RFS central controller in our company and have access points in other ip subnets segregated by PaloAlto statefull firewall’s as illustrated in this picture.
This week we were doing maintenance work on the firewall and accidently blocked all traffic between the RFS controller and the remote AP. We quickly saw this error and fixed the rule base, however even though we could see TCP packets flowing the AP was still not re-registering on the RFS controller. No matter how hard we looked at it we could not find traffic being blocked in the firewall log’s. What we did have as a last entry of traffic on our faulty deny rule that was many days old. It seemed as though the devices where no longer talking to each other but when we sniffed interfaces on the RFS and AP we could see traffic flowing.
After putting in a call with a Motorola technician we got the info we needed to help us figure this out. The technician explained the details of the AP to controller traffic and there he mentioned AP and RFS keep in contact using a constant UDP stream on port 24576.
The pudding of the problem was in the words constant. With a PaloAlto statefull firewall UDP traffic sessions remain active for 60seconds by default. When looking at the session info on the PaloAlto for traffic on UDP 24576 we could see the traffic was still being blocked using command:
show session all filter destination-port 24576
show session id <session id>
Session 74
c2s flow:
source: xx.xxx.xxx.xxx [zonexxx]
dst: xxx.xxx.xxx.xxx
proto: 17
sport: 24576 dport: 24576
state: DISCARD type: FLOW
src user: unknown
dst user: unknown
qos node: ethernet1/2, qos member N/A Qid 0
The traffic from RFS and AP has matched the deny rule many days before but seeing that the udp traffic is constant and never stops for +60 seconds the statefull firewall was still processing it over the old session.
by simply killing the deny session we forced the traffic to be re-evaluated by our new firewall rule using command
clear session id <session id>
looking at the new session we know we were in the clear
Session 36270
c2s flow:
source: xx.xxx.xxx.xxx [zonexxx]
dst: xx.xxx.xxx.xxx
proto: 17
sport: 24576 dport: 24576
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
qos node: ethernet1/1, qos member N/A Qid 0
It’s all very logic when you look at it but it did take us a while to figure out and this post might just trigger you if you ever see alike behavior.