authpf, authpf.rules unable to modify filters

[kbeaucha]

Hello all:

I am wondering if anyone has seen symptoms like these.

I use authpf to open access to users who authenticate to our firewall. Since we have many different groups to manage, I have created /etc/authpf/users/Templates where I keep rulesets for the different groups and then in individual users' /etc/authpf/users/Username directories I just create a symbolic link to the appropriate file in Templates. It's worked for quite a while.

Today a user called and said that their putty session would close immediately after they had logged in. In /var/log/daemon I saw:

May 8 11:47:02 our-fw authpf[14121]: pfctl exited abnormally

First I logged in with my authpf account and had no trouble getting authenticated. Since my account links to a different ruleset file, I then created an account that linked to the same ruleset as my other user and got this when I logged in:

pfctl: DIOCXCOMMIT: Device busy
Unable to modify filters

After some tinkering, it seems that if I have a table defined in the authpf ruleset file, pfctl can't load the changes.

I suspect that if I reboot our firewall, this will go away but I'd like to see if I can diagnose the problem better. Any suggestions on other things to investigate?

thx
kmb

[jggimi]

Googling for

pftcl: DIOCXCOMMIT: Device busy

returns only one reported problem, the source of which was an incorrectly upgraded system; the kernel and userland were incompatible. So one possible reason for the error is a Frankensystem. You might want to ensure your /sbin/pfctl and your kernel are from the same -release.

If you are certain your software is in sync, the next step would be to grep through the pfctl and PF source code to determine the logic that would produce the message. You will find pfctl(8) source in src/sbin/pfctl, and pf(4) source in src/sys/net.

[jggimi]

Hah. No wonder only one hit. Well, two, but duplicate finds. You had a typo in your error message. Try Googling with "pfctl" instead of "pftcl". Lots more hits. Including the pf(4) man page. Highlight mine:

DIOCXCOMMIT struct pfioc_trans *io

Atomically switch a vector of inactive rulesets to the active rulesets. This call is implemented as a standard two-phase commit, which will either fail for all rulesets or completely succeed. All tickets need to be valid. This ioctl returns EBUSY if another process is concurrently updating some of the same rulesets.

[kbeaucha]

Sorry - finger trouble. Just washed my hands and can't do a thing with 'em.

So, one possibility is that another process is updating the same rules, which makes me look at my sym-linking of ruleset files. But..., as far as I know we've had multiple users in the same group (same linked file) online simultaneously before with no issues.

kmb

[kbeaucha]

Still thinking about the linking and realized that I may have excluded the possibility of this being the problem.

When I wanted to do more testing I copied user.rules to broken.rules - and my test account was the only one linked to that file.

[jggimi]

OK; if multiple users of your symlinked tree is not the cause, then let's start over. You won't need clean hands.

1. What's the output of $ ls -l /sbin/pfctl ?
2. What's the output of $ sysctl kern.version ?

[kbeaucha]

1. What's the output of $ ls -l /sbin/pfctl ?

-r-xr-xr-x 1 root bin 528664 Aug 17 2011 /sbin/pfctl

2. What's the output of $ sysctl kern.version ?

kern.version=OpenBSD 5.0 (GENERIC.MP) #63: Wed Aug 17 10:14:30 MDT 2011
deraadt@amd64.openbsd.org:/usr/src/s...ile/GENERIC.MP

[jggimi]

OK ... pfctl and the kernel are in sync, you do not appear to have a Frankensystem.

At this point I believe you have five options you may select from. And you may select more than one.

1. Review the source code to determine the logic that is occurring to produce the error, to aid with root cause determination and (if you are a C programmer) perhaps develop and submit a patch if a flaw in logic is uncovered. Even if you do not find the trouble and develop a fix on your own, this may help should you avail yourself of options 2 or 3 and that results in patches being sent to you by a developer for testing.
2. Post an informal problem report to the misc@ mailing list to ask advice, similarly to what you have done here. You will need to include more information than you've posted here, however. Minimally, you should post a complete dmesg and a clarified description of the symbolic structure you've touched on here.
3. Post a formal problem report. The problem tracking database is currently shut down so sendbug(1) may be used as the fill-in-the-blanks form with the results mailed to the bugs@ mailing list.
4. Upgrade the gateway platform (or a test gateway) to 5.1-release, and then to -current, in order to determine if the problem has been eliminated with code committed since August of last year, when 5.0 was readied for release.
5. Wait and hope someone else replies here.

[jggimi]

Oops, forgot to add helpful links, on problem reporting and using the mailing lists:

http://www.openbsd.org/report.html
http://www.openbsd.org/mail.html

[kbeaucha]

I just tried to recheck my main ruleset with "sudo pfctl -nf /etc/pf.conf" and got this message:

pfctl: Current pool size exceeds requested hard limit

[jggimi]

Ah, hah!

$ man pf.conf
/set limit

[jggimi]

Once again -- forget to give you a link. Defaults are described here in the PF User's Guide:

http://www.openbsd.org/faq/pf/options.html

[kbeaucha]

I wanted to see if I could free up some memory space without restarting the firewall.

The man page for pfctl says I can use -F to flush specific types of items, including states but I couldn't see how to report what the current usage level is. I was interested in this because if I'm going to set the option I want to have an idea of what the system is using now.

[jggimi]

See if the pfctl -s or -vs info option is of aid. Systat has pf views, also. Those are the first that come to mind, I'm sure there are plenty of additional tools.

[kbeaucha]

I ran pftcl -vs and saved the output, but I could not map the entries there to a particular pool used by pf. At the same time I had more users calling with problems, so first I tried:

sudo pfctl -F all
sudo pfctl -f /etc/pf.conf

which just returned the same "pfctl: Current pool size exceeds requested hard limit" message. Then:

sudo pfctl -d
sudo pfctl -e

which didn't make any difference either. After that I rebooted the whole firewall.

The problem is gone (for now).

Thanks for your help.

kmb

[jggimi]

The problem is gone (for now).

It will come back.

What was the output of # pfctl -vs info ?

[kbeaucha]

Omitting the Filters and States info, here's the contents:

SOURCE TRACKING NODES:
2xx.2xx.2xx.2xx ( states 1, connections 1, rate 0.0/15s )
age 01:27:04, 3179 pkts, 245041 bytes, rule 23

INFO:
Status: Enabled for 70 days 22:37:59 Debug: err

Hostid: 0x15eae5c6
Checksum: 0x9b99f13557076d1b96d1485fa56fcb1e

State Table Total Rate
current entries 194
searches 11607351726 1893.7/s
inserts 6858155 1.1/s
removals 6857961 1.1/s
Source Tracking Table
current entries 1
searches 7580 0.0/s
inserts 2305 0.0/s
removals 2304 0.0/s
Counters
match 9875286688 1611.1/s
bad-offset 0 0.0/s
fragment 201 0.0/s
short 7597829 1.2/s
normalize 202 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 35048 0.0/s
proto-cksum 0 0.0/s
state-mismatch 30788 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 306 0.0/s
synproxy 0 0.0/s
Limit Counters
max states per rule 0 0.0/s
max-src-states 0 0.0/s
max-src-nodes 0 0.0/s
max-src-conn 60 0.0/s
max-src-conn-rate 249 0.0/s
overload table insertion 306 0.0/s
overload flush states 306 0.0/s

TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 6000 states
adaptive.end 12000 states
src.track 0s

LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000

TABLES:
-pa---- ACU_MAC
-pa-r-- AX_SRV
-pa-r-- AX_STN
-pa-r-- BOGONS
-pa---- BUCCC
-pa-r-- DBLINK
-pa-r-- DEPT
-pa---- DMZ
-pa-r-- FTP
-pa-r-- HRIF_USR
-pa-r-- JAIL
-pa-r-- ME
-pa-r-- PRINT_SRV
-pa-r-- PRIV_DHCP
-pa-r-- PSCL
-pa-r-- R2_SRV
-pa-r-- R2_STN
-pa-r-- SHARE_SRV
-pa-r-- SHARE_STN
-pa-r-- TELNET
--a-r-- TIMBUK_SRV
--a-r-- TIMBUK_STN
-pa-r-- TS_SRV
-pa-r-- TS_STN
-pa-r-- UNI_SRV

OS FINGERPRINTS:
700 fingerprints loaded