Safely change firewall rules remotely

| 6 Comments

(whoops - Movable Type 4 decided to mess up the permalink from the index page on this entry earlier; it should be fixed now. Darn bugs in run-periodic-tasks, grrh).

One of the NTP Pool server operators emailed recently and told that his server was unavailable for a few hours because he had changed some firewall rules remotely and managed to lock everyone out. Oops!

There's an easy way to do safe remote firewall changes.

He should have used the at daemon to automatically recover! If you don't have a console server then you absolutely must get in the habit of using it before doing anything that can lock you out of your server.

The basic concept is to setup something that in say 5 minutes will undo whatever you're about to do. So if you are messing with the firewall rules, the easy way is to tell it to disable the firewall in five minutes. For example on Fedora and RHEL you can use service iptables stop to stop the firewall, so as root enter:

at "now + 5 minutes"
service iptables stop

You can type a whole list of commands and when you're done, press ctrl-d to stop.

It will look something like:

# at "now + 5 minutes"
service iptables stop
job 6 at Tue Sep 18 17:53:03 2007
#

Another variation to "program" the at daemon would be echo service iptables stop | at "now + 5 minutes".

Anyway, now you go mess with the firewall for 5 minutes and when you're done you delete the at job. You use the atq command to see a list of queued jobs and then atrm to remove it

# atq
6       Tue Sep 18 17:53:00 2007
# atrm 6

Editing firewall rules is a great use case for this, but other neat ways to use at could be:

You mess with the network configuration. Make the system just reboot in case you get it wrong (be sure that the working configuration files are put back; or that you don't edit the configuration files until you've gotten it working with ifconfig).

You take something down for maintenance; use at to schedule it being started again in an hour. So if you forget to restart it, it'll come back in an hour at the latest.

For example, start MySQL replication at 8pm:

$ at 20:00
mysql -uroot -e 'start slave;'
^D
job 8 at Tue Sep 18 20:00:00 2007

Send yourself a reminder (Sandy on the cheap):

$ at 18:32
mail -s "put money in parking meter!" ask@example.com < /dev/null
^D
job 8 at Tue Sep 18 18:32:00 2007

You have to make sure atd is running of course, but on most Unix systems it should be (it's installed on OS X, but isn't running by default).

6 Comments

A related trick is to run a cron job to restart sshd in case of an accident with it. If all's well, it just means an hourly error log entry when it can't bind to the already-busy port.

A related trick I use when I'm testing a new "/etc/pf.conf" is to type within a screen window:

pfctl -v -f /etc/pf.conf; sleep 60; pfctl -d

which loads the new rules, and if for some reason I don't ^C during the sleep, automatically disables the new ruleset leaving all ports unprotected so I can get back in. :)

[...] his server was unavailable for a few hours because he had changed some firewall rules remotely and managed to lock everyone out [...]

That would be yours truly! :-) Woo hoo!

He should have used the `at` daemon to automatically recover!

I would have, if my firewall was UN*X, but it's not. Don't assume...ask, Ask. :-)

Anyway, I did figure how to "stage" rule changes on my FW, in the event I do that again!

Chipster: Uh-oh! :-) I didn't even think of that. How do you stage it on the Cisco to get a fail-safe?

I guess the other tip of advice is to have your console server outside the firewall :-) (or maybe just have a console server for the firewalls, uh, outside the firewall).


- ask

How do you stage it on the Cisco to get a fail-safe?

It's hacky, but in ADSM for ASA, you can create time-based access rules (great for testing).

I'd love a console server. Anyone want to donate a Cyclades? :-)

Cisco has "reload" command. So long as you have not copied the running config to the startup config. http://www.oreillynet.com/pub/a/network/2002/01/02/ciscotips.html
This has saved me several times.

Leave a comment

About this Entry

This page contains a single entry by Ask Bjørn Hansen published on September 18, 2007 8:09 AM.

Find recent content on the main index or look in the archives to find all content.

Pages

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.33-en
/* bf */