I needed to configure a FileMaker Server (FMS) that would be an isolated system and accessible via the Internet.

Besides other precautions, like locking it down on the firewall as much as possible, changing the default SSH port, etc., I wanted to give the system some ability to defend itself against malicious access. FMS will prevent this to some extent, largely due to how the FMS Admin Console and the FileMaker client are designed.

But in order to get a bit more protection, I wanted to install Fail2Ban, an open source package that installs easily on Mac OSX, and scans your log files looking for suspicious activity. I couldn’t find any filter files online for use with FMS however, so I’d have to create my own. The main task to do this is to create the needed regular expressions that will trigger an alarm. This is what I have so far in my filters.d/filemaker-client file:

Fail2Ban configuration file

#

Author: Simon Brown

#

$Revision: 1 $

#

[INCLUDES]

Read common prefixes. If any customizations available — read them from

common.local

before = common.conf

[Definition]

_daemon = fmserverd

Option: failregex

Notes.: regex to match the password failures messages in the logfile. The

host must be matched by a group named “host”. The tag “” can

be used for standard IP/hostname matching and is only an alias for

(?:::f{4,6}:)?(?P[w-.^_]+)

Values: TEXT

#

The filter for Administrator failures may not be useful,

since in testing this always reported null as the IP when

using the .jnlp based admin tool.

failregex = .[(:0)?]” authentication failed on database.
.Administrator authentication failed: .[].*

Option: ignoreregex

Notes.: regex to ignore. If this regex matches, the line is ignored.

Values: TEXT

#
ignoreregex =

This was based on what I saw when generating password failures and also the document at http://help.filemaker.com/app/answers/detail/a_id/7275. The administrator authentication, it turns out, may not be that useful, since oddly the FMS Admin Console failures only listed the IP address when using the command line tool fmsadmin (!). Also, I haven’t verified yet whether there are any useful messages generated in the other log files. Because of Fail2Ban’s design these would require separate filter files anyway.

With the filter file done, I now needed to configure the jail.conf file to use my new filter. Although I used MacPorts to install Fail2Ban, I preferred the config file provided on our Ubuntu Linux systems, so I’ll be relying default actions configured in jail.conf in my jail.conf file (which you will need to create):

[filemaker-client]

enabled = true
port = 5003,http,https
filter = filemaker-client
logpath = /Library/FileMaker Server/Logs/Event.log
maxretry = 6

I had some additional work to do to get to this to work with rules or even the ones provided with MacPorts. The Fail2Ban source generally assumes you’ll be using the iptables firewall for blocking bad addresses, but we need to use either ipfw or pf on Mac OSX. Since I’m using the action “macros” that Ubuntu uses, I can easily change that for all active filters by setting the banaction variable:

banaction = ipfw

The “ipfw” value corresponds to the file at action.d/ipfw. Unexpectedly to me, I needed to edit the localhost value there in order to get the ipfw rules written out correctly, so be sure to set it to your system’s IP address instead of the 127.0.0.1 address that’s in there by default.

Simon.