[Raw Msg Headers][Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Own db of blocked IPs



On Tue, Jan 27, 2004 at 11:01:06PM +0100, Robert Kurjata wrote:
> Witaj Marek,
> W Twoim liście datowanym 27 stycznia 2004 (22:03:47) można przeczytać:
> 
> MK> Thank's for the prompt replies. 
> MK>
> MK> If I am to go to the RBL, I need to have the DNS which can use the SQL
> MK> database as the source of records. I have (really!) about 10 seconds from
> MK> the detection of "tricky" IP to the next connection attempt, and during that
> MK> time my DNS needs to start to serve those IPs as blocked.
> MK>
> MK> Any ideas?
> 
> Yes, really try to use a dns based rbl like djbdns. If you need
> something very unusual (you said smthn' about sql) for now I have
> found something interesting:
>       http://untroubled.org/sqldjbdns/
> An postgresql based dns :) It is very, very old, but may be a guide.
> 
> For performance issues I would rather say that it may be inefficient.
> I suppose from the address you are talking of Onet's mail servers. As
> I can imagine it has to process a lot of mails :) and 100 times more
> malicious connections from worms all over the world :)

  That is more of a matter of controlling, how and what is queried
  from the database.  Running thru complicated iterative lookups
  will definitely slow things.

  Optimize (SQL) database key/data sets:
     key='4.3.2.1.rbl123.our.domain:A'
     data='127.0.0.1'

     key='4.3.2.1.rbl123.our.domain:T'
     data='Some explanation, if any'

  Keys in canonized form (whatever you decide that to be),
  no "case insensitive" lookups!

  There is no point in playing mr. kosher-models, and writing canonic
  forms of SQL datasets.  You want to get necessary data with single
  lookups, optimize for it.  (At first there comes an A lookup, then
  latter TXT lookup.  Giving an error on TXT lookup is ok if A has
  been given data reply.  NXDOMAIN is suitable error code.)

> So it would be better to build something like fast dynamicaly updatable DNS
> with HUGE in memory cache if you need such a fast response.
> 10s may be too short to rebuild very big database into zone file for rbldns.

  Such would be madness  :-)

  A while back I converted one system from editing text alias file to
  updating SleepyCat DB in CDB mode, and having routers use that live.

  Frankly, SQL might be nice sounding idea for many things, but usually
  all you really need is solvable with e.g. SleepyCat DB in CDB mode.
  ... and using DB directly can sidestep SQL overhead.  Suitably done,
  you get multithreaded database access free of charge.

  You can even distribute programs to feed data to the centralized
  database into external systems, and let them call SleepyCat DB
  location over SleepyCat's RPC mechanisms to store data.

  With single disk I think you can get at least 30 - 50 updates/lookups
  per second.  With more spindles behind the filesystem where that db
  resides, you might get more IOs.  Running the thing on journaled
  filesystem (like Linux EXT3, XFS, JFFS, et.al.) may or may not be
  slowdown depending on how much changes (new / removed entries)
  you will be pushing in.  If lookups are e.g. 99:1 against updates,
  you don't need to worry about slightly sluggish writes.

  The input goes in in fully random fashion, same applies to lookups,
  thus those two won't benefit anything from B-tree.  Indeed B-tree's
  block-split might harm things.  Better to use B-hash.

  In addition to the A and T entries, you will want to have some more
  data for automated garbage collection.  Nothing is worse in these
  DNS BLs, that obsolete data.

  Because the 'TXT' lookup usually follows 'A' lookup, you might consider
  storing data as:

     key='4.3.2.1.rbl123.our.domain'
     data='127.0.0.1 time_t Some more explanation'

  If the lookup yields result, pick first part of the data
  for the A value.   If then TXT is asked for, skip the
  time_t value, and if 'Some more explanation' is non-
  empty string, return it as TXT data.

  Initial lookup result can stay in DNS server program resident
  lookup cache, and be used for something usefull  :-)

  A garbage collector will scan the database every now and
  then, and remove entries with excessively old entries.
  All that in CDB mode.  If the aliveness period is shortish
  enough, the DNS server can use the timestamp to know, when
  to throw the data away, and claim non-successfull lookup.

> PS.
> 
> If you need some code for it, contact me :) maybe we can do something
> usefull for all, and I'm from Poland too :)
> 
> greetings to all
>  Robert Kurjata

-- 
/Matti Aarnio	<mea@nic.funet.fi>
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to majordomo@nic.funet.fi