This is the new home of the egghelp.org community forum.
All data has been migrated (including user logins/passwords) to a new phpBB version.


For more information, see this announcement post. Click the X in the top right-corner of this box to dismiss this message.

duplicate bans remover

Requests for complete scripts or modifications/fixes for scripts you didn't write. Response not guaranteed, and no thread bumping!
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

duplicate bans remover

Post by simo »

is there a way to have a monitor for set bans to check for duplicate bans within same range and have the less restrictive ones removed

for example :

Code: Select all

+b *!*@lets.try.this
+b *!*@*.try.this
+b *!*@*.this
to have it leave only the most restrictive no matter what order they have been added so as well if added in this order in reverse to still have leave the
most restrictive :

Code: Select all

+b *!*@*.this
+b *!*@*.try.this
+b *!*@lets.try.this
Pixelz provides this proc as start im not sure how to proper code additional code to make it useable

Code: Select all

proc findRedundantBans {chan} {
set redundant ""
foreach ban1 [chanbans $chan] {
set ban1 [lindex $ban1 0]
foreach ban2 [chanbans $chan] {
set ban2 [lindex $ban2 0]
if {$ban1 eq $ban2} {
continue
} elseif {[matchaddr $ban1 $ban2]} {
lappend redundant $ban2
}
}
}
return [lsort -unique $redundant]
}
User avatar
caesar
Mint Rubber
Posts: 3776
Joined: Sun Oct 14, 2001 8:00 pm
Location: Mint Factory

Post by caesar »

I've seen that you posted a couple of times about this matter and since didn't get any answer was thinking about having a look at this.

On what network are you trying to add those bans? I'm asking cos for example on Undernet the IRC server will remove the other bans if there's a more restrictive one on it's own. I pushed a ban via the bot from DCC chat:

Code: Select all

<cez> .tcl pushmode #channel +b "*!*@something.here.com"
<bot> Tcl: 
<cez> .tcl pushmode #channel +b "*!*@*.here.com"
<bot> Tcl: 
and the result I see on the #channel channel is:

Code: Select all

* bot sets mode: -b+b *!*@something.here.com *!*@*.here.com
I even tried the same thing with my mIRC client and the result is the same:

Code: Select all

* cez sets mode: +b *!*@something.here.com
* cez sets mode: -b+b *!*@something.here.com *!*@*.here.com
so this means the server dose the heavy lifting and leaves only the most restrictive ban on.
Once the game is over, the king and the pawn go back in the same box.
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

thanx for the reply caesar,

its mainly meant for dalnet and some other nets wich dont have that in place to track duplicate bans
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

as networks like dalnet and others are hammered by all kind of floods ending up with a full ban list the smart banlist managing should clean it up and take the load off a bit leaving buffer for extra bans
User avatar
SpiKe^^
Owner
Posts: 831
Joined: Fri May 12, 2006 10:20 pm
Location: Tennessee, USA
Contact:

duplicate bans remover

Post by SpiKe^^ »

Test the function of proc findRedundantBans with this very short script.

Code: Select all

bind pub o|o .cleanbans checkChanBans

proc checkChanBans {nk uh hn ch tx} {
  foreach ban [findRedundantBans $ch] {
    pushmode $ch -b $ban
  }
}

proc findRedundantBans {ch} {  ;# thanks Pixelz #
  set redundant ""
  foreach ban1 [chanbans $ch] {
    set ban1 [lindex $ban1 0]
    foreach ban2 [chanbans $ch] {
      set ban2 [lindex $ban2 0]
      if {$ban1 eq $ban2} {  continue  }
      if {[matchaddr $ban1 $ban2]} {
        lappend redundant $ban2
      } 
    }
  }
  return [lsort -unique $redundant]
}

SpiKe^^

Get BogusTrivia 2.06.4.7 at www.mytclscripts.com
or visit the New Tcl Acrhive at www.tclarchive.org
.
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

tnx spike i tested it it seems to work well

is there a way to have it work on ban settings rather than manually cleanup
User avatar
SpiKe^^
Owner
Posts: 831
Joined: Fri May 12, 2006 10:20 pm
Location: Tennessee, USA
Contact:

duplicate bans remover

Post by SpiKe^^ »

Try this.

Code: Select all

bind mode - "#% +b" checkRedundant

proc checkRedundant {nk uh hn ch mod ban} {
  foreach ban2 [chanbans $ch] {  set ban2 [lindex $ban2 0]
    if {$ban eq $ban2} {  continue  }
    if {[matchaddr $ban $ban2]} {  pushmode $ch -b $ban2
    } elseif {[matchaddr $ban2 $ban]} {  pushmode $ch -b $ban  }
  }
}


bind pub o|o .cleanbans checkChanBans

proc checkChanBans {nk uh hn ch tx} {
  foreach ban [findRedundantBans $ch] {
    pushmode $ch -b $ban
  }
}

proc findRedundantBans {ch} {  ;# thanks Pixelz #
  set redundant ""
  foreach ban1 [chanbans $ch] {
    set ban1 [lindex $ban1 0]
    foreach ban2 [chanbans $ch] {
      set ban2 [lindex $ban2 0]
      if {$ban1 eq $ban2} {  continue  }
      if {[matchaddr $ban1 $ban2]} {
        lappend redundant $ban2
      } 
    }
  }
  return [lsort -unique $redundant]
}

SpiKe^^

Get BogusTrivia 2.06.4.7 at www.mytclscripts.com
or visit the New Tcl Acrhive at www.tclarchive.org
.
User avatar
caesar
Mint Rubber
Posts: 3776
Joined: Sun Oct 14, 2001 8:00 pm
Location: Mint Factory

Post by caesar »

Ah, matchaddr! Don't know why I din't think about this. Probably because got a lot of stuff on my mind at work.. :roll:

Anyway, Pixelz came with a good idea to use that lsort -unique BUT it's used at the end of the proc when should be used at the start on the banlist the bot fetches because this way the most restrictive bans are sorted out first.

Code: Select all

% puts $banlist
*!*@*.here.com *!*@something.here.com *!*@foo.some.com *!*@*.foo.com *!foo@blah.com
% lsort -unique $banlist
*!*@*.foo.com *!*@*.here.com *!*@foo.some.com *!*@something.here.com *!foo@blah.com
this way the most restrictive ones are kept and actually the "redundant" ones are removed.

I know SpiKe at times can't be bothered to correct the code a user came with, but there's no point into looping and asking for the bans twice when can define a variable containing the banlist and loop tough it as many times you want later.

Code: Select all

proc findRedundantBans {chan} {
	if {![botonchan $chan]} return
	set chanbans [chanbans $chan]
	foreach {ban bywho age} [join $chanbans] {
		lappend banlist $ban
	}
	set banlist [lsort -unique $banlist]
	set len [llength $banlist]
	while {$len > 0} {
		set b1 [lindex [split $banlist] 0]
		set banlist [lreplace $banlist 0 0]
		foreach b2 $banlist {
			if {[matchaddr $b1 $b2]} {
				if {[info exists redundant]} {
					if {[expr [lsearch -nocase [join $redundant] $b2]] > -1} continue
				}
				lappend redundant $b2
			}
		}
		incr len -1
	}
	if {[info exists redundant]} {
		return $redundant
	}
}
Tested briefly so you need to do more tests and let me know if get any errors.

I would do a few changes to other stuff as well and turn them into:

Code: Select all

bind mode - "#% +b" checkRedundant
bind pub o|o .cleanbans checkChanBans

proc checkRedundant {nk uh hn ch mod ban} {
	purgeRedundant $ch
}

proc checkChanBans {nk uh hn ch tx} {
	purgeRedundant $ch
}

proc purgeRedundant {ch} {
	if {![botisop $ch]} return
	set redundant [findRedundantBans $ch]
	set len [llength $redundant]
	set max 6
	while {$len > 0} {
		if {$len > $max} {
			set mode [string repeat "b" $max]
			set bans [join [lrange $redundant 0 [expr {$max - 1}]]]
			set redundant [lrange $redundant $max end]
			incr len -$max
		} else {
			set mode [string repeat "b" $len]
			set bans [join $redundant]
			set len 0
		}
		puthelp "MODE $ch -$mode $bans"
	}
}
Once the game is over, the king and the pawn go back in the same box.
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

both work only spikes unsets bans in stacked unlike ceasars

ceasar code only stacks mode settings when less restrictive is set last by chanop while
User avatar
caesar
Mint Rubber
Posts: 3776
Joined: Sun Oct 14, 2001 8:00 pm
Location: Mint Factory

Post by caesar »

Replace

Code: Select all

puthelp "MODE $ch -$mode $bans"
with:

Code: Select all

pushmode $ch -$mode $bans
but I can't test if multiple bans will removed at the same time OR will get some weird behavior, at least I got some on Undernet.

Anyway, there's a key difference in the way I loop vs. the way Spike's code loops. And no, is not about me using while and a foreach and him using two foreach loops cos it's basically the same thing.

The difference lays in:

Code: Select all

set b1 [lindex [split $banlist] 0]
set banlist [lreplace $banlist 0 0] 
Let me explain. Let's say the channel banlist is a list of numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10 (in my example will use from 1 to 5). Now, if we look at the code:

Code: Select all

foreach ban1 [chanbans $ch] {
	set ban1 [lindex $ban1 0] 
	foreach ban2 [chanbans $ch] {
		set ban2 [lindex $ban2 0]
		if {$ban1 eq $ban2} {  continue  }
		if {[matchaddr $ban1 $ban2]} {
			lappend redundant $ban2
		}
	} 
}
And simplify it to see what's going on:

Code: Select all

foreach a [split $bans] {
	foreach b [split $bans] {
		puts "$a vs $b"
	}
}
the result will be:

Code: Select all

% foreach a [split $bans] { foreach b [split $bans] { puts "$a vs $b" } }
1 vs 1
1 vs 2
1 vs 3
1 vs 4
1 vs 5
2 vs 1
2 vs 2
2 vs 3
2 vs 4
2 vs 5
3 vs 1
3 vs 2
3 vs 3
3 vs 4
3 vs 5
4 vs 1
4 vs 2
4 vs 3
4 vs 4
4 vs 5
5 vs 1
5 vs 2
5 vs 3
5 vs 4
5 vs 5
meaning 25 attempts to match one ban mask with another. With just a single lines inside the first loop that will basically remove the first element from the list every time the loop executes:

Code: Select all

foreach a [split $bans] {
	set bans [lreplace $bans 0 0]
	foreach b [split $bans] {
		puts "$a vs $b"
	}
}
the result will be:

Code: Select all

% foreach a [split $bans] { set bans [lreplace $bans 0 0]; foreach b [split $bans] { puts "$a vs $b" } }
1 vs 2
1 vs 3
1 vs 4
1 vs 5
2 vs 3
2 vs 4
2 vs 5
3 vs 4
3 vs 5
4 vs 5
meaning 10 attempts to match one ban mask with another. That's 15 matches that have been already done, meaning a 60% decrease in time used for matching bans.

If where to compare 30 bans then this means 900 attempts vs. 435 attempts ONLY by adding that lreplace line. :)

Code: Select all

% set bans [list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
% set c 0
0
% foreach a [split $bans] { foreach b [split $bans] { incr c } }; puts $c
900
vs.

Code: Select all

% set bans [list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
% set c 0
0
% foreach a [split $bans] { set bans [lreplace $bans 0 0]; foreach b [split $bans] { incr c } }; puts $c
435
This means an 51.66% less matching. If where to compare 50 vs. same 50 then there would be 2500 matches vs. 1225 with just a single lreplace line. That's INSANE! :)

I just noticed that I forgot to take that if {$ban1 eq $ban2} { continue } line into account, but this doesn't make much of a difference, basically with 10 bans instead of 100 will do only 90 matches, still double of the other method that has the lreplace trick in. ;)

I think we can push this a notch further by adding another lreplace, but need to do some testing first. :)
Once the game is over, the king and the pawn go back in the same box.
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

hm ok yes i think u already mentioned it since what i also discovered after testing is the more bans match then it removes most but leaves like 2 or 3
User avatar
caesar
Mint Rubber
Posts: 3776
Joined: Sun Oct 14, 2001 8:00 pm
Location: Mint Factory

Post by caesar »

Something is not working or what are you saying? Mind posting an example so I can come on Dalnet and test stuff on my own?
Once the game is over, the king and the pawn go back in the same box.
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

it doesnt remove all matching duplicates
sets mode: +b *!*@*.IP
sets mode: +bbbbbbbbbbbbbbb *!*@RifSytes-zcd.299.27.20.IP *!*@RifSytes-599.460.33.69.IP *!*@RifSytes-3g5.144.20.79.IP *!*@RifSytes-066.185.88.80.IP *!*@RifSytes-ll9.546.74.68.IP *!*@RifSytes-6u3.339.13.92.IP *!*@RifSytes-kqs.407.57.83.IP *!*@RifSytes-b7v.883.54.19.IP *!*@RifSytes-56d.862.69.27.IP *!*@RifSytes-ox5.511.75.48.IP *!*@RifSytes-6sw.885.18.89.IP *!*@RifSytes-5i6.288.53.70.IP *!*@RifSytes-v32.936.41.94.IP *!*@RifSytes-54f.459.78.41.IP *!*@RifSytes-bwp.836.85.33.IP
sets mode: +bbbbbbbbbbbbbbb *!*@RifSytes-m36.196.43.47.IP *!*@RifSytes-915.770.49.94.IP *!*@RifSytes-5qy.659.08.92.IP *!*@RifSytes-3ut.148.34.53.IP *!*@RifSytes-3o9.200.73.87.IP *!*@RifSytes-pm7.923.75.08.IP *!*@RifSytes-996.695.55.62.IP *!*@RifSytes-gk0.421.24.66.IP *!*@RifSytes-071.544.81.59.IP *!*@RifSytes-sv8.105.76.27.IP *!*@RifSytes-5ob.997.12.63.IP *!*@RifSytes-nh3.819.91.12.IP *!*@RifSytes-50n.997.16.02.IP *!*@RifSytes-4q7.694.07.31.IP *!*@RifSytes-o8a.187.92.31.IP
sets mode: -bbbbbb *!*@RifSytes-zcd.299.27.20.IP *!*@RifSytes-599.460.33.69.IP *!*@RifSytes-3g5.144.20.79.IP *!*@RifSytes-066.185.88.80.IP *!*@RifSytes-ll9.546.74.68.IP *!*@RifSytes-6u3.339.13.92.IP
sets mode: -bbbbbb *!*@RifSytes-kqs.407.57.83.IP *!*@RifSytes-b7v.883.54.19.IP *!*@RifSytes-56d.862.69.27.IP *!*@RifSytes-ox5.511.75.48.IP *!*@RifSytes-6sw.885.18.89.IP *!*@RifSytes-5i6.288.53.70.IP
sets mode: -bb *!*@RifSytes-v32.936.41.94.IP *!*@RifSytes-54f.459.78.41.IP
sets mode: -bbbbbb *!*@RifSytes-m36.196.43.47.IP *!*@RifSytes-915.770.49.94.IP *!*@RifSytes-5qy.659.08.92.IP *!*@RifSytes-3ut.148.34.53.IP *!*@RifSytes-3o9.200.73.87.IP *!*@RifSytes-pm7.923.75.08.IP
sets mode: -bbbbbb *!*@RifSytes-996.695.55.62.IP *!*@RifSytes-gk0.421.24.66.IP *!*@RifSytes-071.544.81.59.IP *!*@RifSytes-sv8.105.76.27.IP *!*@RifSytes-5ob.997.12.63.IP *!*@RifSytes-nh3.819.91.12.IP
sets mode: -bb *!*@RifSytes-50n.997.16.02.IP *!*@RifSytes-4q7.694.07.31.IP
User avatar
SpiKe^^
Owner
Posts: 831
Joined: Fri May 12, 2006 10:20 pm
Location: Tennessee, USA
Contact:

Post by SpiKe^^ »

In that test above, the script appears to have left 2 bans that should have been removed.

This seems like an extreme example, no one would have 30 similar bans set on a channel.

Which of these 2 scripts are you running??
Last edited by SpiKe^^ on Tue Jan 30, 2018 9:32 am, edited 1 time in total.
SpiKe^^

Get BogusTrivia 2.06.4.7 at www.mytclscripts.com
or visit the New Tcl Acrhive at www.tclarchive.org
.
s
simo
Revered One
Posts: 1069
Joined: Sun Mar 22, 2015 2:41 pm

Post by simo »

Its an extreme example Indeed with 30 duplicates, i also tested with like 15 duplicates with same result

also i thought of the example of *!*@* set by accident will result in entire ban list removal and should prob be exempted from triggering

I used your version in this test spike
Post Reply