Discussion about an enhanced network module

De Kus · Post by **De Kus** » Tue Jun 06, 2006 5:55 am

I haven't really started the module (just a few lines, mostly from the module manual), since I am not sure, if its new features would be really of use. The general idea was, that "connect" doesnt support proper exeception handling (you never know why the connection was droped), "socket" with "fileevent" is generally slow and it seems "::http" became even more slower. Now I know that nothing is faster than syncronous C network code. Oh yes, sure we should not use syncronous code in the main loop, so of course this needs to be event driven for the TCL frontside and the process must run threaded.

Now I was intending to offer most of socket and bind paremeters as configure option to allow:
- binding sockets to IPs you want (while connect uses my-ip, I believe socket from TCL doesnt)
- binding sockets to ports you want
- binding to UDP sockets
- using temporar syncronous connection for datacollecting
- using asycronous connection with a higher event resolution than fileevent (which seems to be only twice a second. with a boolan var in C Code you could fetch the event within the main loop without lose of noticeable speed)
- implementing some real and fast HTTP stuff with compression support etc.

And then I was wondering, how such a module written for 1.6(1.7) would be portable to 1.9. Someone an idea if it might be even possible to archieve that using CPP?

KrzychuG · Post by **KrzychuG** » Tue Jun 06, 2006 2:11 pm

If it's going to be an enchanged network module consider adding own synchronous and asynchronous DNS functions (host to ip, ip to host) with optional IPv6 support ;)
The one provided by eggdrop DNS module aren't the best and doesn't support IPv6 too.

De Kus · Post by **De Kus** » Tue Jun 06, 2006 6:30 pm

well, actually I was not planing on adding a new dns module... but in a threaded prozess you can lookup the host via gethostbyname, which is basicly the same what eggdrop does (and probably also the dns module does).
I havent thought on IPv6 support, but it shouldnt be that hard to implement ^-^.

demond · Post by **demond** » Wed Jun 07, 2006 2:42 am

I hate to disappoint you, given your enthusiasm, but most (if not all) of the stuff you want to implement is kind of pointless to eggdrop

if you want to know why the connection was dropped and/or wish to bind to particular IP/port, get familiar with [socket] and use it instead of [connect]
while there is little doubt that Tcl network facilities are slower than functionally equivalent pure C application, it's also true that eggdrop is not (or at least not meant to be) a realtime control system having to handle & respond to events happening with very high frequency
therefore a module developed to mirror some of Tcl network facilities on a lower and presumably faster C level, even by someone who has years of network/MT programming in C (who I doubt you are) would provide only negligible and virtually pointless advantage
moreover, it seems to me you don't fully understand some of eggdrop mechanisms you talk about (for example, [fileevent] frequency is not "only twice a second" but depends on how fast your bot handles its other I/O chores in the main loop, since Tcl_DoOneEvent() which fires it is called on each main loop pass and not twice a second)
perhaps you need to do more C programming on less ambitious projects first, before getting on with advanced matters like multithreading (your enthusiasm on the subject is typical for someone who hasn't done much - if any - of it; this stuff is hard to get right, even for experienced professional programmers, and there are many pitfalls you are not aware of for sure)

De Kus · Post by **De Kus** » Wed Jun 07, 2006 3:46 am

I took time messuring on fileevented sockets and the time was always a few ms obove a 500ms resolution, maybe you are right and it isnt fileevent, but the asyncronous queue. But it was too exactly and too predictable to be just a coincidance. Or do you want to tell me eggdrop loops exactly twice a second nomatter how fast it could? I doubt that, it couldnt handle enough events

.

Well, I am not very enthusiastic, I was rather seeking for enthusiasm, since I somehow felt it wouldnt be easy (why has noone else made it? ^-^).

But you should also read a little closer... I believe I have had enough expierience with socket in TCL, and my expierience is, that isn't the you can archive with other. I even have a case in which socket fails to send or recieve any event from a host, while PERL or C do well with that host. Also I have used error event handling with fconfigure and its fine, but however... it has a strange resolution, it feels like its slow on purpose

. And I have no idea why ::http is so slow and why eggdrop supports no decompression of filehandles, but only disc files... (I am sure using ::http with -binary and a header accept with deflate would speed up a lot ^-^).

So I know most features would already exist, the only benefit would be a little configureability and speed (though you should consider that the speed would probably not consume more CPU)

.

KrzychuG · Post by **KrzychuG** » Wed Jun 07, 2006 5:04 pm

De Kus wrote:well, actually I was not planing on adding a new dns module... but in a threaded prozess you can lookup the host via gethostbyname, which is basicly the same what eggdrop does (and probably also the dns module does).
I havent thought on IPv6 support, but it shouldnt be that hard to implement ^-^.

Using threading isn't the best idea too. Some shells admins may not be happy that a single process clones itself even for a short ammount of time.
Eggdrop DNS module doesn't use gethostbyname, it's a real DNS "client" but badly written, slow and not very expandale.

demond · Post by **demond** » Thu Jun 08, 2006 2:22 am

KrzychuG wrote:
De Kus wrote:well, actually I was not planing on adding a new dns module... but in a threaded prozess you can lookup the host via gethostbyname, which is basicly the same what eggdrop does (and probably also the dns module does).
I havent thought on IPv6 support, but it shouldnt be that hard to implement ^-^.
Using threading isn't the best idea too. Some shells admins may not be happy that a single process clones itself even for a short ammount of time.
Eggdrop DNS module doesn't use gethostbyname, it's a real DNS "client" but badly written, slow and not very expandale.

real multithreading is not process cloning (which it used to be, long time ago, in early Linux implementations of POSIX threads)

the real point here is that proper MT programming is hard, and I somehow doubt DeKus has done any of it

Kappa007 · Post by **Kappa007** » Fri Jun 09, 2006 7:42 am

Uhm honestly I can't see any benefit in trying to do some "new network module".

@De Kus:
Could you try to explain a bit more in detail what you want and what is not working as expected?
Did you mix C/POSIX Sockets & Tcl "buzzwords" in your first post?

Anyway if you want to use pthreads:
Any threading attempt in modules is complicated by the fact that the core & Tcl are not thread-safe.
You cannot call any methods of the eggdrop core or read any global data from another thread.
You cannot access the tcl interpreter used by the eggdrop core from another thread.
If you don't want to do any of that => easy
If you want access stuff from core within your thread => hard

De Kus · Post by **De Kus** » Fri Jun 09, 2006 1:22 pm

do you mean the "worest case" scenario, that the thread would write to a var, while another thread is reading from? To be honest, I haven't seriously thought about that. would thread safe than pause reading till writing is done, or does it involve even more? I mean you cannot exclude the fact that the core might read some data, while the thread saves new data coming in ^-^. Well you could pipe the new data to the core which however of course doubles the memory usage for the data

. Might not be serious for 100kb, but might be for several mb ^-^. Not to mention the piping is again an asynrounous socket...

.
Someone mentioned in IRC something about something like unique threading (forgot the name... yeah yeah my memory

), which allows only 1 instance of that thread, but I am not sure if that would already eleminate thread safety

.

I was only rather brainstorming the whole thing, as more as I read, I get the idea it would be to much trouble for relative low use.

Kappa007 · Post by **Kappa007** » Fri Jun 09, 2006 9:59 pm

Actually it has nothing to do with "worst case" scenario.
Think about it like that: The core already is a distinct thread. You want to have a second thread access its data.
You can neither read nor write its data without lock/guard.
Reading might result in reading incomplete data.
Writing might result in overwring data which is currently read by the core.

This ofcourse only applies to non-atmoic operations.