Janko in a Jar

Image via WikipediaWell this week I was again making some spider in REBOL and this time I needed to hold a cookie to be able to crawl some pages. REBOL’s read function has some refinement to set custom http headers where you can also set cookies, but somehow it didn’t work for me. Because I was in hurry I ran listening HttpProxy made a request to that server with FireFox copied the headers and made rebol do the same request with raw TCP. I quickly discovered I need to load more pages like that so some generalization was in order. I made read-ff-cookies function out of it (most of it is a string… {this is multiline string with “quotes” btw in Rebol}):

read-ff-cookies: func [ address cookies ] [
    host: get-host address
    print join "loading: " [ host " ..." ]
    port: open to-url ( join "tcp://" [ host ":80" ] )
    insert port join {GET } [ address { HTTP/1.1
Host: } host {
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en,sl;q=0.7,en-gb;q=0.3
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Cookie: } cookies {

} ]
    result: copy port
    close port
    result
]

Series

btw.. REBOL has this (core) notion of series. You can use series to read/write to files, TCP connections, Strings, data structures, and even REBOL CODE. Series have this bunch of interesting functions that you can then use everywhere. Look at these two examples:

; here we manipulate a string with series functions
>> a: "Hi, I am Janko Metelko"
>> insert find a "Janko" "Mr. " print a
Hi, I am Mr. Janko Metelko

; here we use series functions to generate and then even change
; a function at run time
>> a: copy []
>> insert a "HI!"
>> probe a
["HI!"]
>> insert a 'print
>> probe a
[print "HI!"]
>> do a
HI!
>> say: func [] a
>> probe :say
func [][print "HI!"]
>> say
HI!
>> change next second :say-hi "BYEBYE!"
>> say
BYEBYE!
>> probe :say
func [][print "BYEBYE!"]

An interesting tidbit that also uses series and I copied from some web example is get-host function that I needed in first examle. Definition is:

get-host: func [ url ] [ third parse url "/" ]

Raw TCP

So, to create a TCP client that sends some data and reads the response you do just this:

>> con: open tcp://localhost:5321
>> insert con join "007" newline
>> print copy con
James, James Bond

And if we are making a client, let’s make a simple blocking server too that will act as shown in above client example. You give it the number of agent and it returns your name:

REBOL []
print "secret MI5 server"
agents: [ 	"006" "Seinfeld"
		"007" "James, James Bond" ]
srv: open/lines tcp://:5321
forever [
	con: first srv
	wait con
	insert con ( second ( find agents ( first con ) ) )
	close con
]

—

This entry was posted on Wednesday, November 19th, 2008 at 12:25 am and is filed under Just programming, rebol.