Overclock.net banner

1 - 13 of 13 Posts

·
Premium Member
Joined
·
6,520 Posts
Discussion Starter · #1 ·
Hey fellow programmers. I am trying to write a program in python that monitors my subscription feed and sends me a text when it changes but I need to log on to view it. I have google'd around and every solution I have come across doesn't seem to work with OCN so if anyone could give me any insight or advice, it would be greatly appreciated. Here is what I have so far (still a work in progress):

Code:

Code:
#!/usr/bin/env python

import urllib, time, os, smtplib

url = "http://www.overclock.net/users/subscriptions/index/view/activity"
source = urllib.urlopen(url).read()
new_source=source 
counter=0
flag=True

while new_source==source:
    if flag:
        time.sleep(5)
    try:
       new_source = urllib.urlopen(url).read()
    except IOError:
        print "Error: cannot read URL"
    flag=False
    continue 
    counter+=1

    flag=True

fromaddr = '[email protected]'
toaddrs = "[email protected]"
body = "New Item Available!"

msg = 'From: [email protected]\n\n' + body

smtp_server = 'email-smtp.us-east-1.amazonaws.com'
smtp_username = '*****'
smtp_password = '*****'
smtp_port = '587'
smtp_do_tls = True

server = smtplib.SMTP(
host = smtp_server,
port = smtp_port,
timeout = 10
)

server.starttls()
server.ehlo()
server.login(smtp_username, smtp_password)
server.sendmail(fromaddr, toaddrs, msg)
The code above works perfectly with pages that you don't need to be logged on to view so I only need help passing login credentials to OCN from python. Here is a bit of code that I found and tried but it didn't work. I don't have much experience at all with urllib and frankly most networking practices within python so I'm stuck on where to go from here.

Code:

Code:
#!/usr/bin/env python

import urllib2
import sys
import re
import base64
from urlparse import urlparse

theurl = 'http://www.overclock.net/users/subscriptions/index/view/activity'
# if you want to run this example you'll need to supply
# a protected page with your username and password

username = 'myOCNusername'
password = 'myOCNpassword'

req = urllib2.Request(theurl)
try:
    handle = urllib2.urlopen(req)
except IOError, e:
    # here we *want* to fail
    pass
else:
    # If we don't fail then the page isn't protected
    print "This page isn't protected by authentication."
    sys.exit(1)

if not hasattr(e, 'code') or e.code != 401:
    # we got an error - but not a 401 error
    print "This page isn't protected by authentication."
    print 'But we failed for another reason.'
    sys.exit(1)

authline = e.headers['www-authenticate']
# this gets the www-authenticate line from the headers
# which has the authentication scheme and realm in it

authobj = re.compile(
    r'''(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=['"]([^'"]+)['"]''',
    re.IGNORECASE)
# this regular expression is used to extract scheme and realm
matchobj = authobj.match(authline)

if not matchobj:
    # if the authline isn't matched by the regular expression
    # then something is wrong
    print 'The authentication header is badly formed.'
    print authline
    sys.exit(1)

scheme = matchobj.group(1)
realm = matchobj.group(2)
# here we've extracted the scheme
# and the realm from the header
if scheme.lower() != 'basic':
    print 'This example only works with BASIC authentication.'
    sys.exit(1)

base64string = base64.encodestring(
                '%s:%s' % (username, password))[:-1]
authheader =  "Basic %s" % base64string
req.add_header("Authorization", authheader)
try:
    handle = urllib2.urlopen(req)
except IOError, e:
    # here we shouldn't fail if the username/password is right
    print "It looks like the username or password is wrong."
    sys.exit(1)
thepage = handle.read()
Thanks for any and all advice!
thumb.gif
 

·
Premium Member
Joined
·
6,520 Posts
Discussion Starter · #3 ·
Quote:
Originally Posted by ar3f View Post

Did you look through overclock.net page source and page info?
Yes I tried but I'm still unsure as to how to pass credentials. This is the important part of the page source but it didn't really help me get any closer to a solution.

Code:

Code:
<!-- Search Area -->

                                                Remember Me

                                                                        [URL=https://www.overclock.net/users/lost_password]Forgot Password?[/URL]
 

·
Premium Member
Joined
·
10,351 Posts
I'm no expert on web-related subjects, but when in doubt, use Wireshark to see exactly what's being passed back and forth. And OCN doesn't use HTTPS for secure login anyway, so you'll be able to see everything in plaintext.
 
  • Rep+
Reactions: DizZz

·
Premium Member
Joined
·
8,040 Posts
Quote:
Originally Posted by The Hundred Gunner View Post

I'm no expert on web-related subjects, but when in doubt, use Wireshark to see exactly what's being passed back and forth. And OCN doesn't use HTTPS for secure login anyway, so you'll be able to see everything in plaintext.
Not specifically aimed at you Gunner, but I've replied to you because I thought you might be interested in the reply
smile.gif


Wireshark, while undoubtedly an excellent tool and I understand your logic in suggesting it, I think in this instance it would would be a little overkill for this scenario. However there are similar tools for tracking HTTP/S requests based around setting up a HTTP proxy.

The tool I've used the most is Fiddler, which is free and works exceptionally well. Sadly it's Windows only, but there are cross platforms Java equivalents which also do the job (I've tried WebScarab; which works well enough too).

Honestly though (and this is aimed specifically at the OP), I doubt you'll need a proxy to track HTTP requests as you can tell the from the URL and hyperlink that it's a standard GET request with no query string; so the only way Huddler / OCN could track you would be via IP + user agent (dangerous) or by cookies (which is what every other forum uses). So all you need to do is view the cookies in your Firefox (or whatever browser you use) and paste them verbatim into your Python code (as you probably already know, Firefox allows you to search through cookies site by site within it's standard preference settings window - so you wouldn't even need to install any plugins to check. Chrome also shows cookies in the developer tools).

If that doesn't work, then include a Firefox / Chrome user agent string in your Python script.

The only drawback to this is that you're hard coding your cookies rather than manually logging in. But honest, the time you'd spend writing a log in routine vs how infrequently your session cookies would expire at the server side (and also how easy it is to grab updated cookie strings), it wouldn't be worth your effort in writing extra boilerplate code to log in.

One last thing; if you didn't know this already, OCN has the option to fire you e-mails when subscribed threads have unread content. This might not fit your use case; but it's worth baring in mind if you weren't already aware.
 

·
Premium Member
Joined
·
10,351 Posts
Quote:
Originally Posted by Plan9 View Post

Wireshark, while undoubtedly an excellent tool and I understand your logic in suggesting it, I think in this instance it would would be a little overkill for this scenario. However there are similar tools for tracking HTTP/S requests based around setting up a HTTP proxy.
I told you web isn't my forte... I forgot about proxies. There's also OWASP ZAP, but I guess that's a bit more geared toward pentesting compared to WebScarab.
 

·
Premium Member
Joined
·
8,040 Posts
Quote:
Originally Posted by The Hundred Gunner View Post

I told you web isn't my forte... I forgot about proxies. There's also OWASP ZAP, but I guess that's a bit more geared toward pentesting compared to WebScarab.
Yeah, ZAP is something different again. FYI, WebScarab is an OWASP project as well.
smile.gif
 

·
Registered
Joined
·
501 Posts
There has to be some code since web browsers (FFox, for ex) can automatically log in to overclock.net.
I'd try to find the procedure in whatever open source for and implement it in Python.

Here's a way from: https://stackoverflow.com/questions/7513569/auto-login-in-python-using-mechanize

from urllib import urlencode

from urllib2 import Request, urlopen

req = Request('www.site.com',urlencode({'user':'userhere', 'password':'passwordhere'}))

open = urlopen(req)

Replace 'user' and 'password' with whatever overclock uses
 

·
Premium Member
Joined
·
8,040 Posts
Quote:
Originally Posted by ar3f View Post

There has to be some code since web browsers (FFox, for ex) can automatically log in to overclock.net.
I'd try to find the procedure in whatever open source for and implement it in Python.

Here's a way from: https://stackoverflow.com/questions/7513569/auto-login-in-python-using-mechanize

from urllib import urlencode

from urllib2 import Request, urlopen

req = Request('www.site.com',urlencode({'user':'userhere', 'password':'passwordhere'}))

open = urlopen(req)

Replace 'user' and 'password' with whatever overclock uses
I'd already answered how browsers auto login, via cookies. Browsers wouldn't enter the username and password on each page request.

Your example code would not only fail, but you're also broadcasting confidential login details in clear text, which is grossly insecure.

Edit, in fact even in the link you gave, the top answer stated you need session cookies.
 

·
Premium Member
Joined
·
10,773 Posts
Quote:
Originally Posted by Plan9 View Post

I'd already answered how browsers auto login, via cookies. Browsers wouldn't enter the username and password on each page request.

Your example code would not only fail, but you're also broadcasting confidential login details in clear text, which is grossly insecure.

Edit, in fact even in the link you gave, the top answer stated you need session cookies.
Meh, when you submit the form here you send everything in clear text anyway so there's not much difference in that solution... (obviously subsequent requests don't do any of that once you're authenticated and have the cookies, like you say). Well its not really a solution that code because it fails to really show how to pass along the session cookies once authenticated...

Doesn't python have a component that can act as a web client? Its not too difficult to do this in .NET, I mean you give it a cookie jar and send a POST request and off you go!
 

·
Premium Member
Joined
·
8,040 Posts
Quote:
Originally Posted by tompsonn View Post

Meh, when you submit the form here you send everything in clear text anyway so there's not much difference in that solution... (obviously subsequent requests don't do any of that once you're authenticated and have the cookies, like you say).
That's the thing though, with ar3f's solution the OP would be needlessly sending login credentials with each page request. When really you only need to log in once ever few months / years and the rest of the time it's just a session cookie that's sent (which, arguably, could still be stolen and used to log in by someone else, but at least it's not leaking a password, which people tend to reuse)
Quote:
Originally Posted by tompsonn View Post

Doesn't python have a component that can act as a web client? Its not too difficult to do this in .NET, I mean you give it a cookie jar and send a POST request and off you go!
GET request surely? If you're just pulling the subscriptions then you're not posting any data.
 

·
Premium Member
Joined
·
10,773 Posts
Quote:
Originally Posted by Plan9 View Post

That's the thing though, with ar3f's solution the OP would be needlessly sending login credentials with each page request. When really you only need to log in once ever few months / years and the rest of the time it's just a session cookie that's sent (which, arguably, could still be stolen and used to log in by someone else, but at least it's not leaking a password, which people tend to reuse)
Ya that's true, but my way of thinking is I can't be assed copying cookies out of my browser
tongue.gif
But its not with each page request. Its just once to get the session cookie and auth token, just like you do logging in for the first time via the official page.

Though I would like to think Huddler's cookies are HTTP only to help stop cookie stealing via client side attacks.

With Huddler and other systems the session cookie actually has an expiry. They pass around an auth token that really keeps you "logged in". So you would need to still authenticate via POST first (so that you can a new session) otherwise the system SHOULD freak out that someone is trying to use an already existing session (or the session has expired already).
Quote:
Originally Posted by Plan9 View Post

GET request surely? If you're just pulling the subscriptions then you're not posting any data.
Sorry should have been more specific. POST was for logging in (if too lazy to copy cookies out of your browser). After that, yeah send a GET with your cookie jar.
 
1 - 13 of 13 Posts
Top