buzfortuna

I don't know if this is the best way to phrase this question but I'll try anyway.

I have experience with JavaScript, but not much with XMLHttpRequest- so I don't know where to start.

What do I need to look up in order to learn to log in and grab data from outside sites If I wanted to do this PHP, I could do it rather easily with cURL libraries, but I have no idea how to do this without PHP.

I may be thinking of this the wrong way though. Similar methods would be how you specify information in Gmail gadgets and it checks your email. What do I need to research to do something like this

Thanks!


Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

Jonathan Abbott

There's lots of example on here if you do a search. Here or Here for example





Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

buzfortuna

I looked at both of those. The "downloading and searching a web page" thread helped with part of my question. Essentially what I need to do is log into a website with a user name and password stored in Gadget.Settings, use regex to get a certain number in the page, and then update that number if it ever changes. I know how to get a page and use regex to get something I need with JavaScript.

My problem lies in sending the user information to the site to allow me to log in. I know how to send cookies and session information with PHP's cURL libraries, but don't know where to begin when developing for a sidebar.

I guess to specify my question I could say "How do I have my gadget log in to a site given user information (through settings), and return the source of the unique-to-each-user logged in page " From there I could return the page source and search through it.

Thanks again!




Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

Andy E

Using XMLHttpRequest to get past login screens and scrape data can be a little tricky (at least, as much as I know about it).

Firstly, you need to take a look at the HTML of the login screen. Get the form action URL, method (usually POST for passwords), and the names of the input fields. Then use that information in an XMLHttpRequest like so:

Code Snippet
obj = new XMLHttpRequest();
obj.open("POST", "http://link.to/loginSubmit.aspx", true);
obj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
obj.onreadystatechange = function() {
if (obj.readyState == 4) {
if (obj.status == 200) {
html = obj.responseText;
xhtml = obj.responseXML;
doSomethingWith(html, xhtml);
}
}
}
obj.send("uname=yourusername&pw=yourpassword");


uname= and pw= are the names of the input fields from the login form and http://link.to/loginSubmit.aspx should be replaced with the form action URL. Unfortunately, this will only get you the page that submitting your login information redirects you to. However, a lot of websites have a hidden field to choose the landing page after you login - you can easily work this into your code. Also, if the page you land on is XHTML valid, you can use responseXML to parse the data much easier. If not, you have to use responseText. A useful way to parse the HTML is to add it to an iframe, and then parse the HTML using the DOM.

I've done this myself with a couple of sites, so if you need any further help, let me know which site you're logging into and I'll see what I can do for you.

Andy






Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

buzfortuna

Fortunately, the page I need to get the info from is the page that you're redirected to once you've been logged in. Unfortunately it's far from valid xHTML, but I can deal with it.

This all makes sense to me, I've replaces uname and pw with the correct form input names, and I replaced the form with the correct login form- but as I mentioned, I really have no experience with XMLHttpRequest, so I (to be honest) don't have much of an idea what's going on here. I understand it's sending the POST information to the login form that I specify, but what does it return, and how do I access it (I assume thats the doSomethingWith())...

I really appreciate the help you guys are giving me, hopefully this will be functioning once I get passed this hurdle.




Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

Andy E

XMLHttpRequest(); fetches a particular page on the internet.

The readyState property indicates how complete the process is, so checking if readyState == 4 is actually checking to see if the page is fully downloaded.
Checking the status is important, too, if the status == 200, that means the document is ok. If it's 404 for example, that means the page is not found.

Basically, the request returns the HTML of the remote document (through the responseText property). So if you set html = obj.responseText; then the entire content of the html page is stored in the variable html as a string. So you can then run your regexp on it.

Andy




Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

buzfortuna

Marked as solved and helpful because my initial questions were answered perfectly. Hopefully they can help someone else, too. I have to take you up on getting a hold of you about your gadgets, too. If you can, get a hold of me. My contact information is in my profile. Thanks!




Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

buzfortuna

Also, slightly interesting is, it worked for a little while, but now it doesn't. I put in some debugging code and this is what I get:

Code Snippet

function retrievePage() {
obj = new XMLHttpRequest();
obj.open("POST", "theform.cfm", true);
obj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
obj.onreadystatechange = function() {
if (obj.readyState == 0) { document.write("0"); }
if (obj.readyState == 1) { document.write("1"); }
if (obj.readyState == 2) { document.write("2"); }
if (obj.readyState == 3) { document.write("3"); }
if (obj.readyState == 4) {
if (obj.status == 200) {
document.write("4");
html = obj.responseText;
xhtml = obj.responseXML;
}
}
}
obj.send("email=myemail&password=mypassword");
}



And it just displays "1" forever. Was there something I changed (I changed some things in the mean time) that made it stop functioning




Re: Sidebar Gadget Development Log in with JavaScript / AJAX...

buzfortuna

Actually, most of that problem is fixed. Here's what I changed to see where the new problem lies:
Code Snippet

if (obj.readyState == 4) {
document.write("Object Status: " + obj.status);
if (obj.status == 200) {
document.write("Page valid");



The page just says "Object Status: 0" forever. Ugghhhhghh.