Retrieving Minimal Information

A place to share links to other websites and interesting articles

Retrieving Minimal Information

Postby euler » Thu Sep 27, 2012 1:42 pm

A new script has been made available to members to retrieve minimal information relating to the problems and account details. It is necessary to be logged in (an active session), but once this is done you can use the following three calls:

http://projecteuler.net/minimal=problems
This will return a new line for each active problem in the database containing the following information (delimited by ##).
id, description/title, date_published (timestamp), date_last_updated (timestamp), solved_by (number of members), solved_flag (0=no / 1=yes), answer (if solved_flag=1 otherwise empty)

http://projecteuler.net/minimal=problems;csv
This provides the same information as above except it will force the browser to open/save as a comma separated file (delimited by commas).

http://projecteuler.net/minimal=X (where X is the problem id)
This will return the raw problem content data in HTML.

http://projecteuler.net/minimal=profile
This will return details for the current account in a minimal format (delimited by ##).
username, alias, country, language, problems_solved (total number), level, solved_string
[The solved string is a "binary string" equal in length to the highest active problem and contains either 0 or 1 at a position corresponding to the problem id]

http://projecteuler.net/minimal=new
Returns a list of the release dates of any new problems (one per line if more than one and delimited by ##).
id, release_date (timestamp)
[If no new problems are currently scheduled then it will display the message, "No new problems currently scheduled"]
Image
impudens simia et macrologus profundus fabulae
User avatar
euler
Administrator
 
Posts: 2042
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England

Re: Retrieving Minimal Information

Postby sivakd » Sat Sep 29, 2012 2:46 am

euler, thanks for these. If I may suggest, why not generate csv or tsv files instead of html output with ## separators? csv/tsv files can be easily opened with Excel for analysis.
Image
puzzle is a euphemism for lack of clarity
sivakd
 
Posts: 217
Joined: Fri Jul 17, 2009 8:37 am
Location: California, USA

Re: Retrieving Minimal Information

Postby euler » Sat Sep 29, 2012 5:34 pm

I've added support to retrieve the minimal information for problems as a csv file. See the main post for how to access this feature.

I used ## as a delimiter because not all languages have native support for comma separated files and as the description/title in the problems can contain commas it is not as simple as just using a string to array function with a comma delimiter. Also none of the minimal options specified text/html output, but because I was not specifying a header the output was being best interpreted as text/html by default. I've now changed the other minimal outputs to use text/plain.
Image
impudens simia et macrologus profundus fabulae
User avatar
euler
Administrator
 
Posts: 2042
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England

Re: Retrieving Minimal Information

Postby sivakd » Sat Sep 29, 2012 10:12 pm

euler, thanks for the csv support. I think there is no easy way to provide a pure text (stripping out the html tags and entities) header for csv. Also, in addition to the solved flag, providing the rank if available would be good.
Image
puzzle is a euphemism for lack of clarity
sivakd
 
Posts: 217
Joined: Fri Jul 17, 2009 8:37 am
Location: California, USA

Re: Retrieving Minimal Information

Postby euler » Sat Sep 29, 2012 10:41 pm

What is the rank?
Image
impudens simia et macrologus profundus fabulae
User avatar
euler
Administrator
 
Posts: 2042
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England

Re: Retrieving Minimal Information

Postby sivakd » Sun Sep 30, 2012 3:55 am

Sorry, I meant the position in the fastest solvers table.
Image
puzzle is a euphemism for lack of clarity
sivakd
 
Posts: 217
Joined: Fri Jul 17, 2009 8:37 am
Location: California, USA

Re: Retrieving Minimal Information

Postby SoboLAN » Sun Sep 30, 2012 6:25 am

These pages are very useful for developers, thank you euler.

There is however a problem: introducing the CSV page is not a good idea because there are some problem titles that contain commas (","). CSV parsers can be easily confused by this. Using "##" is much better.

EDIT: I saw now that you already found this problem, you can ignore this post.
Image
User avatar
SoboLAN
 
Posts: 34
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Postby SoboLAN » Sun Sep 30, 2012 7:01 pm

Except for the problems 1, 143, 395 and 396, all problems have the same value in the "last_update" field: 18th September 2012. Why is this ? (I suspect why, I want to see if I'm right :P )
Image
User avatar
SoboLAN
 
Posts: 34
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Postby Homez » Mon Oct 08, 2012 9:22 pm

Hello to all folks!

I'm a Project Euler common solver (level 7) but I started thinking sometime ago of writing some framework to do keep some things more simple. This could be a JAVA SE application maintained in Eclipse. I could, for example, see the same list of prolems with their short descriptions, my solution status and the number of people having solved this problem. This now can all be taken from Project Euler site thanks yo yours scripts, euler! From me I could add there my actual solution times, so I could be eager to improve them. The solutions could run inside this app, so solution times could be maintained automatically in some config file or database like SQLite. It could use also some sorting and/or filtering so I could easily find problems solved by many people but which I haven't solved already by myself (I do it all now manually because I beleive that the problem I solve next is always among my unsolved problems that were solved by a larger quantity of members).

I could write all this just for my personal usage, but I could also share it to people that would like that (of course, without a single solution or common algorithm).

But I wonder: how can I login programmatically in order to get rid of these scripts? I know that I have to deal with session cookies in order to stay logged but I can't even login from my Java starting app :(

I saw this nice sample: http://ubuntuincident.wordpress.com/2011/11/08/download-login-protected-pages-with-python-using-mechanize-and-splinter-part-3/ Here some library browsers are used with Python. But I'm not ready to switch to Python, now I'm a Java guy :)
Image
Homez
 
Posts: 3
Joined: Mon Oct 08, 2012 9:02 pm

Re: Retrieving Minimal Information

Postby Homez » Mon Oct 08, 2012 10:10 pm

I can procceed! I've just added "login=true" to POST params - and now I see the list of problems, page 1. Now I can try to ged rid of session.
Image
Homez
 
Posts: 3
Joined: Mon Oct 08, 2012 9:02 pm

Re: Retrieving Minimal Information

Postby Homez » Tue Oct 09, 2012 1:08 pm

And I can't :( I try to set cookie that I got after posting a login form, for the next query. But I always get About page, being logged off. Can anyone help?
Image
Homez
 
Posts: 3
Joined: Mon Oct 08, 2012 9:02 pm

Re: Retrieving Minimal Information

Postby Sjums » Wed Nov 14, 2012 9:51 pm

Hey euler, I'm all new (problem 8) but already hooked! Thanks for a nice riddle :-)

A simple request for the minimal data is; would it be possible to strip the HTML and serve it as plain text? I'm specifically thinking of the problem detail page:-)

Regards, sjums
Sjums
 
Posts: 3
Joined: Wed Nov 14, 2012 9:30 pm

Re: Retrieving Minimal Information

Postby euler » Wed Nov 14, 2012 10:09 pm

The problem content depends on HTML as it may contain images, superscript/subscript, or data contained in tables. If this formatting is removed then it would be impossible to interpret the problem correctly; for example, consider x = 32 without superscript: x = 32.
Image
impudens simia et macrologus profundus fabulae
User avatar
euler
Administrator
 
Posts: 2042
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England

Re: Retrieving Minimal Information

Postby SoboLAN » Sat Dec 08, 2012 4:50 pm

Hi,

It makes sense to need an active session when going to http://projecteuler.net/minimal=problems or http://projecteuler.net/minimal=profile. The output seen in these cases depends on the user currently logged in.

But can you please remove this requirement for http://projecteuler.net/minimal=X (raw HTML of a problem) ? The output is always independent of the session, so it makes no sense in requiring it.

Programmatically logging in or automatically including cookies in the software written to retrieve the HTML is actually more difficult than I thought. It's a problem we shouldn't have...
Image
User avatar
SoboLAN
 
Posts: 34
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Postby lucaswerkmeister » Thu Feb 21, 2013 9:41 pm

SoboLAN wrote:can you please remove this requirement for http://projecteuler.net/minimal=X (raw HTML of a problem) ? The output is always independent of the session, so it makes no sense in requiring it.


Yes, that would be super cool... I just deployed a broken update to my plugin because of that.
Image
lucaswerkmeister
 
Posts: 3
Joined: Tue Jan 15, 2013 7:11 pm

Next

Return to Resources

Who is online

Users browsing this forum: No registered users and 1 guest

cron