This is a line to test my blog integration with empire avenue, in my other lil experiment…
So I have been scraping data online for sometime. While scraperwiki have an API that allow third party app to get data in json/xml form. I think I can make it easier, because scraperwiki query involve doing a sql query on the sqlite datastore. Thus I take the opportunity to learn new python web framework.
Finally to run the app, just run python server.py (or any python file with the bottle run function). You have an webapp.
Because bottle is a micro framework, there is no manage.py script like django, no ORM, I uses sqlalchemy here. There is no session support too. But interestingly I don’t feel that I missed anything. In fact, it is pretty pleasant to use. Though session will definitely bite me if I ever have to implement login, but solution is on the documentation.
Overall, it is a fun framework to use, even though this is a small project. The documentation is pretty good. I might use it for future project.
Note: You can find the docs in the python doc page http://docs.python.org/library/sqlite3.html#sqlite3.Connection.create_function
This is more of a experience. Not too long ago, I have scrape from the parliament website on profiles of Member’s of Parliament, you can find the result here.
The thing is, as I use the data from the sqlite database, I download from the site, I realized that, the Title is part of the name of the MP’s. So one would get “XXX , Y.B Tuan”. Y.B Tuan is the title.
That would make query like ‘select Parti from swdata where Nama=name’ hard. Because this is precisely what I am looking at, for another project.
On the other hand, sqlite3 module, apart comes with python standard library since 2.6. Actually have a function called, Connection.create_function.
So I wrote a little function called get_name, and the example show how it works.
s = sqlite3.connect(‘dbname’)
# attach the python function
# and use it
result = s.execute(‘select get_name(Nama) from swdata’)
Just define a python function, make sure it return datatype that is compatible with sqlite, attach it with create_function. Now you can use it in your sqlite query in python
Hope this is useful for someone. CHEERS
A little plug, this is something we try to work on in this little group call Sinar Project, and this is still in an early stage