Code

I have been doing a lot of work for the past 8 months or so creating content/applications/etc for the iPad using HTML5, and I have ran into a lot of frustrating/annoying things along the way. I started not knowing Javascript, so there was that. Beyond that it was really hard to find iOS-specific information (or even webkit) when I started, so I began to compile a list of useful links when I found them. Here is the list so far, so that anyone who is starting out won’t have to make brutal mistakes or look for hours like I had to This is mostly a webkit-centric list, but Mozilla Developer Center is also a great place to check out. If anyone else knows good links feel free to suggest them and I’ll add them.


General
———————————————————————-

Dive into HTML5
http://diveintohtml5.org/
Free web book with a great overview of all HTML5 features.

Appe HTML5 Showcase
http://www.apple.com/html5/
Great showcase of HTML capabilities from apple. Source code is a bit proprietary and hard to understand.

W3 Schools
http://www.w3schools.com/html5/
De-facto resource for all things web, their documentation of HTML5 standards. Usually useful although sometimes incorrect.

Surfin’ Safari
http://www.webkit.org/blog/

The blog for the Webkit project (HTML5 backend of regular and mobile versions of Chrome, Safari, etc.) Great examples of CSS3 features.

Mir.aculo.us
http://mir.aculo.us/
HTML5 expert Thomas Fuchs’s blog. Amazing amount of info. Updated frequently, and offers a number of HTML5 classes.

Fonts
———————————————————————-
Font Squirrel @font-face Generator
http://www.fontsquirrel.com/fontface/generator

Awesome @font-face generator that gives you the font files, CSS and examples for (almost) any font you throw at it.

Mobile/iPad
———————————————————————-

Making an iPad HTML App Really Fast
http://mir.aculo.us/2010/06/04/making-an-ipad-html5-app-making-it-really-fast/

Post on Mir.aculo.us that walks through the basics of creating an HTML5 iPad app and optimizing it. Great overview.

Touching and Gesturing on the iPhone
http://www.sitepen.com/blog/2008/07/10/touching-and-gesturing-on-the-iphone/

Great overview of touch events.

Offline HTML5
http://diveintohtml5.org/offline.html

Overview of creating offline apps, which is the basis for HTML5 web apps that work without internet connection.

Creating an HTML5 Web App
http://sixrevisions.com/web-development/html5-iphone-app/

Shows how to create a web-app and a lot of grew things to consider. Aimed at iPhone but translates pretty much exactly to iPad.

iPad + HTML5 + Javascript Memory Management
http://www.vargatron.com/2010/08/ipad-html5-js-memory-management/

Article I wrote as an overview of how memory management works on the iPad and how to deal with it.

Frameworks
———————————————————————-
Raphael.js
http://raphaeljs.com/

Framework for dealing with SVG based graphics.

Processing.js
http://processingjs.org/

HTML5 Canvas based port of the extremely popular Processing language. Great for creating interactive canvas based applications.

jQuery
http://jquery.com/

The leading Javascript general library out there. Good for basic effects in websites, but not so much for web-based apps.

jQTouch
http://jqtouch.com/

JQuery-based mobile framework, great for create simple UI based apps, still a bit under development. Not so great for custom apps

Sencha
http://www.sencha.com/products/touch/

Probably the best all-inclusive solution for creating cross-platform mobile apps. Based on the Ext.js framework, Sencha uses an extremely different system for creating apps. Similar to jQtouch (although more complete/powerful) in that it is easy to make simple UI based applications, but not great for custom apps.

MooTools (My Object Oriented Tools)
http://mootools.net/

My favorite (and in my opinion the best) general framework for creating custom HTML5 apps. Supports full object-oriented programming (classes, constructors, extending and implementing classes, etc). Not as large of a user base as jQuery, but a lot more serious/experienced users. Great comparison between MooTools and jQuery (by the creatore of MooTools) at http://jqueryvsmootools.com/

OK this is another post that is more for my own future sanity than anything else. This has been a huge pain in the ass for me many times and I want to document the steps I took to get this working.

First, this is a good general overview

http://cd34.com/blog/programming/python/mysql-python-and-snow-leopard/

So I downloaded/installed MySQL, then I had to add it to my PATH var to get the MySQLdb install to work.

//Open With textmate
mate ~/.profile

export PATH=”/usr/local/bin:/usr/local/sbin:/usr/local/mysql/bin:$PATH”

then make sure you update your profile or close and open a new terminal window
source ~/.profile

that will let the MySQLdb install work. Then when you run Python and try to use MySQL db you’ll get errors. What you need to do is the following:

change your PATH to export PATH=/Applications/MAMP/Library/bin:$PATH

which links to MAMP mysql, then run this command

sudo ln -s /Applications/MAMP/tmp/mysql/mysql.sock /tmp/mysql.sock

Then you should be good to go. Might be able to skip the first step but I don’t know if the MySQL that comes with mamp is 64bit and you need 64 bit for snow leopards install of Python..

Why is python such a pain in the ass?

The first major challenge of my thesis project has been acquiring historical data, parsing it into a usable format (database), and then going through it and trying to make some sense of it. I’ll try to go over some of the steps I took to get where I’m at (as usual both for anyone trying to do the same thing and for my own sanity the next time I need to do this).

First of all, thanks to the extremely hard work of a few super dedicated individuals, historical data for every baseball game  is available online at the site http://www.retrosheet.com

These files have an insane amount of data for almost every game played for at least the last 50 years (there is more but before then the complete info is a bit hit or miss). Unfortunately, due to the way that they are recording the data (I’m not really sure how, but I assume it involves some sort of spreadsheet), the files are not really in an easily interpreted format. Luckily, again due to the work of a few dedicated individuals, the open source community has a solution for this as well. The tool “Chadwick” (named after the man who is historically credited with inventing the first widely accepted form of baseball scorekeeping) is a command-line unix tool which can take these files and parse them to Comma Separated Value (CSV) files. This does entail learning a proprietary command language for the compiling (it looks sorta like regular expressions but I don’t think its related), but the end result is worth it.

From this point (once you have your CSV files) you can create a database in the SQL of your choice (MySQL, sqLite, etc) and import the database.

Fortunately there are a few people out there who have already done a great bulk of this work, and have written some scripts to take care of a lot of this process for you. Unfortunately, they are using Python with the MySQLDB extension, which under snow leopard is near impossible to compile at this time.

The main site I have been working from to get this data to work can be found here:

http://blog.wellsoliver.com/2009/06/retrosheet/

This guy is the perfect combination for my thesis: a data geek and a huge baseball fan. He has made several amazing tools (a few of which I look forward to getting into in the future) that allow people to get baseball data in a bit more friendly manner, and shares his experiences obtaining the data.

I spent the better half of a weekend attempting to get his code to work on my computer, due in no way to any error on his fault but solely based on the fact that snow leopard refuses to compile and install a workable version of MySQLDB to python.

I finally gave up, and decided that I would go to plan B and run the script on my server (Dreamhost). This too proved to hit a few roadblocks (the version of Python installed on my server wasn’t new enough, had to install new Python then compile and install new MySQLDB a couple times )

Then I had to figure out how to compile the Chadwick source code under Unix, which seemed very unclear until I actually opened the readme file and realized all I had to do was type ./configure in the Chadwick directory and it would compile no sweat.

I finally got the code to run and (seemingly) work! Then I realized I had another error…my server was killing my requests and I was only getting part of the data. After emailing back and forth with Dreamhost for an hour or two, I finally realized what the issue was. The python code that I was working with ran in 20 seperate threads (which would have been fine had I been able to run it on my own computer) however my Dreamhost account had a limit on concurrent threads that were allowed to run and i was greatly exceeding this. After changing the amount to 5, I finally was able to execute the script and build my database. It took about 15 minutes (!) to run the script through, and I was left with a complete, 4+GB database of baseball information.

This has been a huge headache (to say the damn least!) but now that I have gone through this process once it should be a lot easier from her on out if I need to grab new data (hopefully…).

I’m now working on understanding and parsing this enormous amount of data (the one table of events alone has over 8 million rows…) and make some sense of it. I’m close to having a workable XML format and I’ll post more about this in the future.

One more thing….

I found this link: http://www.wantlinux.net/2009/04/retrosheet-baseball-mysql-database-download/

which is basically the entire code already compiled into a database and downloadable On one hand this totally bums me out, but on the other hand it was a great experience having terminal/unix try to beat me down and coming out victorious! I am considering doing what this person has done and hosting the 2009 database on my site once it has become available, I’ll make a post if I decide to do so.

I recently ran into an issue that was a bit of a pain in the ass and I wanted to put it up here both to help anyone else out who runs into it and for my own reference.

I have been doing a lot of data mining/visualization recently which requires a large number of calls to several data apis. I am using PHP because I am familiar with it, I like it, and I can’t stand the structure of Python. One advantage of php is that you can execute it in the browser, which I normally take advantage of when I am working on API requests or data scraping. As my scripts have gotten more and more complex lately, I’ve quickly run into Internal Sever Error 500 messages which can only be fixed by limiting the amount of calls to make at a time. At first I thought this was just a limit of my server/php, but then I remembered how I recently was able to import a 4GB database into MySQL through the terminal with no problems after PhpMyAdmin was unable to handle anything near this. So I decided to try to run my script through the command line using SSH with dreamhost.

I attempted to execute the script, and then I got a bizarre error: “Fatal error: Call to undefined function: mysqli_connect()” This error never came up in the browser, yet it was coming up in terminal. After some quick searching I found out this error was attributed to not having PHP 5 installed on my server. This is super strange because PHP 5 is on my server according to dreamhost. I then did a version check on php using the -v parameter. Sure enough, version 4.4.9.

So what was the solution? I found a couple resources after playing around with phrases to search on google, and figured out that while php 5 WAS installed on my server, the command line space is different than the web space, so I had to explicitly point to PHP 5 in order to use it.

Running the following command led me to the path of PHP5 “whereis php” which returns the following path “/usr/local/php5/bin/php”.

So super short finish to a long explanation, use “/usr/local/php5/bin/php” instead of “php” to execute php scripts using version 5 on Dreamhost.

I have been using my own file templates for openFrameworks for a while now, but since I found out about this in China and was messing around with it on a train between Shanghai and Hangzhou I totally forgot about it. Recently people in class have been asking about creating classes in OF, so I’m taking this as an opportunity to fix some things I have been meaning to correct in my file templates and put them up for people to share/use.

These templates are file templates, if you’re looking for Project Templates Memo Akten has a great post on how to install his XCode project templates for both desktop and iphone. You can check that out here:

Memo’s iPhone Templates

To install these templates, you first need to create the following folder:
(Your Hard Disk)>>Library>>Application Support>>Developer>>Shared>>XCode>>File Templates

The next step is to download the zip I’m including below and copy the files into your File templates folder (specified above).

Now if you go into XCode, open a project, right click on the SRC folder and select “Add>New File…” you’ll be presented with a dialog. You should see the category “User Templates” on the left, and under that “openFrameworks”. Click on “openFrameworks” and select “basicClass”. Hit “next”, name your class, and click “finish”.

You now should have a new OF/C++ class ready to go with the basic setup(), update() and draw() functions, header includes, definitions, etc.

Enjoy!

Files for Download:
openFrameworks File Templates

Here are a few things I have done recently, I think all for my setPixel class. My thesis work is in other posts and my data vis class is still getting rolling work-wise (expect a first project from there by next week), so most of the work I have done this semester is from this class.

I won’t go crazy into descriptions, but basically the images here use algorithms I wrote to accomplish “square-packing” and “circle-packing” by finding the largest square or circle that would fit in each image, drawing it, then finding the next largest, so on and so forth till the image is recreated using squares or circles. Really hard for me to figure out but the effect is awesome and was well worth the effort I think!

These videos are made by averaging the pixels of two movies (The Darjeeling Limited) and reproducing the colors of the frames from every second of the movie. Really interesting how a movie as colorful as the Darjeeling Limited reduces to a lot of browns (most likely due to complimentary colors)

stringTheory from Steve Varga on Vimeo.

stringTheory is an experimental program I made that is not as much data vis as it is abstract data. Using the concept of String Theory (theoretical physics) to visualize my email, the program first shows everyone that has sent me email in the last year as particles within a group of atoms. Each node when rolled over shows the numer of emails this address has sent me as well as playing a tone based on this number. After clicking one of these nodes, that person’s messages are abstracted further as particles (protons & neutrons maybe? that doesn’t quite work yet but its loosely based…) which contain particles of the subject of the email. Rolling over one of the messages displays the message’s subject and plays a tone for each letter, with the timing based on the overall length of the subject and the tone of each letter based on both the message length and a value I’m giving to each letter.

The project uses PHP to collect my email to a MySQL database, then openFrameworks is used to grab a php file which generates XML from the database. This XML data is then used to drive the visuals.

The audio is generated by pushing OSC commands from openFrameworks to SuperCollider. The tones are somewhat harmonious but I’m just getting started with SuperCollider and I’d love to see what I can do with this data once I get a better understanding of the full power of super collider.

Overall I can already think of a lot more interesting things I could do with this but I’m super happy with the results so far….

Respond//React is a text messaging interface for openFrameworks. Software is written for S60 python on nokia N80 which first connects to a computer via bluetooth serial port and then receives messages and forwards them through the port. The messages are queued in the application and displayed one at a time every 15 seconds. Ideally this project would be presented as a mobile urban projection, allowing users to interact in a large public space.

Respond//React from Steve Varga on Vimeo.