Build Slide Show Movies from Flickr with Video.Conductr

Posted by Chad

In the past I mentioned an application that I was building using Rails and the Flickr API. After building it, it turned out to be way too resource intensive for me to host with Dreamhost. Since then, I rewrote the application in PHP and it is a bit faster. It’s still a bit of a CPU hog , and Dreamhost probably isn’t too happy about that, but I wanted to put it up anyway.

The application is video.conductr. It searches Flickr for images with your search terms in their tags, text descriptions, etc. and pulls the images together into a Flash video. The videos can be viewed on site, embedded on other pages, or even downloaded as MP4. No account registration is required, however, it will help you keep track of your movies, registered users’ movies have priority in the queue, and you will get a friendly email notice when your movies have been produced. To keep down on resource use, I am currently producing about 3-6 movies per hour maximum. If the site gets heavy usage, I will put it on a better machine and this limitation will be gone.

“Urban” was a recommended tag today when I visited the Flickr homepage. Here is the video and embedded below.

TODO:
  • Fix movie introduction text
  • Fix view counter
  • Add commenting system

Dreamhost's "Fat Finger" Billing Cost Users $1.3 Million

Posted by Chad

Background

Earlier this week Dreamhost, the company hosting this blog, accidentally over billed their users for $7.5 million. Over a couple days time, they looked into the issue and have mostly corrected the problem and now consider it solved.

Of course, this has resulted in thoroughly pissing off their users and the delivery of the explanation was so poor TechCrunch called them out on it. I wasn’t too upset, my credit card had expired and I didn’t have the auto-bill feature activated so I didn’t get hit by it. If I had, it would have been awful. Dreamhost bills my bank account via check card where I usually operate around a $1000 buffer/emergency balance. However, at this time I was shifting some funds around and arranging my various balances because I am gearing up to move to Dallas. Long story short, I only had $50 in my account at the time and a $120 hosting bill would have caused an overdraft.

The cost of my overdraft fees are $35 but, there is more. I also had 4 small transactions pending, all of which would have triggered fees when they attempted to settle. So, I dodged a $175 bullet fired by a trigger happy “fat finger.” Luckily, I never trust any service enough to allow them to auto-bill me and my card was expired anyway but, for many users this event had a real cost.

Financial Analysis

This is a very rudimentary and somewhat ridiculous analysis of how much “fat fingers” can cost your users. I’ll explain my logic but you can get the Excel file and make up your own assumptions.

First, I identified three ways this event has real costs:
  1. Finance Charges
  2. Overdraft Charges
  3. Affect on Credit Rating

Finance Charges

This cost is associated with having an elevated credit card balance. Most credit cards calculate finance charges based on an average daily balance. This means an elevated balance for more than one day, even if refunded in the future, will cost you money.

Given:
  • $7.5 million dollars worth of balance elevation
Assumptions:
  • 18% APR / 365 = 0.049% Daily rate; ignored compounding
  • 3 days elevated (days until refund)
  • 85% of balances were credit cards, otherwise no finance charge

Cost to Users: $9,432

The only given in this analysis is that $7.5 million was billed to users. I assumed the average APR is 18% and that it would take 3 days on average to refund the charges. Also, I figured about 85% of transactions were on credit cards, it could be much more/less. Charges to non-credit card accounts are assumed to bear no cost.

Overdraft Charges

This cost is associated with those users who experienced overdraft, over-the-limit, or some type of fee due to the unexpected charge.

Assumptions:
  • $125 average bill
  • 60,000 bills
  • 10% of bills defaulted
  • $35 fee

Cost to Users: $210,000

Based on Dreamhost’s prices and the user’s comments I estimated $125 as an average billing price. With $7.5 million in total billing, that gives us 60,000 bills that were erroneously collected. The default rate is rather subjective, I wouldn’t be surprised if it was actually 1% while it may be upwards of 20%. And, $35 is on the high side of fees but they are usually in that general range.

Affect on Credit Rating

This cost is associated with the long-term affects of a blemished credit rating. Specifically, the risk that your interest rates will be increased.

Assumptions:
  • 6,000 users affected
  • $9,000 average debt held
  • 2% rate increase
  • 1 year affected

Cost to Users: $1,080,000

If affected, this cost can be huge. However, it is difficult to estimate and the formula I used is extremely sensitive to change. I’ve read reports that puts average U.S. consumer debt at any where from $8,000 to $12,000; so, I went with $9,000. I know the 2% rate increase and 6,000 users affected (all users who received overdraft charges) are probably over-estimates. But, I also only assumed they would be affected for one year. In reality, those affected may be paying up for several years, all the while interest is compounding. For that matter, I also ignored daily compounding throughout the year.

Result

Total Cost to Users: $1,299,432

with

Total Cost per User: $21.66

With the continuous growth of large subscription based sites, I wonder how often this happens? Dollar amount over billed? Real cost to users? And lastly, how long until these issues turn into class action lawsuits?

Silly little time consumers

Posted by Chad

I’ve wasted a pretty good amount of my coding time lately because of some oversights made while programming. To be expected when learning a new language and framework I suppose (Ruby on Rails). No, no, no—that’s the easy excuse. Most of these errors were not related to Ruby or Rails, just me being careless.

The RMagick debacle

This one cost me the better part of an entire day. You’re going to laugh. As I said in the previous post, this RMagick thing had me going mad and running in circles. I ended up trying to contact the DreamHost support to see if they could help me out. I know the run around, I used to give it when I was running my web hosting business back in the ‘03, they will tell me they can not help me debug my program and that RMagick is working for everyone else, so I must be doing something wrong. I knew that, I was hoping for help as to what I was doing wrong.

Fast-forward to the following day. I had to go to work and effectively clear my mind from everything Ruby, Rails, RMagick and DreamHost. The next day, I had class but when I came home (I just then received a reply from DH, telling me it’s my problem) I decided to give it a quick glance. This is where I acknowledge that sometimes when I get stuck on something, I like to walk away, take a break, walk the dog, eat a sandwich, etc. I can rethink the problem and come back with a new approach. So, during my 2 day hiatus I was able to clear my mind from the problem.

When I returned to the code, my first thought was; “concentrate on what was different in my local environment as opposed to the DreamHost environment and try to identify the problem”.

The Problem

Windows + require 'rmagick' => success!!!

Debian + require 'rmagick' => failure!!!

The moment I thought about it that way, it clicked. I needed to use require 'RMagick'! And of course, it worked. Windows is case insensitive while Linux is case sensitive. Go ahead, laugh, I told you that you would.

Answering my own questions

This is a story about paths. This way, that way, any way… but not there? Most of the story unfolded in a one way conversation with myself on Dmytro Shteflyuk’s awesome post about using ffmpeg/mencoder via Ruby. What that side of the story doesn’t tell is that I spent about an hour wrestling this issue before I decided to ask Dmytro for help.

For the most part, I was using his instructions to encode some video through my app. By putting the code in a model, I was trying to load/write the images/videos from my app/models directory.

This is another trend that I have noticed that helps me troubleshoot problems. The simple act of writing an explanation of the problem helps clarity it and sometimes often times I think of the solution mid-explanation and the post never even gets submitted.

In this case, I did submit the post. Afterwards, I proofread it. While proofreading, I got the idea that maybe it was a path issue. Sure enough, as mentioned in the reply to myself, I edited the code to include the images/videos absolute paths and problem solved.

Total waste of time? (What I learned)

Be more thorough in the initial draft of my code. Developing on Windows can make me sloppy in some areas. Troubleshooting should not continue into frustration. It’s simply not productive and may compound the problem. Take a deep breath, clear your mind, come back with a fresh approach. Put the problem in writing, it will give you another way to look at it and will open up a little creative right brainedness that may be needed.

RMagick + Dreamhost = The Death of Me

Posted by Chad

I have been working on a little application that I am ready to put into production mode. It only uses two gems (plus their dependencies); RMagick and ObjectiveFlickr. This is probably the simplest app since “Hello World” but, of course, I ran into a snag almost immediately.

The issue involves RMagick, if you didn’t already guess by the title of this entry. Any time I issue require 'rmagick' in the app, it breaks. The production.log shows MissingSourceFile (no such file to load -- rmagick) as the problem. Meaning, that RMagick can not be found. I know it is installed because the bash command gem list shows that it’s there.

There are instructions to install my own versions of ImageMagick and RubyGems. I have already installed my own RubyGems at /home/<user>/.gems so that I can install/update certain gems (eg. ObjectiveFlickr). While I do not need the most recent version of RMagick, the instructions are straight-forward and if it works I’ll take it.

The compiling of ImageMagick went without problems (I think). However when it came time for the gem install rmagick I get an error almost immediately saying the required libraries are not installed (eg. ImageMagick). My suspicion is that I need to tell it where to look for my installation of ImageMagick but I have no idea how do that.

Google is not giving me any useful results, although, I have found others posting about all types of Dreamhost + RMagick problems.

As you could guess with the other gem being an interface to the Flickr API, I am processing images and therefore need RMagick. The app is rendered useless until I can get this issue resolved. It’s a pretty nifty little app, I couldn’t find any thing else out there doing the same thing with the Flickr API, I hope to share it with you soon, until then the functionality is my little secret!