Icon_search
Stephanie Sharp Nick Reistad Buck Sharp Kim Landrum Charlie Maffitt Juli Tredwell Kevin Warrick
Charlie

Charlie's Blog

RSS

Eliminating comment spam in Rails with Akismet and Defensio

Posted on 04/30/2010
55 Comments

A lot of the sites Plexus builds have a built-in blog component which allows for user comments. As with any front-end form, these can attract a lot of spam, primarily of the "Cheap Rolex Watches Louis Vuitton Handbags" variety where a spam-bot loads up the comment with a ton of spam keywords and links. Over the years at Plexus we have employed several tactics for combatting this: Originally we tried requiring the site administrator to approve all comments before they got shown on the site. This is an effective means of keeping the spam off the site, but it puts a lot of extra responsibility onto the site admin, and the queue of comments waiting to be approved can really pile up if the admin doesn't frequently log in to approve or reject them. This method also takes away most of the "real-time" aspect of commenting, which isn't good.

The next method we employed to some success was the use of CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart). You're probably already familiar with CAPTCHA -- that's where you have to look at an image of jumbled letters and numbers and then enter them correctly into a text field in order to prove that you're a human and not a spam-bot. Requiring the user to correctly enter the CAPTCHA code does cut down on spam, but these days it doesn't eliminate it entirely. It seems the spammers are constantly improving their technology to keep up with anti-spam methods, and some of them appear to have cracked CAPTCHA. Thus, even though we had employed CAPTCHA on the Plexus blog, we were still receiving a lot of spam through the comments. That wasn't good, because we were requiring legitimate commenters to jump through an extra hoop, and we weren't even catching 100% of the spam.

Our next step was to look into some of the spam-fighting services available to web developers and to make use of those services on our sites. The first such service I found out about was Akismet (http://akismet.com/), and later I discovered Defensio (http://defensio.com/). They both work in pretty much the same way: When a comment is submitted on your site, you use an API key to make a call to the service, which analyzes the content of the comment along with such information as the submitter's IP address and request headers, then returns its verdict -- either spam or not spam. Based on the verdict from the service, you can then put the comment into an approval queue (not shown on the site) or show it immediately (if the service decides the comment is legit). Defensio even offers filtering for profanity and blocks links to certain categories of websites.

The good news for us is that there are good Ruby wrappers already written for both Defensio and Akismet, which makes integrating them into a Rails app a breeze. I used the Defender Ruby gem to interface with Defensio and the Akismetor plugin to interface with Akismet. API keys for both services are available for free: Akismet offers a free personal key and paid keys for corporate use. Defensio's keys are free for both personal and corporate use while Defensio is in beta -- they will soon announce a pricing structure for corporate API keys.

In order to test the two services head to head, I've put Akismet on the Plexus site and Defensio on another new site we recently rolled out. I'm going to compare the amount of spam each site receives through its comments and see which does a better job of recognizing spam. So far in my tests I have found both to be quite accurate. In the event that the service is mistaken and erroneously marks a legitimate comment as spam, or vice versa, you can correct it. Both services claim to "learn" as they receive more and more of this correctional input from users, so over time the accuracy of the service is supposed to improve.

The thing I like about each of these services is that by integrating them into your own code, you're not necessarily handing the decision over to them. Your code can get the service's verdict and then decide for itself what to do with the comment. For instance, Defensio doesn't simply return a thumbs up or down, it also returns a decimal value between 0 and 1 representing the "spaminess" of a comment, with 0 being a completely legitimate comment and 1 being obvious spam. So you could use that value to handle comments differently depending on their "spaminess" value: For example, allowing everything with a spaminess value up to .4, quarantining everything with a value between .4 and .7 for admin review and automatically discarding everything with a spaminess value above .7. Defensio actually recommends that you not use the spaminess value for anything other than sorting, but you can see the potential for filtering there. I suppose you could even submit the comment to both services, get each's opinion, and then make the call from there, but that would probably be overkill and add unnecessary time to the request.

So if you find yourself constantly weeding through comment spam, look into a spam-fighting service like Defensio or Akismet to keep your meatcloud-sourced content spam and malware-free.

UPDATE (5/25/2010):In the month or so since I installed Akismet on this site, I found it had about a 95% success rate. Not bad, but not perfect. Some spam did still slip though. Today I switched to Defensio, and I will report back how that turns out.

Tagged:  spam, comments, antispam, akismet, defensio, captcha

Back to top

Dynamic Select Menus in Rails using Observe_Field

Posted on 03/16/2010
8 Comments

I frequently find myself building forms which contain dynamic select menus. This means that a user selects a choice from one menu, and that choice populates options into a second menu. An example of this might be a scenario where the user selects their state from a drop-down menu, then selects their city from a second menu. We only want that second menu to contain cities appropriate for the user's state, but when the user first loads the page we don't know what state to pull cities for, so we have to wait for that input to populate the second field.

I had been using Javascript to help solve this problem, as described in this Railscast, and that worked just fine until I had to change the JS framework on a certain site from Prototype to jQuery as part of a sitewide upgrade. That caused syntax problems with the JS that was driving my dynamic menus.

Now, I could have gone through the Javascript and just changed all my Prototype syntax to jQuery, but I started wondering if there was a better way to accomplish the same trick without using Javascript (and therefore making the whole deal JS framework-agnostic).

It turns out that making use of the observe_field helper in Rails makes this process pretty simple, and requires no explicit JS code.

I'll outline the process here...

First, let's look at the basic elements of the form in my view:

The first field in the form is a select menu which gathers all the states in the database and offers them as choices. The next bit of code is our observe_field helper, which observes the state selector and fires the "update_city_menu" action when it is changed. The last bit is an empty div where we are going to put our second select menu once we have values for it.

The next thing we'll to do is set up the "update_city_menu" action in the controller.

All it does is find which state in the database the user has selected, and assigns an instance variable to it.

Finally, we set up a view for the output (in this case, called update_city_menu.html.erb)

In this case I coded it so that the second drop-down menu only appears if it gets populated with actual choices. If you want the second menu to always be visible and to display a message if there are no options, you could write your view like this:

I'm a lot happier with this solution than I was with the JS solution because it takes a lot less code and and works the same whether you're using Prototype or jQuery.

Tagged:  dynamic select menus, ruby on rails, ruby, rails, observe field, javascript, jquery, prototype

Back to top

Web Site vs. Web Application... What's the Difference?

Posted on 12/18/2009
8 Comments

When people ask me what I do for a living and I tell them I develop web applications, they often give me a blank look. When that happens, I backtrack, generalize, and tell them I build web sites and they go, "Oh, okay!"

So what is the difference between a web site and a web application?

We all know what a web site is, so let's focus on web applications. Web applications, or Rich Internet Applications (RIA), are presented as either a web site or as part of a web site, but not all web sites are web applications. A web application is a web site that DOES something other than display content to the masses. It's intended for user interactions and transactions, performing actual business functions, and not simply displaying information to a user.

Ebay is a web application. So are Paypal, Twitter, Facebook, YouTube, Flickr, eHarmony, eTrade, GMail, and Wikipedia. Your bank, assuming it offers online banking, has a web application component on its site. If you can think of a site where you create an account, log in, and conduct some actual business, it's probably a web application.

By contrast, a web site usually refers to the front-end interface through which the public interacts with a business online. Web sites are typically informational in nature with a limited amount of advanced functionality. Simple websites consist primarily of static content where the data displayed is the same for every visitor and content changes are infrequent. More advanced websites may have features such as eCommerce, content management and dynamic content. Simple websites are basically clearing houses for information. The UGA Athletics Site is such a site, as are CNN.com, Nytimes.com and Nike.com.

Many large sites are a blend of the two. For example, Fidelity.com is a good example of a website with a web application built into it. A member of the general public can go to Fidelity.com and do research on current market trends, learn about the Fidelity company and the financial products they offer, learn tips on saving for retirement, etc. This part of the site is just a regular old web site.

However, if I have Fidelity accounts, I can sign on to a protected part of the Fidelity site where I can buy or sell stocks & mutual funds, track potential investments, get advice from Fidelity's advisors regarding my accounts, all through a nicely designed user interface. This part of Fidelity's site is a web application -- it facilitates business actions over the web, and I can conduct business on the site just like I could if I were sitting in a brick and mortar office.

Here at Plexus, we build web sites with an integrated content management application. So in that sense, every site we build has a web application built into it. Most of our clients don't need anything more than that, but a small percentage of our clients need their site to perform specific interactions with users. Our most recent web application is Listingtank.com, a site where renters looking for apartments can get cash rewards for a finding and signing lease through the site. Its business function is to match renters with property managers and to generate revenue for everyone involved. Is it a web site? Yes. But to simply call it a web site simply doesn't do it justice. Another client, Bulldawg Illustrated, uses their site to manage subscriptions to both their print and online publications. Another client recently built an application on their site to track data on local public health clinics.

Given this knowledge, the trick for any business with an online presence is to decide whether its web site could actually be used as a web application, and what business actions that application could conduct. Savvy business owners, such as Listingtank, have figured out how to adapt the technology we've made available to them to create a site that not only performs business functions, but is actually THE means by which they conduct business.

There are countless firms out there that can build you a web site. But can they build you a web application, customized to meet your business' specific needs? Can they take your traditional business functions and translate them to an interface allowing your customers to do business with you from their home or laptop? Plexus can. We love building web applications because each one is different, and each business has a specific set of needs. What business function do you want your web site to take on? The sky's the limit.

Tagged:  web site, web application, Business

Back to top

How the Worst Camera on the Market Became the Most Popular Camera on the Web

Posted on 10/22/2009
8 Comments

This August, Mark Milian of the LA Times reported that Apple's iPhone had surpassed the Canon Rebel XTi as the most popular camera on Flickr. Let's distill that down to its essence -- the most popular camera on the world's most popular photo-sharing website isn't a camera, but a phone with a camera in it.

If you have an iPhone, or have seen pictures taken by one, you know that the camera which comes bundled inside Apple's ubiquitous pocket-computer isn't the best in the world. In fact, it's barely passable as a camera. The pictures it takes remind me of the first commercially available digital cameras that came out in the late 1990's - terrible resolution, washed out colors, poor focus and virtually no depth of field. Forget about action shots or night photos. Those cameras were basically toys which cashed in on the novelty of a film-less camera. Then, as the technology improved and became more affordable, digital cameras became more acceptable in terms of photo quality, and now there are DSLRs available for under $500 which can take pictures of a professional quality.

The Canon Rebel XTi, while still a bit above that $500 mark, is one of the current breed of DSLRs that shoot incredible pictures. A search of XTi-taken photos on Flickr quickly reveals a array of stunning images. Canon and Nikon also both make point and shoot cameras which are remarkably capable of taking high-quality shots in the right hands.

The iPhone, on the other hand, takes blurry, washed-out photos that no self-respecting photographer would want to add to their portfolio. Since quality of images doesn't explain it, what IS the reason for the iPhone's popularity on Flickr?

One word: Connectivity.

The iPhone lets you go from point and shoot to sharing on the web in a matter of seconds, thanks to its internet connectivity via WIFI or 3G. I can take a photo with my iPhone, post it to Twitter, Flickr, Facebook, etc , and be receiving comments in less than a minute. The picture itself becomes more relevant because it is being seen almost in real time. On the web, freshness counts. Which means that if we want to quickly snap a photo to share with friends online, the iPhone is the instantaneous choice, despite its poor quality. It's already right there in your pocket anyway. While a Nikon D90 or Canon Rebel might take a picture that's 100 times better, it requires downloading the shot to a computer, then uploading it to the web, which means you have to be near or return to your computer before you can share anything. So lately we've all been sacrificing picture quality for speed, convenience, and above all connectivity.

I'm no different. In the last 15 years I went from being a photo snob that only shot on slide film with an SLR to someone who grudgingly accepted digital cameras because of their web-friendliness, to someone who doesn't even use the digital camera I have anymore because basically every picture I take is on the iPhone so that I can get it up onto Twitter quickly & easily.

So what are the implications?

First, it indicates to me that high-end camera makers like Canon and Nikon need to be developing technology to integrate internet connectivity into their cameras. If I could shoot pics with a Nikon D90, then upload those pics to the web from the camera via WIFI, I would be a lot more inclined to use said expensive camera because then I'd have the best of both worlds -- quality and web connectivity. Real-time photo sharing within a micro-blogging framework with (gasp) high-quality images. At present, there have been a few half-hearted efforts at the point-and-shoot level, but none have full web connectivity.

Secondly, Apple will doubtlessly improve the iPhone's camera quality over the next few years. Just like the awful digital cameras of the 90's, the technology will improve and become so cheap that before long we will carry around 10+ megapixel cameras bundled inside our phones. A competitor's phone will out-camera the iPhone, and Apple will respond. The soon-to-be-released Motorola Droid purports to have a much better camera than the iPhone. But in the meantime, the web is being flooded with terrible pictures of interesting things, and I think that's a shame. We're stuck in another low point in the evolution of cameras, (like the 90's digital camera gap) where the one everyone uses is awful, despite the fact that high-end cameras are available and affordable.

Think back to your family's pictures from the 70's, when everyone was using 110mm film. Remember the rounded corners on the prints, the flat tonality of the colors, the yellow-orange tone to everything? Those shape our memory of that time. When we look back nostalgically at the Ought's (as I intend to refer to this glorious time), will the only tangible supplements to your memory be blurry, washed out photos, scalable to no larger than 640x480?

I'm writing this post as much as a chide to myself to use my good camera and not my pocket novelty camera, but that connectivity issue is such a big one that I think it's worthy of mention. So make an effort to take a picture with a good camera this week, put it on the web, and then share it with me, because if I have to look at too many more blurry, washed-out iPhone images I just might wipe the inch of dust off my Nikon SLR and start shooting on slide film again. I need some clarity.

Tagged:  photography, cameras, photo sharing, web, flickr, wifi, connectivity, iphone, canon, nikon

Back to top

Carbon Motors Getting More Press

Posted on 03/30/2009
23 Comments

One of Plexus' clients, Carbon Motors Corp., has been getting a lot of attention from the media lately. I was the lead developer on their web site, and thus feel I have a lot invested in their success, so it makes me feel proud to see so much press coming their way.

Most recently, our friends at Carbon were featured on the front page of the Atlanta Business Chronicle. A week prior to that, they were featured in a CNN.com feature story, and will be the subject of an upcoming CNN television feature.

With so much media coverage, the website (http://carbonmotors.com) has been getting a lot of traffic, so much that we recently moved it to its own server in order to handle the bandwidth.

In addition, Carbon have actively been working with our SEO specialist Kelly Jones to get more traffic coming to the site, and search engine traffic has been steadily rising over the past few months. While it's difficult to say whether this is the result of Carbon's tireless efforts to spread the gospel of the E7, or Kelly's SEO wizardry, the short version of the story is that every day more and more people visit the Carbon Motors site. Is this the result of more press? Is the increase in press driven by the increased visibility of the website? In reality, it's a complex relationship with many factors, but what matters is that Carbon's public presence is on the upswing.

A whole new set of people will be introduced to Carbon Motors and the E7 in 2009. When they are, one of the first places they'll visit is carbonmotors.com. Thanks to Plexus and to proactive content management by Carbon, those visitors have an informative and content-rich destination they can use to inform themselves about a potential paradigm shift in how police cars are built. We at Plexus are proud to be at the forefront of that effort.

Tagged:  carbon motors, police car, media, SEO, public relations, e7, cnn, atlanta business chronicle

Back to top