From now on, on Premium Accounts only, Pagelime will back up not just the content history (as we have done so far) but also full page code each time a user publishes a page. We will securely store a copy of the page source code, as it was right before we upload the new version.

(more…)

As most of you know by now, we were affected by the Amazon EC2 outage, which resulted in approximately a day of on/off downtime for Pagelime. We’ve communicated openly about it to anyone who reached out, and we sent out a mass email with our personal cell numbers and personal emails. We wanted to make sure to stay open, and available when you needed to reach out to us. I’m going to take time with this post to explain the details behind the outage.

Here’s what happened, how it affected us, and what we’ve done since to mitigate this issue:

• The North Virginia region of the Amazon Elastic Cloud infrastructure had a major set of issues with their storage: the Elastic Block Store (the EBS). The EBS is meant to be a highly redundant form of storage with very low rates of failure, where any single disk failure should not affect availability of the actual data. Turns out this isn’t exactly the case: the entire block store seems to have become unavailable within a number of regions.
• A number of web companies were affected, including Foursquare, Reddit, Quora, and HootSuite to name a few. A number of web apps like ourselves assumed the issue would be resolved promptly.
• Amazon took about a day to repair the issue, at which point service was restored, and things began to operate normally. This puts our current uptime at 99.4% for the year. We need this to be better for both our users and our peace of mind.

Here’s why we use Amazon AWS and not a custom brewed hosting solution:

• Amazon AWS is fast. It gives us really good response times, and the storage performs very well for the price. Which allows us to keep our costs down for our users, while providing the best service.
• Amazon AWS is highly available. It allows us to host servers in Virginia, North California, Ireland, and Asia Pacific at the same time, so that a Pagelime server can always be available to close to where you are.  The same goes for their simple storage service: S3.
• Amazon AWS is highly scalable. We can provision new computing and storage resources very quickly. It puts scaling into our hands.

An ideal setup with AWS should not have failed even in the outstanding scenario we had over the past two days. Here’s why ours failed:

• All of our data is stored in multiple availability zones (around the world) by both our databases and our data files stored in S3. This worked like a charm, our database immediately failed over to an available instance, and our data was un-affected. This is good.
• The same goes for servers hosted in multiple availability zones. Only the ones in the US-East zone were affected. This is good.
• However, for speed of use, Pagelime caches all of the data files, such as content, images, and documents, on Elastic Block Store volumes… the very volumes that failed completely. This cache allows us to quickly publish content without a lot of round-tripping between the database and the servers. This crippled us for a day, and we’re fixing it.
• Pagelime also runs the publish engine from ONE single destination. The reason is that we want the publish engine to always originate from one IP address, so that firewalls and hosts can white-list us. This publish engine happened to be in the affected zone. We’re fixing this as well.

Soon after the outage happened, we initiated plan B, and began to migrate all of our cache/engine to a different availability zone. This was great as an emergency response, but we want to be resilient to these failures in the future. Here’s what we’re doing to prevent this from happening again:

• We are purging the publish cache. From now on, the data will be published directly from the data store. This may result in longer load times when you press the publish button, or when you publish an image gallery, but it should reduce the potential of future failures. We unfortunately have to cut this performance optimization for the sake of reliability.
• We are adding code to the Pagelime application that will actually fail-over in the software itself to different storage models should one appear to be failing.
• We are creating a backup publish engine in a different part of the world. And for those folks who have bypassed firewalls, we will send this IP out as well, to be added to their web host’s firewalls.

We’ve learned a lot from this. We were really proud of our cloud infrastructure, and the speed / reliability we were getting for the price. After this incident we’re a bit sobered, and we realize that we need to put even more effort into it.

We’re grateful for the outpouring of support we’ve received from you via email. Thanks for standing by us – we’ll make sure to pay it back in kind.

I received an email this morning from a user asking what all the different permission levels mean. I figured I would take a moment and do a brief explanation: (Note: He’s a business account user and has a few extra permissions)

Hey everyone! Right now we’re working on 2 bugs that have recently popped up. We are looking for community feedback to help work through these issues.

The first bug is occurs where sites that had been working suddenly stop working (or never work in the first place). We’ve narrowed it down to a specific issue. Somewhere in the last 3 weeks many server hosts have upgraded their FTP servers to 220 ProFTPD 1.3.2 Server. To confirm you are experiencing this specific bug, click “Browse” in the site settings and then the log should look like this

It appears to be primarily with IX Webhosting, but ProFTPD is a common component of many hosts. If you could please leave a comment with the domain that is failing, and where you host your sites (Host Gator, Media Temple, etc). We’re actively working on rolling out a fix for this ASAP.

The second is a bug with our iPhone app on original iPhone 3G phones. It appears like the app loads quickly, then crashes immediately. We’re looking for people with iPhone 3G’s (NOT 3GS) to help us beta test a fix before our next release.

Thanks! We’re always trying to get everything fixed as fast as possible. Happy New Year everyone! Go get some more clients!

So our iPhone app has been sitting in Apple’s review box for a while now… and while you all patiently wait, we’ve put together a ballin’ Pagelime iPhone wallpaper. Why, you ask? Because we’re just awesome that way.

Also, don’t ask us what the whole leather and stitch theme has to do with anything, we just thought it was kinda cool.

Download the Pagelime iPhone 4 Wallpaper

For months folks have been asking when we were going to get our stuff together and start offering Pagelime in multiple languages. We’ll the wait’s over, and as always we tried to raise the bar on internationalization for a web-app: We have developed an interface for you to translate Pagelime into any language or language-culture combination right from your account management dashboard!

(more…)

Dear PageLime users, we’ve recently looked at our cloud hosting energy bill, and after sifting through our accounting, determined that our main expense is air conditioning. This is unacceptable, and we decided to proactively move our entire hosting operation to a custom server facility to be built here. This move will literally cut our air conditioning costs in HALF, and allow us to offer better long-term service to our customers.

Obviously this move will take some time, so expect a few months worth of down time while we put together the expedition and explore possible approaches to what we determined to be the optimal location for the new facility. We’ve also been advised that this area does not have high speed internet access, but only AOL dial-up. While we don’t expect this to be an issue, we are building a backup cloud infrastructure, as outlined in the image below:

We’re commited to offering the best service on the market, and as such we will continue to find new ways to innovate and improve our web-app with great ideas like these. If anyone’s interested in joining our mountaineering expedition, please leave your contact details in the comments, and we’ll get in touch. Survival rate is very high… but iPhone 3G coverage is poor.

Happy April 1st.

A few months ago we rolled out repeating regions, and ever since, we’ve had two types of bubbles: edit bubbles and repeat bubbles. This was fine for the time, as these two were separate features. Over time, however, most people are starting to use these in conjunction, so we decided to consolidate the bubbles and use this as an opportunity to redesign them.

As you can see from the picture attached to the post, the new bubbles will range from the super simple edit bubble, to the full blown combination repeat/edit bubble. The repeat controls do not show until you mouse over the little blue arrow that suggests there are more options on this bubble than just editing.

The new bubbles will be black with a grey outline and a glass effect. This style will provide the best visibility on both black and white pages while remaining color agnostic (we love the lime green edit bubbles, but sometimes it just interferes with a page design).

On a final note, this style will allow us to add more features to the bubbles as we build them. For example, we’re planning drop-in modules like image galleries, calendars etc for more advanced users. The bubble design will be flexible to accommodate sites ranging from the simplest ones with just a few editable areas, to more interactive ones with additional modules.

As always, let us know what your thoughts are in the comments or in the forum.

Pssst… take a quick look at one of the things that’s been keeping us busy over the past few months: PageLime Touch.

Yes! It’s the rumored and long awaited PageLime iPhone app. On the preview site linked below, you can watch a video showcasing some of the user experience and functionality. Just like our web-app, this version of PageLime will be fully brandable for our Professional users.

Check it out, get excited, and let us know what you think:

​

Click here for a sneak peak of PageLime Touch