Tag Archives: tweetmarker

Watermark transition plan

I sent an email to Watermark customers over the weekend, letting them know that the service as it currently exists will be going away on May 15th. As I wrote about last year, Twitter has improved their search enough that a part of what Watermark is good at is no longer as necessary as it once was. However, I still see interest in Tweet Marker, from developers and users, so I wanted to keep the web-based timeline and sync from Watermark and make it available to all Tweet Marker subscribers.

You can learn more about Tweet Marker here. I’ve had to significantly scale out the backend servers this year, including adding a second load balancer, so I’d love your support. The new timeline feature will roll out later in May.

While I’m happy to keep offering a part of Watermark (now back in Tweet Marker), I have less good news for Tweet Library. I’ve found it very difficult to justify the time to finish the new version. It’s now looking likely that the current version will be the last.

Waiting out trademarks

Not long after I launched Tweetmarks in 2011, I realized that there was a trademark for that name, and an existing .com domain. I started worrying about the conflict so much that I couldn’t get any real work done. I talked to friends about it, tried to get other perspectives, and then finally renamed it to Tweet Marker. Whew, I had made a decision and moved on, free from ever worrying about it again.

I had to fix the tweetmarks.net redirect recently and checked around on some of the old stuff. That domain name I had been so worried about, which I literally lost sleep over? It’s gone.

I’m not going to tell you that trademarks don’t matter. Nothing I write on this blog should ever be considered legal advice. But it’s another reminder that there’s enough real stuff to focus on without wasting time on imaginary problems.

Updated Tweet Marker extension

I updated the Tweet Marker Safari extension to version 1.2 today. This version fixes the extension to accommodate recent design changes on twitter.com. Existing Tweet Marker paid subscribers can grab the new extension here.

Here’s a screenshot of the “Set Marker” link that the extension adds to twitter.com. Then when you launch a compatible iOS or Mac Twitter app, it will scroll the timeline to that position.

If you’d like to support Tweet Marker or use the Safari extension, you can subscribe for $1/month.

Winding down my Twitter apps

I’m the guest on this week’s Mac Power Users podcast. In addition to workflow and apps I use, the discussion went off the rails a little into the Twitter app ecosystem, especially the fact that I no longer post to Twitter yet still have apps like Tweet Library, Watermark, and the Tweet Marker API that depend on Twitter. For the last 2 years this has been an odd decision on my part; I want to do the right thing for my customers, but I’m increasingly frustrated with life as a third-party Twitter developer.

Last week, Twitter announced that they’ve expanded their search index to include the full history of tweets going back to 2006. I was thrilled by this upgrade to the Twitter service. That the search was so limited for so long was the primary reason I built Tweet Library and Watermark to begin with. Unfortunately, this functionality is only for the official Twitter apps. It will not be made available to third-party developers.

It’s time for me to wind down development on my Twitter-related apps. I’ll continue to sell Tweet Library through the end of 2014, then remove it from the App Store. Watermark will also shut down at that time. Because all the tweets stored in Watermark are public tweets (by design it never supported DMs or protected accounts), I will attempt to make the entire Watermark database archive of millions of tweets available publicly. Existing customers can also sync tweets and collections to Dropbox for personal archiving.

Published collections from Tweet Library or Watermark will be maintained indefinitely. No URLs will break, ever. Updating published collections will also continue to work for anyone who already owns Tweet Library.

I will also continue to host the Tweet Marker API, but starting in January I will be more strict about requiring developers to pay for the service. Many developers have been paying for API access for a year (thank you!), but others have missed or ignored my requests to move to a paid plan. It’s not fair to the Twitter developers who have been paying for Tweet Marker access if some continue get the API for free.

Many friends have told me over the years that I have too many products. But letting any one product go is not easy. There’s an implicit promise when shipping software that the developer should maintain and improve it for customers. Stopping development on these apps is the right decision and possibly long overdue, but it’s still difficult. What gives me hope is that it will let me focus on new projects currently in development, which I couldn’t be more excited about.

Killing or selling the product

Nick Bradbury on selling Glassboard to Second Gear:

“When finding a new owner was first discussed, I was very unsure of the idea. We wouldn’t be just handing over our software to someone else – we’d also be handing over our customers and their data. That was such a concern of mine that I suggested we simply kill the product.”

A few times this year I’ve thought about whether I should sell Tweet Marker, but it usually comes back to a similar concern as what Nick says above. I wouldn’t feel comfortable selling it to most companies that would actually want to acquire it. So I’ve made a change or two, and we’ll see how 2014 goes.

As for Glassboard, I’m excited to see what Justin Williams does with it. We use it for Core Intuition questions and feedback (invite code “coreint”), and I’m sure I’ll be using the 3.0 version around conferences next year.

Tweet Marker now paid-only

If you’ve read my blog posts about Tweet Marker over the years, including this one when introducing a paid plan, you might hear a little indecision on the right path forward. I’ve run it more like a community service and less like a business, and admittedly the window for turning it into a profitable endeavor has slipped away. But I still want to make the service work well and improve it.

After vacationing this week leading up to Thanksgiving, I finally realized I need to remove the support burden of giving out free access. Starting today, Tweet Marker will transition to a paid-only service. To make this easier, I’ve introduced a less expensive $25/month plan for smaller developers. (Previously the only choice was $75/month, which is now the top tier plan with unlimited active users.)

This solves the problem of automating access to the API. Instead of having to manually fulfill API keys so that developers could try out Tweet Marker, which usually meant weeks or months of delay until I could get to it, by signing up online, developers get immediate access to the API without having to wait on me.

In the short term, this change means that new apps must subscribe before using the API. In the long run, all existing apps should also transition to a paid plan. I haven’t set a hard deadline for existing apps yet, but will work with developers over the coming months to do so.

You can learn more at tweetmarker.net/developers/.

Update: To be clear, this post is only for developers. There’s a $1/month subscription for end users but it is optional.

Announcing the Tweet Marker developer plan

I’m launching a new paid developer plan for Tweet Marker today. It’s $75/month and includes a new admin dashboard with stats on active users, hits, and more. I’ve also expanded the API to support syncing which direct messages have been read.

Why charge developers now, after keeping the service free for 2 years? In part it’s because of something I learned from publishing the Core Intuition podcast with Daniel Jalkut. Because for the first few years of Core Intuition, Daniel and I had trouble getting episodes out very regularly; there was always something more important to work on. Adding sponsors pushed us to stick to a weekly schedule, and it’s worked out even better than I expected.

I hope the same thing will happen for Tweet Marker. Although I’ve put countless hours into maintaining Tweet Marker (and plenty of money on hosting), I couldn’t justify the effort to create new APIs because it wasn’t a revenue-generating product. Now I can dedicate more time to it, even with a modest level of support from developers.

Of course, I’m sensitive to the difficulty of transitioning from a free to paid product. That’s why I’m doing two things to make it easier for everyone.

First, I won’t be turning off any existing developers’ access to the service. The last thing I’d want is to break third-party Twitter apps currently in use. But I do strongly encourage commercial app developers to subscribe if they have the means to.

Second, I’ve created a referral program for app developers to let their customers know about the $1/month user subscription. This is a great way for developers to show their support even if they can’t subscribe to the developer plan. But even better, for developers who do subscribe, their account will be credited for each paid user they refer. This can effectively make the new developer plan free or significantly discounted.

This is a big change for Tweet Marker, but an important one to make Tweet Marker strong. I’m excited to keep working on it, so that Twitter apps can work even better together. Sign in here to learn more about it.

Twitter API v1.1

My apps Watermark and Tweet Marker were recently updated to version 1.1 of Twitter’s API. I’ve also finished the coding changes in Tweet Library to support 1.1, but it has not yet been submitted to Apple. I expect it to be another week while I wrap up a new feature included in the update.

I don’t like having my release schedule dictated by others, but that’s life as a Twitter developer. And why I no longer post there.

This matters today because Twitter is running an API blackout test. Tweet Library will not be able to load new tweets this afternoon for about an hour (3pm PST) while v1 is unavailable. I’ll post again here and from @riverfold on App.net when the new version of Tweet Library is shipped off to Apple and live in the App Store.

Replacements for Google Reader

With the success of Tweet Marker, several people suggested I should build a sync server for RSS. This was last year and earlier this year, before Google Reader officially shut down, but after it was clear that we needed something better. I jotted down some notes for a couple ideas but ultimately decided not to do it. I’ve already got my hands full with my current shipping products!

Luckily many great developers are now on this. Feed Wrangler from David Smith, hopes for a possible NetNewsWire Cloud, more interest in Fever, and other established web apps like NewsBlur and Feedly. As Marco Arment said, this could end up being a great thing for innovation in blogs and RSS again.

But just because I’m going to watch on the sidelines for the server sync part of RSS, doesn’t mean I’m going to completely skip building better RSS support into my own products. There’s a lot I’d like to do with client-side RSS support in Watermark.

Business vs. user experience

Some companies seem willing to do anything for a profit. The worst domain name registrars and their pages filled with up-sells. News blogs that spread articles across several sections to increase page views. We see examples all the time of blatant attempts to increase sales just a little at the expense of usability.

But the reverse can also be a mistake. For example, my own Tweet Marker. I wanted the setup user experience to be so effortless that the user merely needs to flip a switch to enable it in their favorite apps, or do nothing for the apps that choose to use Tweet Marker by default. There’s no formal registration, no prompt for an email address.

Now I find myself with 500,000 total users who have tried Tweet Marker, but no way to follow up with them to see if they are interested in upgrading to the $1/month subscriber plan. The service is, frankly, a financial failure. More like a charity experiment than a business.

I’ve been giving this a lot of thought, and introducing the subscriber plan was the latest part of a renewed effort with Tweet Marker. I’m determined to make it work, even if it’s too late to shift the balance between business needs and user experience to something that makes more sense.

Safari extension for Tweet Marker

Since introducing the Tweet Marker $1/month subscriber plan earlier this week, I’ve received a few questions about how the Safari extension works, and whether Watermark customers will also receive the new features. Yes, Watermark subscribers automatically have access to the Tweet Marker extension, which can be downloaded here.

I’ve prepared a screencast to show how the extension works. It’s about a minute long, and you can view it right here.

Thanks to everyone who has already subscribed to either Tweet Marker or Watermark.

Tweet Marker new subscriber plan

The original goal for Tweet Marker Plus was to help cover the hosting costs for Tweet Marker. It succeeded for a little while, but it also ended up evolving into a larger independent service: Watermark, with much higher hosting costs for archiving and search, and a bunch of new features like App.net support, Dropbox sync, saved collections, and more. I’m really excited about the future of Watermark.

I also hear from Tweet Marker users who don’t need Watermark. They still want to support Tweet Marker, though, to make sure it continues running and that it’s as fast as possible.

So today I’m introducing a separate, inexpensive subscription option for Tweet Marker. Just $1/month! You can subscribe from the new Tweet Marker home page. And as a bonus you’ll get the first official Safari extension for Tweet Marker, shown in this screenshot:

Tweet Marker extension

Stream Marker

Watermark now supports App.net’s Stream Marker. You can click a post in Watermark to set its marker, the current marker will be retrieved when refreshing posts from the server, and the “Scroll to marker” link scrolls to the marker or loads more posts trying to find it.

Here’s what a marked post looks like:

Watermark screenshot

I don’t know if any other clients support Stream Marker yet, but I expect many will in their next updates. I thought it was appropriate that I should be one of the first to support it.

I launched Tweet Marker in June of 2011. It wasn’t exactly a new idea — Echofon had a private sync before, and developers had long been asking Twitter for an official timeline position sync API — but Tweet Marker was the first to support sync between different apps. And it was the first to bring this idea to many more users, to make it a must-have feature that customers asked for.

In a way, I think of Stream Marker as a next-generation Tweet Marker, even though it’s not mine. The API and name are different, but it feels familiar, like an evolution from what we learned from Twitter apps. It’s a no-brainer to support it in Watermark.

What the Tweet Marker award means

It’s been over a year since I launched Tweet Marker, and I’ve been thinking a lot lately about what its second year will look like. Hosting costs have gone up significantly over the last year, though I’ve offset that by combining the hosting for Watermark and Tweet Marker together so that they share the same core servers. I’ve considered other options, too: run a Kickstarter-like fundraiser, charge developers, or ask for donations again.

Now that I’ve stopped posting to Twitter from my personal account, people also ask whether I’ll just shut Tweet Marker down. The answer is no. I’ll keep it running, even if it means funneling some revenue from Tweet Library and Watermark to pay for it. Even if it means having to put out fires and deal with other web hosting distractions.

Hosted web services have a different level of commitment than traditional apps. When I stopped selling Wii Transfer, existing customers could continue to use it for as long as they wanted. Not so with something like Tweet Marker, which becomes useless the minute I shutter the web server.

The Macworld Eddy statue sits on my desk as a constant reminder that real people like this thing and find it useful. There is a lot of uncertainty with Twitter, App.net, Tent.io, and the future of microblogging, but no matter what takes off and what sync looks like, I think Tweet Marker played an important role in the evolution of Twitter clients. I’ll always be proud of that. It would be a disservice to my customers and the Macworld award to ever consider turning off the API while people value it.

Announcing Watermark

I’m renaming Tweet Marker Plus. Its new name — to better reflect its gradual move away from Twitter and syncing — is Watermark.

As part of the relaunch it immediately gains a new feature: App.net posts. You can now add an App.net account and it will download any posts from your friends, making them available for search. Watermark is already storing tens of millions of tweets, and I’m excited to start adding App.net posts to that archive as well.

So what happens to the basic Tweet Marker API? For now, nothing. The sync API that over 22 Twitter apps support will still be called Tweet Marker and remain Twitter-focused. Think of Watermark as a separate app: a new kind of client and archive tool, independent of Tweet Marker.

Twitter lock-in

With every day since Twitter’s new rules were announced, my opinion grows stronger that this is the end of the platform as we’ve known it for the last few years. It hurts that so much of what was pioneered by the developer community has been co-opted and trademarked and turned against developers. Nothing will change tomorrow; we can expect new versions of Tweetbot, Twitterrific, and my own app Tweet Library. But in the long run it’s a dead-end.

Lex Friedman, writing for Macworld:

“Some developers are notably hesitant to speak on the record, lest they incur Twitter’s wrath; the fear seems to be that since Twitter is now exerting more control than ever over access to its API—which developers leverage to make their Twitter apps work—that irking Twitter too much might result in a developer’s API access getting revoked.”

I have less at stake than developers with popular mainstream apps, so in a way I feel it’s my duty to speak out when it’s warranted, where others can’t. I’m quoted in Matthew Panzarino’s article on the Twitter API change, and I was interviewed for a radio spot on San Francisco’s KQED.

I could live with some of the API changes. I could live with the display requirements, even the user caps. Maybe I could live in a developer-hostile environment, finding a market niche far below Twitter’s radar. Except for two things:

  • “Timeline integrity”: The display guidelines say you can’t mix other content in a timeline of tweets. You can’t easily have a multi-platform app that shows both App.net posts and Twitter tweets in the same view.

  • Trademarks: You are allowed to use “Tweet” in the name of your app, but only if Twitter is the only service you support. You can’t add Facebook or App.net or Heello support alongside Twitter.

These are anti-competitive and unworkable. So I’ve decided to spin off Tweet Marker Plus, my paid web-based app that builds on top of the Tweet Marker sync engine. I’ll relaunch it with a new name, slowly introducing new features that aren’t tied exclusively to Twitter. For a hint of where it’s going, you can follow me on App.net.

Tweet Marker SSL mistake

It usually takes a couple problems hitting at once to cause a major server outage. This happened last week when Tweet Marker’s SSL certificate expired. I have the SSL set to auto-renew, but it still requires manually installing the new certificate, and other problems happened along the way.

First mistake: I didn’t realize it was expiring. Those emails go to an account I don’t check very often, littered with spam. And the email to confirm the renewal went to yet another email address that no longer worked. When I had moved the DNS hosting to Amazon’s Route 53, I had neglected to move over the MX records.

After fixing all of that, I tried updating the app on Heroku to use the new cert, only to get stalled as Heroku’s new SSL add-on rejected it. Certain I had done something wrong, I fumbled through a dozen Heroku SSL how-to posts before finally reverting to their old SSL add-on. It’s no longer documented and is in fact actively discouraged by Heroku, but it also has the lucky trait of actually working with my certificate. Updating DNS caused another hour-long delay because of the high TTL.

I sent two support requests during this process, so I thought I’d rate how each company did:

  • DreamHost: Before I figured out the bad email address, I sent DreamHost a question about why the SSL certificate hadn’t showed up yet. They responded very quickly, and even included a “P.S.” that they were fans of Tweet Marker. Basically they provided excellent support, the best you could ask for.

  • Heroku: When the new SSL add-on wasn’t accepting my certificate, I filed a support request with Heroku as well. The response was an automated reply that they don’t do support past 6pm. For a hosting company that charges a premium, this was a disappointing response. (They responded first thing the next morning, though.)

This SSL glitch was the only significant outage Tweet Marker has had in its first year. I learned a few lessons, took the opportunity to check backups and EC2 servers, and now I’m ready to move on. Hoping for an even better year 2.

Subscription change, new features

When I launched Tweet Marker Plus, I documented that it would store “about a month” of tweets. I didn’t want to promise too much before I fully understood the storage requirements. After a few weeks, I officially bumped it up to “at least 2 months”. I also added a full archive of your own tweets, which are never deleted.

The truth is I’ve yet to write the code that actually deletes any tweets from the database and search index. Eventually I’ll have to, but not yet. So I’ll continue to evolve the service in a way that makes it more useful and sustainable.

Recently I increased the $2/month price to $5/month, with the search index expanded to 3 months of tweets. Today I’m officially bumping the storage again, to 6 months of tweets. I’ve also changed to monthly billing instead of once every 3 months. Everyone who already has the $2/month plan will get to keep it. No price increase for you, and you still get the new 6-month storage and new features, as a thank-you for being an early subscriber.

And I’ve added a major new feature. You can now create custom collections of tweets and publish them to share with others. This is a feature from my iOS app Tweet Library, and in fact any published collection from Tweet Library will also show up in Tweet Marker Plus. New in Tweet Marker, you have the option of keeping a collection private or making it linkable, without it showing up when someone browses the list of collections.

The screenshots linked on the account page include an example of how collections work.

As a bonus for Mac users, there’s also a new menubar search app. This little Mac app hides in the menubar and gives quick access to searching your Tweet Marker Plus timeline and archive. Here’s a screenshot of what version 1.0 looks like.

Bootstrap CSS

I’ve been using Twitter’s Bootstrap in an internal project at VitalSource for a few months, and over the weekend I finally switched to using the CSS framework in Tweet Marker too. The layout now works in more browsers and provides a much better foundation for design changes. It also allowed me to integrate this excellent date picker.

Here’s a short screencast video showing the date picker in a new browsing feature in Tweet Marker Plus. I’m very happy with how this turned out — both the look and functionality. On the server the date ranges are implemented with a Sphinx query, so they can be combined with search terms to help find old tweets.

Hosting costs and config

A few things have changed since I last talked about hosting. Tweet Marker passed 200,000 total users to the API. There are more apps and more platforms. And of course Plus launched with new requirements for database and search indexing of tweets.

Here’s a graph showing monthly hosting costs for the last year, stopping just short of $600/month for April:

Tweet Marker hosting chart

The first dip was when I moved away from Heroku’s dedicated PostgreSQL database, to Redis on EC2. The more recent increase is when I updated capacity for Tweet Marker Plus.

Tweet Marker currently runs on both Heroku and Amazon EC2. On Heroku, there are 6 dynos: 5 web dynos with 3 Unicorn processes each, and 1 background worker. I also run hourly scheduled tasks that add a small number of extra dyno-hours, and sometimes I’ll fire up additional temporary workers.

For Amazon, I run with 3 medium server instances: 1 for Redis master, 1 for Redis slave, and 1 for search with MySQL and Sphinx. The search server is partitioned across multiple EBS volumes, each one mounted as a separate MySQL database and Sphinx index. It is possible for me to move a database or search index shard to a different EC2 instance if I need to, as well as move customers between shards.

The volumes look like this in Amazon’s admin UI:

Tweet Marker volumes

I picked 20 GB shards because it seemed like about the point where the database would be too big to be fast, given the modest hardware. It’s enough to hold several months of tweets for all the users in a shard. I estimate how many users should be in each shard, and when it reaches that number I roll new accounts to the next shard, and so on.

Backup dumps for the Redis database and MySQL get sent to S3 every hour and every day, where I keep the last 24 hourly backups and the last 31 daily backups. I also do occasional EBS snapshots.

I don’t currently need a MySQL slave for backups. If I lose a drive, when I restore the last hourly backup, Tweet Marker Plus will automatically add any missing tweets lost during the downtime to bring things back to a current state.

Overall I’m happy with this setup. It’s as simple as I could design it. Hosting is not cheap, but I think I can run for the rest of the year with very few changes and mostly fixed costs. I also plan to switch to reserved instance pricing at Amazon, which should be a significant discount.

If you’ve made it this far, you probably care enough about servers and Twitter that you should consider signing up for Tweet Marker Plus yourself. Check out the details here.