Mission, logic and architecture behind WarpURL

20 June 2021

Mission

It is WarpURL’s mission to speed up as many short URLs as possible. There are 2 entities that benefit from this; the user and the environment.

When a user clicks on a short URL, it redirects to a long URL and the time it takes is known as latency. A user clicking on a WarpURL link will see regionally consistent low latencies redirects. With emphasis on the words regionally consistent. The user will see the same amount of latency whether they are located in North Virginia (US) or in Mumbai (Asia Pacific).

We have only used a construct of fast, pinning that to an actual number is more concrete, that is why we build the URL Shortener Latency Comparison page. WarpURL does the redirect from short to long URL on average 50ms independent of where your user is located. That is between 5 to 7 times faster than standard shorteners.

Getting the user to your site as quick as possible reduces your drop off rate and in return increases your potential sales. See the links at the bottom for other articles that explore the correlation between first page load and drop off rate.

WarpURL really shines if the link is shared with the user directly, think SMS, email, etc. Then any of the standard shorteners like Bitly can be compared directly with WarpURL on the URL Shortener Latency Comparison page. Short links are not always shared directly but are sometimes shared using social media.

Social media sites will shorten your already shortened link again with their own shortener (double linking). On average a user in Mumbai clicking on a Bitly link shared on LinkedIn adds ~1100ms. So it forces the user to wait more than 1 second before they even reach your website! More details in the blog post we wrote on Double Linking.

Unfortunately, we can’t control the latency the social media sites add *yet, but we can control the latency of the URL shortener we use.

WarpURL uses smart caching to bring the URL as close as possible to the user. If the above-mentioned link was shortened with WarpURL, the total latency amounts to ~900ms. Unfortunately, LinkedIn is the biggest contributor to latency here adding ~850ms.

Now let’s talk about the grand scale and environmental impact of speeding up a short URL. Let’s say you have a popular link and it gets 100,000 clicks. Let’s assume all the clicks are from the US, that means WarpURL will save you ~30ms per click, saving all users ~3000 seconds or ~50 minutes of wait time. Assuming all clicks are from Mumbai, WarpURL will *save you ~230ms per click, that amounts to ~27000 seconds or ~450 minutes which is ~7.5 hours!

The time that is saved translates to:

Giving time back to your users as a collective.
Involving less servers and compute intensive operations, reducing your carbon footprint and contributing to a better tomorrow.

The unfortunate truth about URL Shortneres - It forces latency onto every link and person! Billions of links are clicked every day, billions of seconds wasted 🤯.

"We" need/can do better, the social sites need to realize the impact their shorteners are forcing unto the end user. They will probably not buy out/use WarpURL, but I hope to inspire a change in mindset and *hopefully they will act on it.

Logic

The following logic explains how WarpURL optimizes for latency and brings the URL closer to the user.

The value X in the following calculations is only used to show ratios between the total time for comparison. X is actually the average time it takes WarpURL to move from shortened URL to the long URL. This is approximately between 30 to 50ms after the first click.

When a user clicks on a standard URL shortener link:

FIRST CLICK:

The link domain is resolved (using DNS) to an IP address for the server.
The data/request is sent from the user to the server. This path involves going through multiple servers and long distances to reach the server.
The server receives the request, takes the short URL and looks it up in a database to get the long URL.
The long URL is returned in the response. The return path might be different from the request, but we can assume there is still latency.

Total time: DNS + time to server + short to long URL lookup + time back to user = ~ 6X

SECOND CLICK:

The second click has the exact same path except that DNS would have been cached. Considering the DNS lookup time to be negatable, the total time is also about the same.

Total time: time to server + short to long URL lookup + time back to user = ~ 6X

When a user clicks on a WarpURL link:

FIRST CLICK:

The link domain is resolved (using DNS) to an IP address for the server.
The data/request is sent from the user to the closest CDN Point of Presence (POP), and from there it is sent to the server. This path goes through less servers (hops) and is sent to the closest of twelve servers and database deployed around the world.
The server receives the request, takes the short URL and looks it up in a database (same region) to get the long URL.
The long URL is returned in the response. The return path might be different from the request, but we can assume there is still latency.

The first path of a WarpURL link click is only slightly faster than the standard URL shortener link. This is mainly because the only difference on the first lookup is that it gets processed at the closest server (12) to the user. This on its own does not provide a significant improvement.

Total time: DNS + (to POP & to server) + short to long URL lookup + (from server back to POP) + to user = < 6X

SECOND CLICK:

The data/request is sent from the user to the closest CDN Point of Presence (POP). The POP then returns the cached response back to the user.

As with the standard URL shortener, we assume DNS is cached for the second request. Only the first link click for that region has to go to the server, all future clicks will be read from the closest POP cache. This is what gives WarpURL its speed, it optimizes for the "second" and all future clicks for users in that region.

Total time: to POP + cached response lookup + from POP + to user = X

It gets even worse when you take into account inconsistent last-mile latency, which is the latency added between the user and the first node in the ISP network. Some examples of these connection points include cellular towers (~100ms), telephone lines – ADSL(~50ms), fiber (~20ms) and satellite(>100ms). The [URL Latency Shortner] tests are done out of AWS data centers and does not include this latency.

Architecture

Warning, heavy tech talk following.

WarpURL is a 100% serverless architecture built on AWS. When a new Subscription is created, it kicks of a CloudFormation deployment that contains your own CloudFront CDNs, one for admin and one for shortening. You also get your own (not exclusive list):

Lambda and Lambda@Edge functions that we use for compute .
DynamoDB & DynamoDB Global Tables (12 regions) used as a database.
Cognito Pool for user management.
API Gateway.
S3 buckets for storage.
SQS for queueing.

This might sound like a lot of work and it is, but the results speak for themselves. No other URL Shortener places a CDN on top of your shortening domain to get the incredible speeds that WarpURL can deliver. Everything is automated, it was a lot of upfront work that should pay dividends throughout the product life cycle.

Creating infrastructure like it is just another programming language construct has its limits, specifically AWS account limits. That is why we deploy customer subscriptions over multiple AWS accounts, this helps reduce the blast radius further.

The benefits of a 100% serverless architecture is that we scale almost instantly and do not over provision servers. Less servers equals a happier environment and reduced carbon footprint. We can go from 0 to thousands of clicks per second without breaking a sweat. WarpURL is built for scale anticipating billions of links and clicks.

Links:

Mission, logic and architecture behind WarpURL

Mission

Logic

When a user clicks on a standard URL shortener link:

When a user clicks on a WarpURL link:

Architecture

Product

Resources

Company