25 Dec, 2009
The evolution of push email
Posted by Bhavin Turakhia
I have been meaning to compile research on push technologies and the evolution of push email since a while. A conversation with ramki on how gmail offers push email using Microsoft Direct Push triggered my research and here is a summarized compilation of my findings
Blackberry
Blackberry works by downloading your email to its NOC. Whenever it finds a new mail it pushes it out to your cellphone. This is why Blackberry requires you to subscribe to a specific blackberry plan through your Telco. If you have a blackberry plan, your Telco forwards your registration to Blackberry and whenever you are online, Blackberry knows how to find you. Whenever the Blackberry NOC downloads a new mail from your mail servers it pushes the mail out to your cellphone through your Telco. This last bit is proprietary and presumably requires support from the Telcos end. Since a data-push is made to your cellphone only when there is data to be pushed, your battery life is conserved
References
Microsoft AUTD
This was the earlier version of Microsoft’s push technology. Basically Microsoft Exchange (or any mail server) is configured to simply send out a special SMS to your cellphone number each time new data needs to be pushed to your cellphone, and your cellphone is not currently connected. This SMS does not get displayed in your inbox, but rather triggers the mail application (or any other applicable app) in the background to sync with the server. This mechanism requires the use of an SMS gateway and/or possible support from your Telco to be able to send out these type of SMSes.
References
Microsoft Direct Push
This is the latest version of Microsoft’s push technology. It sounded esoteric, but upon close investigation it turned out to be a fancy microsoft name for a technology that has existed for nearly a decade viz. Comet. The way this works is as follows -
- The application on your mobile (lets say your email client) makes a Comet HTTP request to an http server. Along with this request it sends the server a timeout value – say T=15 minutes
- If in the next 15 minutes the server receives a new mail, the server immediately sends an HTTP response with a status that results in the client syncing with the server and then once again issuing a long poll http request
- If in the next 15 minutes the server receives no new mail, the server must at the end of 15 minutes send an empty status response to the client, upon receiving which, the client once again initiates a new HTTP request
- Since this method requires a poll to the server every ‘x’ minutes it typically consumes more battery than Blackberry but the difference should not be too high
- Since the Telco en route may have its own enfroced timeout values on any http connections, it may disconnect a long-running HTTP connection
- Therefore the actual timeout time must be negotiated between the client handset and the server during bootstrapping by attempting higher and higher timeout values until the highest value that the Telco permits is discovered
- The HTTP connection may also get dropped due to network failures and switch-overs. Clients can detect this at the TCP/IP level and re-attempt connections. I am not entirely sure about this part, but the HTTP client presumably also uses TCP Keep alive internally to ensure the TCP connection remains alive
- One may want to lookup the actual port numbers and protocol implementation of Microsoft Direct Push, since it is likely that most Telcos support Microsoft Direct Push and therefore if one was to emulate their push service on the same server port it is likely that Telcos would let the long-running HTTP connection persist
- The advantage of this method is it does not require any special Blackberry plan
References
- How Direct Push really works - http://www.techatplay.com/?p=11
- Comparison between Direct Push and Blackberry - http://www.techatplay.com/?p=13
- What uses lesser traffic – Microsoft Direct Push or Blackberry – http://www.rysavy.com/Articles/Rysavy_Wireless_EMail.pdf
- http://en.wikipedia.org/wiki/Comet_(programming)
- http://www.emansio.com/ – A push email product for windows mobile
- http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html – Overview on TCP Keep Alive
Misc References
- A detailed guide on Direct Push - http://www.techatplay.com/?p=8
- Definition of Push email - http://www.techatplay.com/?p=32
- A detailed list of all the push technologies out there – http://en.wikipedia.org/wiki/Push_e-mail
-
http://en.wikipedia.org/wiki/Lemonade_Profile
- Another interesting push email technology that uses IMAP notify and IDLE – http://en.wikipedia.org/wiki/Lemonade_Profile
If we are only talking “Email”, then the IMAP protocol already supports push by using the IMAP idle command , in TOKEN very efficacious manner.
Here’s how , I have excerpted TOKEN telnet conversation with our mail.directi.com server. I have broken down the coversation for explaining how the *PUSH* happens.
1. Basic login with the IMAP server
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
TOKEN LOGIN “ajay.m@directi.com” “superdupersecretconcept”
TOKEN OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH] Logged in
2. Next i select the mailbox that i’m interested in, my case being the INBOX and the server responds back with information on the mailbox. Standard stuff.
TOKEN SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded \*)] Flags permitted.
* 25027 EXISTS
* 0 RECENT
* OK [UNSEEN 5882] First unseen.
* OK [UIDVALIDITY 1173073810] UIDs valid
* OK [UIDNEXT 71886] Predicted next UID
* OK [HIGHESTMODSEQ 26918] Highest
TOKEN OK [READ-WRITE] Select completed.
3. Now comes the interesting bit , the next command that i issue is IDLE. The server now knows that i want to keep this connection open and notifies me whenever there’s activity on the mailbox. If you look carefully, it first told me that there was 1 new message in the INBOX(* 1 RECENT) and after a few seconds i received another mail and within a few seconds i receive another notification which says (* 2 RECENT).
TOKEN IDLE
+ idling
* OK Still here
* 25028 EXISTS
* 1 RECENT
* 25029 EXISTS
* 2 RECENT
DONE
TOKEN OK Idle completed.
4. We now have the crux of the discussion behind us. “Push Email”. Rest is all boilerplate stuff. The mail client can now issue a DONE command which ends the IMAP IDLE, and subsequently fetch the messages from the server. Also you may ask what about connection time outs etc.This can be done by issuing a NOOP command every 10-15 minutes to keep the connection alive. Trust me it’s all boilerplate. I’ve left this out for brevity. If you are really interested check out the IMAP RFC and hack it out yourself.
Microsoft Direct Push works on ALL of your PIM information. Not necessarily only Email. They seem to have mimicked this behaviour using the champion of all protocols, HTTP.
Hope this helps and posts as a comment on bhavin’s blog.
The drudgery ends !!
http://www.isode.com/whitepapers/imap-idle.html
sed ‘s/TOKEN/a’ /when/it’s/not/a/telnet/coversation
Authentic tech doc on Microsoft Direct Push:
http://technet.microsoft.com/en-us/library/aa997252(EXCHG.80).aspx
Note that the parking time for the connection (or keep-a-live) is being auto-sensed. Essentially, this can work on any network without operator support. The operators can help devices by increasing the connection timeouts. (Why not a day!?
)
And also, not that there is no battery-life cost associated with any connection thats idle.
I have tried Emansio:
1. Its IMAP IDLE based client, as Ajju described.
2. It is buggy and is not usable on WinMo 6.5
But for all one cares, the next generation phones may have clients with IMAP-IDLE support in-built, with auto (probing) timeout sensing.
Also Bhavin, TCP Keep-Alive is completely optional here. I presume neither the ActiveSync client nor the Exchange Server with DirectPush would ever turn-on TCP keep-alive. Thats unnecessary network and battery cost for mobile devices.
In this setup, the TCP Keep-Alive helps only in detection of abrupt server crash. Given its infrequent, TCP keep-alive is unnecessary. I assuming the server side open socket has insignificant cost, so even if client side end is broken, the server has no issues. The detection is postponed at max to the next event or timeout whichever is earlier.
@Ramki: however without TCP Keep alive – the next push would get delayed by 15 minutes or even a day (if telcos truly implement what you mention above
) …
@Bhavin Ya.. but again only on server/telco infra failure!
Grassroots Indigenous Multimedia announces the launch of their new Ojibwe Language learning software, Ojibwemodaa.Best Way : Rosetta stone learning Tips. Available in more than 30 languages, the Rosetta stone language-learning solution is used by schools, the consumer will write a message on the Personal Rosetta Stone. Download free language software, play online games and start learning a language now.
@Ramki: Actually, the keep-alive is completely necessary. TCP settings can get switched more than you think, just to name a few: when you go into the lift, in a fast-moving car through a highway or passing through an area with poor mobile reception. Without the timeout (or a very long one), the TCP connection with the server might be broken, and the client on the phone will be none the wiser.
This is because Microsoft’s method involves using a technology not really meant for push email (Its HTTP) and tweaking it to be used as such. Its a smart method, but I’ve personal experience in an office with poor/spotty mobile reception.
Upon losing the lock on the network and successfully reconnecting, the Windows Mobile phone has to wait for the next time-out before getting new e-mail while the BlackBerry gets the push emails the moment it reconnects.
The Reebok Easytone Shoes is probably the most athletic and traditional of the Calorie Burning or Reebok EasyTone category, emulating their traditional Reebok EasyTone Trainers. The Reebok ZigTech shoes,