Datafiniti re-architects its web-crawler engine and moves to the cloud
Summary
Datafiniti, a startup that wants to apply search to big data applications, has rejiggered the technology behind its 80legs web crawler and moved to the cloud. With its new system, CEO and founder Shion Deysarkar claims that a user can scan every single homepage on the internet in roughly a week and then load that data into whatever database he may be using. Datafiniti used to take “a local data center approach” as the infrastructure behind its 80legs system, but found that it was difficult to scale; the startup also wanted to harness newer cloud technology like the open-source Redis key-value and in-memory database, which Pinterest employs to keep track of its users and their followers. The startup decided to go with Amazon Web Services as its cloud, which also connects with volunteer computers around the globe; Deysarkar said there are now 70,000 machines available for Datafiniti to tap into so scaling is no longer an issue. “They spend their lives asking for work from the master distributor.” Datafiniti has also revamped its interface, which now lets users sign in on a web form to generate a crawl or if they know JavaScript, they can write a custom script to search for a particular bit of information.
