{"post":{"id":"60456e6c8458d42fc821c656","_id":"60456e6c8458d42fc821c656","coverImage":{"isPortrait":false,"url":"https://cdn.hashnode.com/res/hashnode/image/upload/v1615104823423/kqfLz65Js.png","__typename":"PostCoverImage"},"slug":"scalability-challenge-how-to-remove-duplicates-in-a-large-data-set-100m","readTime":4,"replyCount":1,"responseCount":1,"brief":"Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even basic tasks like counting distinct elements, membership check, filtering duplicate elements, finding minimum, ...","subtitle":"System Design","isFeatured":false,"contributors":[{"_id":"5f25a0c390ac4260edf355e4","name":"Catalin Pit","photo":"","username":"Catalinpit","__typename":"User"}],"bookmarked":false,"views":1003,"url":"https://pankajtanwar.hashnode.net/scalability-challenge-how-to-remove-duplicates-in-a-large-data-set-100m","totalReactions":81,"title":"Scalability Challenge : How to remove duplicates in a large data set (~100M) ?","dateAdded":"2021-03-08T00:23:08.813Z","cuid":"cklzucacg08tmfss181cc697a","preferences":{"disableComments":false,"__typename":"PostPreferences"},"__typename":"Post","seo":{"title":null,"description":null,"__typename":"SEO"},"ogMetaData":{"image":"","__typename":"OpenGraphMetaData"},"author":{"_id":"5ffaa2c0a538d0013515544a","name":"Pankaj Tanwar","photo":"https://cdn.hashnode.com/res/hashnode/image/upload/v1610284888804/KZNaB4c8y.jpeg","isPro":false,"username":"pankajtanwar","__typename":"User","tagline":"I write about System Design & Web Tech. Curious about how things work!"},"coAuthors":[],"series":{"_id":"6071390f1494367be6d296f2","slug":"system-design","name":"System Design","cuid":"cknbb0p1604xxh2s1a8dccvjl","__typename":"Series"},"tags":[{"_id":"56744723958ef13879b955b0","logo":"https://res.cloudinary.com/hashnode/image/upload/v1496913335/duppaieikvyvepmoj6uz.png","name":"System Architecture","slug":"system-architecture","__typename":"Tag"},{"_id":"5691193ecad8946e563c56e9","logo":null,"name":"scalability","slug":"scalability","__typename":"Tag"},{"_id":"56744721958ef13879b94c7e","logo":"https://cdn.hashnode.com/res/hashnode/image/upload/v1535648192079/H1daWiBvQ.png","name":"General Programming","slug":"programming","__typename":"Tag"},{"_id":"56744722958ef13879b94e89","logo":"https://cdn.hashnode.com/res/hashnode/image/upload/v1513324674454/r1qtxW-zf.png","name":"Design","slug":"design","__typename":"Tag"}],"publication":{"_id":"5ffaa2e4a538d00135155455","url":"https://pankajtanwar.hashnode.net","canonicalURL":"https://pankajtanwar.hashnode.net","title":"Pankaj Tanwar's Blog","displayTitle":"Blogs - Pankaj Tanwar - Software Engineer","isHeadless":false,"favicon":"https://cdn.hashnode.com/res/hashnode/image/upload/v1610284970532/2686bhEjU.jpeg","headerColor":"","metaTags":"\n\n\n\n \n\n \n \n \n \n \n \n\n \n \n \n \n \n","isTeam":false,"domainInfo":{"hashnodeSubdomain":"pankajtanwar","domain":null,"__typename":"DomainInfo"},"urlPattern":"SIMPLE","__typename":"Publication"},"comments":{"edges":[{"node":{"id":"6045c8225a56f82fa3ecd70b","author":{"_id":"5f25a0c390ac4260edf355e4","name":"Catalin Pit","photo":"","isPro":false,"username":"Catalinpit","__typename":"User"},"content":{"html":"

My head is spinning from all those numbers, haha!

Great article; well done, Pankaj Tanwar!

\n","markdown":"My head is spinning from all those numbers, haha! \n\nGreat article; well done, @[Pankaj Tanwar](@pankajtanwar)!","text":"My head is spinning from all those numbers, haha! \nGreat article; well done, Pankaj Tanwar!\n","__typename":"Content"},"dateAdded":"2021-03-08T06:45:54.170Z","stamp":"ckm080igp0a4jers1blz1bti4","totalReactions":2,"__typename":"Comment","replies":{"edges":[{"node":{"id":"6045fa048458d42fc821cfd1","author":{"_id":"5ffaa2c0a538d0013515544a","name":"Pankaj Tanwar","photo":"https://cdn.hashnode.com/res/hashnode/image/upload/v1610284888804/KZNaB4c8y.jpeg","isPro":false,"username":"pankajtanwar","__typename":"User"},"content":{"html":"

Thanks Catalin. Big fan of your articles. Keep up the great work ✌️

\n","markdown":"Thanks Catalin. Big fan of your articles. Keep up the great work ✌️","text":"Thanks Catalin. Big fan of your articles. Keep up the great work ✌️\n","__typename":"Content"},"dateAdded":"2021-03-08T10:18:44.618Z","stamp":"ckm0fm87d0barfss16kq40kle","totalReactions":0,"__typename":"Reply"},"__typename":"CommentReplyEdge"}],"pageInfo":{"hasNextPage":false,"endCursor":"MQ==","__typename":"PageInfo"},"__typename":"CommentReplyConnection"}},"__typename":"PostCommentEdge"}],"pageInfo":{"hasNextPage":false,"endCursor":"MQ==","__typename":"PageInfo"},"__typename":"PostCommentConnection"},"commenters":{"edges":[{"node":{"_id":"5f25a0c390ac4260edf355e4","name":"Catalin Pit","photo":"","isPro":false,"username":"Catalinpit","__typename":"User"},"__typename":"PostCommenterEdge"}],"__typename":"PostCommenterConnection"}}}

Scalability Challenge : How to remove duplicates in a large data set (~100M) ?

2 comments