ePrints@IIScePrints@IISc Home | About | Browse | Latest Additions | Advanced Search | Contact | Help

Online algorithms for estimating change rates of web pages

Avrachenkov, K and Patil, K and Thoppe, G (2022) Online algorithms for estimating change rates of web pages. In: Performance Evaluation, 153 .

per_eva_153_2022.pdf - Published Version

Download (1MB) | Preview
Official URL: https://doi.org/10.1016/j.peva.2021.102261


A search engine maintains local copies of different web pages to provide quick search results. This local cache is kept up-to-date by a web crawler that frequently visits these different pages to track changes in them. Ideally, the local copy should be updated as soon as a page changes on the web. However, finite bandwidth availability and server restrictions limit how frequently different pages can be crawled. This brings forth the following optimization problem: maximize the freshness of the local cache subject to the crawling frequencies being within prescribed bounds. While tractable algorithms do exist to solve this problem, these either assume the knowledge of exact page change rates or use inefficient methods such as MLE for estimating the same. We address this issue here. We provide three novel schemes for online estimation of page change rates, all of which have extremely low running times per iteration. The first is based on the law of large numbers and the second on stochastic approximation. The third is an extension of the second and includes a heavy-ball momentum term. All these schemes only need partial information about the page change process, i.e., they only need to know if the page has changed or not since the last crawled instance. Our main theoretical results concern asymptotic convergence and convergence rates of these three schemes. In fact, our work is the first to show convergence of the original stochastic heavy-ball method when neither the gradient nor the noise variance is uniformly bounded. We also provide some numerical experiments (based on real and synthetic data) to demonstrate the superiority of our proposed estimators over existing ones such as MLE. Our algorithms are readily applicable to the synchronization of databases and network inventory management. © 2021 Elsevier B.V.

Item Type: Journal Article
Publication: Performance Evaluation
Publisher: Elsevier B.V.
Additional Information: The copyright for this article belongs to Authors
Keywords: Approximation algorithms; Inventory control; Iterative methods; Search engines; Stochastic systems; Web crawler; Websites, Change rate; Database synchronization; Finite bandwidth; Heavy ball; Local cache; On-line algorithms; Stochastic approximations; Track changes; Web Crawling; Web-page, Approximation theory
Department/Centre: Division of Electrical Sciences > Computer Science & Automation
Date Deposited: 31 Dec 2021 07:06
Last Modified: 31 Dec 2021 07:06
URI: http://eprints.iisc.ac.in/id/eprint/70661

Actions (login required)

View Item View Item