Bachelors Thesis

For my bachelor thesis, I decided to build a web crawler using the technologies I knew at that time. Google was very popular, and I wanted to understand the nuances of crawling a specific area of the internet.

Looking back almost 20 years later, PHP was not the best choice for efficiency. But the project was called agile because it was relatively simple and easy to evolve thanks to a straightforward coupling between code and database. I was trying to reinvent a lightweight ORM and make simple CRUD operations fast.

Agile web-crawler: design and implementation (PDF)

Related works that can be useful:

• The Anatomy of a Large-Scale Hypertextual Web Search Engine
• Crawling the Web
• Focused crawling: a new approach to topic-specific Web resource discovery