See my question on stackoverflow for more details about what this requires: [login to view URL]
I have a working epoll based web crawling architecture which consumes over 3 Gbps of bandwidth on my dedicated server. When I add MySQL code to keep the state of the frontier and to save crawled information the web crawler slows down. Feedback to my question suggests I need to use asynchronous calls to MySQL so that the web crawler can continue to create new connections while waiting for query results.
I need someone with experience of asynchronous MySQL programming in C to fix this for me.