尽管sync.waitGroup（wg）是规范的前进方式，但确实需要您至少进行一些wg.Add呼叫才能wg.Wait完成。对于诸如Web爬网程序之类的简单事物，这可能不可行，在这种情况下，您事先不知道递归调用的数量，并且需要花费一些时间来检索驱动wg.Add调用的数据。毕竟，您需要加载并解析第一页，然后才能知道第一批子页的大小。
我使用渠道编写了一个解决方案，waitGroup在解决方案中避免了“ Tour of Go-网络爬虫”练习。每次启动一个或多个go-routines时，您就将数字发送到children通道。每当go例程即将完成时，您都会将a发送1给该done频道。当孩子的总数等于完成的总数时，我们就完成了。
我唯一剩下的问题是results通道的硬编码大小，但这是（当前）Go限制。
// recursionController is a data structure with three channels to control our Crawl recursion.
// Tried to use sync.waitGroup in a previous version, but I was unhappy with the mandatory sleep.
// The idea is to have three channels, counting the outstanding calls (children), completed calls
// (done) and results (results). Once outstanding calls == completed calls we are done (if you are
// sufficiently careful to signal any new children before closing your current one, as you may be the last one).
//
type recursionController struct {
results chan string
children chan int
done chan int
}
// instead of instantiating one instance, as we did above, use a more idiomatic Go solution
func NewRecursionController() recursionController {
// we buffer results to 1000, so we cannot crawl more pages than that.
return recursionController{make(chan string, 1000), make(chan int), make(chan int)}
}
// recursionController.Add: convenience function to add children to controller (similar to waitGroup)
func (rc recursionController) Add(children int) {
rc.children <- children
}
// recursionController.Done: convenience function to remove a child from controller (similar to waitGroup)
func (rc recursionController) Done() {
rc.done <- 1
}
// recursionController.Wait will wait until all children are done
func (rc recursionController) Wait() {
fmt.Println("Controller waiting...")
var children, done int
for {
select {
case childrenDelta := <-rc.children:
children += childrenDelta
// fmt.Printf("children found %v total %v\n", childrenDelta, children)
case <-rc.done:
done += 1
// fmt.Println("done found", done)
default:
if done > 0 && children == done {
fmt.Printf("Controller exiting, done = %v, children = %v\n", done, children)
close(rc.results)
return
}
}
}
}
解决方案的完整源代码