Content aggregation options

As the content is organized across many site collections, it is important that the process used to get the information executes very quickly and efficiently. Depending on the type of content you are aggregating and how it will be used, the type of approach may change. In this case the content will also be somewhat dynamic with new project sites being provisioned on a regular basis. It would be impossible to maintain a manual list of all of the active projects.

There are three main approaches that can be considered: reading for the individual sites, using search, and compiling information via a scheduled job.

Reading individual sites or lists

The simplest way to read data from the Server OM is to make a call to SPList.GetItems for a given list. While this works great when you know exactly what you are looking for, and from which specific locations, it does not tend to be a great way to aggregate content, because it is too specific and does not scale as well as some...