Optimizing SpotCollector's Performance
When an incoming spot arrives from a spot source, SpotCollector displays the spot in the appropriate spot source window (unless the spot source is configured to display only announcement and talk messages), and places the spot in a first-in first-out queue. Spots are continously removed from this queue and processed by looking through SpotCollector's Spot Database in reverse chronological order to see if there's already an Entry for the spotted station in the spotted mode near the spotted frequency. It keeps looking until
- it finds an existing Spot Database Entry for the station, in which case it updates that Entry
it encounters a Spot Database Entry that is older than the max separation in time parameter or encounters the earliest Entry in the Spot Database, in which case it creates a new Spot Database Entry
The number of not-yet-processed spots in the queue is displayed to the immediate right of the letter Q in the upper-left corner of SpotCollector's main window. If this queue overflows - indicating that spots are arriving too frequently to be processed given SpotCollector's current configuration - the letter Q is rendered in red font; clicking it will reset its color to black, but SpotCollector will continue to operate normally whether you do so or not. The techniques described in this article can be used to reduce the amount of computation required, eliminating queue overflows.
Thus, there are nine primary factors that most affect SpotCollector's performance and PC resource (CPU and RAM) consumption:
- the number of active Spot Sources
- the configuration of Spot Source windows to display or not display DX spots
- the size of the Spot Database
the Max separation in time setting
the Maintain DXCC entity-band-mode fields setting
the Record individual spot information setting
- the rate of incoming DX spots
- web server
- hardware capabilities
Number of active Spot Sources
Collecting DX Spots from multiple sources is useful because any one source may provide only a subset of all DX Spots, and because individual sources can occasionally become unavailable or inaccessible. Within a minute or two, however, most of the DX Spots received from one Spot Source will exactly duplicate the DX Spots received from another Spot Source. To prevent redundant searches of the Spot Database, SpotCollector maintains a cache of recent DX Spots that it uses to filter out exact duplicates. While searching the cache for duplicates consumes far fewer PC resources than searching the Spot Database, it is not free. Thus reducing the number of active Spot Sources to the minimum that meets your needs will reduce the load on your PC. If your DXing needs are met by 4 Spot Sources, there is no need to connect to 7 just because SpotCollector has this capability.
The number of spots received since startup or during the last full hour is displayed in the Size Control panel on the Configuration window's Spot Database tab.
You can reduce the volume of spots received from each spot source by configuring SpotCollector's Pre-Filtering mechanism to discard spots on bands for which you have no transceiver or antenna, and to discard spots in modes you do not utilize.
Configuration of Spot Source windows to display or not display DX spots
On the Configuration window's Spot Sources tab, an Ann/Talk option is provided for each Spot Source. If this option is enabled, only announcement and talk messages are displayed and retained in Spot Source windows; DX spots are discarded after being entered into the Spot Database. If the Ann/Talk option is disabled, DX spots are retained. These retained spots consume memory; with many active Spot Sources generating DX spots over multiple days, this memory consumption can be signficant enough to degrade overall system performance. Since all incoming DX spots are accessible via the Spot Database Display (except those rejected as too old by the Maximum age of valid incoming spots setting, and those rejected by the by the Special Callsigns mechanism), there is no compelling reason to display them in Spot Source windows. For optimum performance, then, check all the Ann/Talk boxes on the Configuration window's Spot Sources tab, and restart SpotCollector; note that the IRC spot source's Ann/Talk option is permanently enabled.
Size of the Spot Database
By retaining Spot Database Entries, one can observe propagation patterns over time, and discover a needed DX station's operating habits. If you aren't concerned about resource consumption, the only upper bound on the size of the Spot Database is free space on the hard drive that hosts it. However, a larger Spot Database causes SpotCollector's database engine to consume more RAM; as free RAM becomes scarce, Windows spends more and more CPU time shuttling information between RAM and your hard drive so that applications have the RAM they need when they're running (a scheme referred to as virtual memory). Thus there is a direct tradeoff between Spot Database size and PC resource consumption. You can configure SpotCollector to automatically prune away the oldest Entries in your Spot Database in order to maintain a maximum size -- either hourly, or each time SpotCollector starts. To do this, see the Size Limit Panel section in SpotCollector's documentation. If you are observing long-term propagation patterns or staking out a needed DX station, then maintaining a large Spot Database makes sense; if not, you can reduce PC resource consumption by configuring SpotCollector to automatically prune Entries older than a day or two either by specifying a maximum age or a maximum size. If you run SpotCollector 24x7, then enable hourly pruning; if you start SpotCollector at the beginning of a DXing session, then pruning on startup will be more efficient. During contests, with Prune entries older than this age selected, consider temporarily specifying a reduced age setting, as even a setting of 2 days can retain 20 thousand Spot Database Entries; the advantage of selecting and specifying Prune Spot Database to this size is that it maintains that maximum size independent of variations in the rate of incoming spots.
You can configure SpotCollector to limit the Spot Database to a relatively small size (e.g. 1 day) but still retain unlimited Spot Database Entries for specific DX stations whose operating patterns you're trying to discern: add the callsigns of such stations to the Special Callsign list with a NoPrune tag, as described here.
Max separation in time setting
The Max separation in time setting in the Combination Criteria panel on the Configuration window's Spot Database tab determines how far back in time SpotCollector searches the Spot Database for an existing Entry when processing an incoming DX spot. Thus larger values of Max separation in time mean that SpotCollector could review a lot of Spot Database Entries each time a new DX spot arrives -- and thus consume more PC resources. Smaller values of Max separation in time reduce PC resource consumption, but could produce multiple Spot Database Entries for the same DX station, which is much less a resource consumption issue than a usability issue: were you to set Max separation in time to 0, then the Spot Database Display would contain as many redundent Entries as a raw telnet cluster. A good baseline for Max separation in time is 60 (minutes). During non-contest conditions, this minimizes redundant Entries while limiting the Spot Database searching to 50-100 entries for each incoming DX spot. Reducing Max separation in time to 15 (minutes) might be appropriate during contests that generate high DX spot volume.
Maintain DXCC entity-band-mode fields setting
If you have enabled the Maintain DXCC entity-band-mode fields setting in the Controls panel on the Configuration window's Spot Database tab in order to pursue unworked and unconfirmed entity-band-modes, the SpotCollector will more frequently query your log database than would would otherwise be the case.
Record individual spot information setting
If you have enabled the Record individual spot information setting in the Controls panel on the Configuration window's Spot Database tab so that you can right-click the Spot Database Entry for P5DX on 20m CW and select Display spots of P5DX near 14025 in CW from the popup menu, SpotCollector is retaining all of the information from every unique incoming spot of a station in that station's Spot Database Entry. During a popular contest, or when using the Reverse Beacon Network as a spot source, the resulting consumption of memory may stress your system.
Rate of incoming DX spots
During contests, the rate at which DX spots arrive increases dramatically above the normal baseline of 50-100 spots per hour to peaks exceeding 500 spots per hour. Furthermore, larger numbers of DX stations can be active, so more new Spot Database Entries are created per hour, which means that the average Spot Database search is longer and thus consumes more PC resources. Spot sources connected to CW, RTTY, and FT8 skimmers can generate high spot volumes in the absence of contests; the Reverse Beacon Network (RBN) can generate 20,000 spots per hour during a popular contest! Since you have no control over the incoming DX spot rate during contests, consider compensating for it by
- adjusting one or more of the factors described above
using SpotCollector's Pre-Filter to discard incoming spots that aren't of interest before they are processed
Process Priority
You can increase the fraction of your CPU's attention that is devoted to SpotCollector by selecting Above Normal or High in the Process Priority panel on the Configuration window's Spot Database tab. Note that doing this will reduce the fraction of your CPU's attention that is devoted to other competing applications.
Web Server
SpotCollector's built-in web server lets you view spots from any PC on your network. If you're not using this capability, you can save both RAM and processing time by disabling the web server on the Configuration window's Spot Database tab
Hardware capabilities
SpotCollector and DXKeeper both employ a database. In both applications, most database accesses are read operations, which are faster and less resource consumptive than write operations. You can generally improve the performance of database systems by
- increasing CPU speed
- making more physical RAM available
- increasing the paging file size
- increasing secondary storage (SSD, hard drive) speed
- adding a second secondary storage device (SSD, hard drive) and installing your applications there (so that application accesses and Windows accesses don't compete)
If your PC's processor provides more than two cores, it's been observed that SpotCollector performance continues to improve with up to 3 dedicated cores.
- For Windows XP, Vista, and 7, instructions for dedicating a core to an application can be found in
http://www.addictivetips.com/windows-tips/how-to-set-processor-affinity-to-an-application-in-windows
- For Windows 10,
start the Windows Task Manager; if the words More details appear in the lower-left corner, click them to display details.
right-click on the entry for SpotCollector, and select Go to Details from the pop-up menu.
right-click the entry for SpotCollector, and select Set affinity from the pop-up menu
For Windows 7, 8, 10, and 11, Process Lasso is a utility that provides automation for dedicating cores to application.
When running a non-trivial application like SpotCollector or DXKeeper, there is always a bottleneck that limits performance. Determining which of the above hardware upgrades will actually improve performance is difficult without instrumentation and/or monitoring gear. Sometimes, a change yields a significant improvement in the performance of one application, but yields no benefit at all to another application. And sometimes, a change overcomes an application's current bottleneck, but performance only increases a few percent before an entirely different bottleneck is encountered.
Thus the general rule is take the cheapest steps first. Optimizing SpotCollector's settings as described above costs nothing, but can yield major benefits. If you already have a quad core CPU, it costs nothing to assign two of its cores to SpotCollector. After that, additional RAM is usually the right next course of action.
Running SpotCollector on Windows Vista, 7, 8, 10, and 11
By default, Windows Vista, Windows 7, Windows 8, Windows 10, and Windows 11 enable "visual effects" that can on some systems result in SpotCollector "stealing" the mouse cursor. This can be frustrating if you're trying to compose an email message or update a document, only to have the text you're typing suddenly begin appearing in the Filter panel on SpotCollector's Main window. You can configure Windows to prevent this.
Mouse cursor "stealing" can also occur if SpotCollector spends so much time searching its Spot Database that Windows erroneously concludes that SpotCollector must not be functioning correctly, and temporarily displays a not responding message in SpotCollector's title bar. If you see this happening, either reduce the size of the Spot Database as described above, or reduce the Max separation in time setting as described above.
Post a question or suggestion on the DXLab Discussion Group