Ground Truth Testing - Why It's All About the Detail

Ground Truth Testing (GTT) is an extremely important tool that can be used to help the auto industry and mapping providers evaluate quality amongst parking data providers. Simply put, GTT involves going to multiple locations and comparing the information available from a data provider with the information available at the location, to expose the level of correctness and accuracy across both static and dynamic data attributes.

While simple in concept, evaluating suppliers’ Point Of Interest (POI) data is a complex process that requires in depth planning in order to give a consistent and fair evaluation between competing suppliers. When planning for GTT, the end-user experience should help identify the areas to be tested:

1. Areas where parking is most in demand are those where drivers would benefit the most from having a parking information service. These can include downtown shopping, entertainment centers, business districts or locations near major mobility hubs. Testing small market towns and residential areas where parking pressures are very low, although in itself still a test, does not impact a driver significantly as parking will be plentiful, so testing should focus on cities, and city areas where parking is most in demand.

2. Defining the type of parking to test is also important, as this can remove confusion at a later stage when validating the evidence collected during testing. The simplest type of parking to test is off-street (i.e. parking lots / car parks). This is because on-street parking is regulated by cities, and there can be a high number of different parking restriction types such as residential, loading only, permit restricted, disabled, drop off, etc to name a few. Focusing the test around PAID spaces (parking spaces that require a payment, for at least part of a day) ensures that false positive results are kept to a minimum in relation to missing POI on-street data, where a tester has potentially confused residential with paid parking spaces.

3. Another important factor to consider upfront is the weighting you attribute to data attributes and categories in the test criteria. High impact issues that have a significant negative effect on driver experience would carry a higher weighting and could include situations where the customer isn’t informed of a POI’s existence, being taken to a location that no longer exists, incorrect GPS for location/entrance position, etc. Low impact issues have a minor negative effect on the driver experience (and are weighted accordingly), and include missing partial address details, missing dynamic data or missing payment source, etc.  By categorising the testing criteria we can apply weighting that truly reflects its importance to a customer. An example of suggested categories and weightings could include:

• Visibility - identifying that the ground truth observed location is in the suppliers data set, and it includes the mandatory attributes.
• Spatial - based on the collected geospatial data, a parking location’s position data can be validated for accuracy, with appropriate tolerances applied.
• Accuracy - using collected images from the ground truth location, the correctness of the POI attributes can be scored with weightings applied to mandatory vs. non-mandatory attributes, as well as scoring the overall number of attributes available for the location.

It should be ensured that the ground truth surveyors record all on-street locations visited. If no off-street car parks or on-street paid for parking are found in a street, then this must be recorded as survey evidence. Ensuring that you have 100% coverage knowledge is critical to evaluate suppliers for an over/under supply of POI data.

The best way to collect test data (by car, bike, on foot, etc) depends largely on the specific conditions, type of data to be collected, and any time constraints. However, the evaluation of the collected ground truth POI data is best undertaken in an office environment where, by separating the collection and verification steps, a consistent evaluation of every parking suppliers data can be achieved without the time constraints of having to move to the next location or technical constraints.

Also key to getting a fair analysis of the test evidence is having a thorough understanding of the suppliers’ data and data schema. Publicly available data on websites or in mobile apps may not be a fair representation of the actual B2B data set. Global variations in approaches to managing parking can make interpreting and marking test evidence complex, for instance parking tariffs may be displayed with or without parking taxes. It is advisable that prior to the evaluation stage, time is spent with the data supplier to ensure that you are comfortable interpreting the suppliers’ data, and the relevant POI attributes. It is also critical to understand methodological requirements in testing the data, as, for example, the probability of finding available parking cannot be tested using the share of free spots in a location. There are statistical test methods available that provide a fair reflection of the level of predictive quality.

Finally, plan to give the suppliers the opportunity to provide feedback based on the evidence you have collated. Having this feedback enables you to re-score (or not) a POI’s measures and attributes.

In summary, data suppliers such as Parkopedia want you to have the best experience of using their data, and they also want a fair evaluation as part of any procurement process. With in-depth planning and a thorough understanding of the details, you can be confident that the final evaluation of a potential supplier’s solution will be the best fit for your users.

Read more of our blogs.