Measuring canopy height in soybean and wheat using a low‐cost depth camera

Canopy height is an essential trait in high throughput phenotyping that is often only captured as a single point, which is not always representative of canopy height. New 3D depth cameras such as the RealSense D415 (Intel Corporation, Santa Clara, CA, USA) may provide a fast and affordable solution for measuring height from portable, ground‐based phenomics systems. Our goal was to determine if the D415 was effective at measuring crop heights under field conditions in wheat (Triticum aestivum) and soybean (Glycine max) plots. The D415 camera was integrated into our PlotCam platform using the open software development kit from Intel. Distance arrays were captured for each plot at weekly intervals over the growing season. These were compared to canopy heights measured using a single point LiDAR (SPL) system operated by hand. Over the growing season the D415 heights were significantly correlated with the SPL heights in both wheat and soybean with coefficients of 0.77 and 0.95 and NRMSE 0.23 and 0.17 m, respectively. Early season D415 height measurements were not as similar to the SPL as the mid‐and late‐season measurements in wheat and soybean. The relatively low cost and open software development kit of the D415 makes it a promising tool for high throughput phenotyping applications.


INTRODUCTION
Plant height is a morphological trait that can be used to measure plant growth, and predict biomass gain and yield (Ehlert et al., 2009;Wang et al., 2018). High throughput phenotyping (HTP) uses traits such as canopy height and vegetative ground cover to predict above ground biomass (Jimenez-Berni et al., 2018). Botanists define plant height as the distance between the ground and the upper boundary of the leaves (Pérez-Harguindeguy et al., 2016). Plant canopy height is Abbreviations: API, application programming interface; DAS, days after seeding; HTP, high throughput phenotyping; IR, infrared radiation; LiDAR, light detection and ranging; NRMSE, normalized root mean square error; SPL, single-point LiDAR.
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2021 The Authors. The Plant Phenome Journal published by Wiley Periodicals LLC on behalf of American Society of Agronomy and Crop Science Society of America usually measured by hand at a single point relative to a meter stick. At maturity in cereals, this is done by placing the meter stick amongst the plants and eyeballing the average height of the heads less the awns (Torres & Pietragalla, 2012). While measuring mature plant height is relatively easy using single point measurements, crop canopies are often multi-level and may not be well represented by a single measurement (Deery et al., 2020). Measuring crop canopies by hand is time consuming, laborious, and prone to bias in measurement and errors transcribing the data (Wang et al., 2018). In plant breeding programs, canopy height is often only measured once prior to harvest, limiting the information that can be gained from this trait. In HTP, canopy height is measured throughout the growing season with accuracy and speed often not attainable by manual measurements (Jimenez-Berni et al., 2018).
Several types of instruments have been proposed to measure plant height. Ultrasonic sensors measure height based on the time it takes for a sound pulse to reflect from the canopy back to the sensor (Bai et al., 2016). Light detection and ranging (LiDAR) sensors use a similar principle as ultrasonic sensors, using laser light instead of sound. Scanning LiDAR sensors use a high sampling rate frequency and thousands of measurements to construct 3D point cloud surface models (Deery et al., 2020;Jimenez-Berni et al., 2018). Photogrammetry techniques using multiple cameras on ground-based platforms, or multiple overlapping images on aerial platforms generate 3D topographic canopy models used to extract plant heights (Holman et al., 2016). Wang et al. (2018) compared an ultrasonic sensor, a single point LiDAR (Garman, LiDAR Lite V3, KS, USA), an array of four digital cameras, and a Kinect v2 camera (Kinect for Windows version 2, Microsoft Corp, Redmond, WA, USA) for their accuracy in measuring sorghum (Sorghum bicolor) canopy height from a ground-based platform equipped with a solar shade. They found that while all proximal sensors were significantly correlated with manually measured height, the elevation model from the digital camera array was superior to the ultrasonic senor, the single point LiDAR, and the Kinect v2 camera, which had a restricted range of operation in direct sunlight.
Recently, the RealSense D415 depth camera (Intel Corporation, Santa Clara, CA, USA) has been developed for 3D scanning applications. The low-cost, small footprint (99 × 20 × 30 mm), light weight (72 g) camera consumes minimal power (5W), which makes it ideal for certain groundbased HTP platforms. The objective of our research was to test the suitability of the RealSense D415 depth camera under field conditions for measuring heights and creating 3D point cloud images in wheat (Triticum aestivum) and soybean (Glycine max) research plots.

MATERIALS AND METHODS
The RealSense D415 stereo vison depth camera contains dual infrared radiation (IR) cameras, an RGB camera, a stereo depth module, a vison processor module, and an IR grid projector. The depth module uses the displacement between points from the two IR cameras to calculate distance from the camera to an object using the on-board image processor. The IR grid projector can be used to improve depth measurement in low light or low texture images. As data was collected under outdoor lighting conditions, the IR laser grid was left at the default setting (150 mW). The D415 has an active range of 0.3 to 10 m. Canopy height profiles were collected at a resolution of 0.307 MP (640 × 480 points). The horizontal and vertical field of view are 65 o and 40 o , respectively. To capture depth images from wheat and soybean plots that were longer than

Core Ideas
• The author planned to determine if the D415 camera is suitable for measuring canopy height in a field setting. • The manuscript examines the ease of incorporating the D415 into a mobile ground-based phenomics platform. • The proposed goal of incorporating the D415 depth camera was to increase the speed and accuracy of height data collection in field trials.
they were wide, the camera was positioned with the 640 point axis running along the length of the rows in the nadir position. Complete details on the RealSense D415 camera can be found in the data sheet (Intel Corporation, 2020). The D415 camera was mounted on the PlotCam, a lightweight, handheld, ground-based phenomics platform developed in-house. The D415 was mounted in line with RGB (Sony HX90V, Sony, Tokyo, Japan) and IR (ICI9640, Infrared Cameras Inc., Beaumont, TX, USA) cameras which were used to measure vegetative ground cover and canopy temperature, respectively. As the PlotCam is carried by the operator between plots it can be used in fields with various plot widths, plot spacing, and terrain. It can be adjusted to optimize coverage for plots of different dimensions by raising and lowering the sensor head which was set at 1.72 m for wheat plots and 1.85 m for soybean plots. The PlotCam is operated with custom software which incorporated the RealSense API. Canopy heights were captured and stored simultaneously with the information from the other cameras and sensors. The D415 depth profile for each plot was stored in a text file as one dimensional array of 307,200 distance points.
Canopy height was measured with the D415 camera in a wheat experiment comprised of 88 backcross derived lines and 24 checks (112 plots) in 6 row plots measuring 1.2 × 5 m. Canopy heights were measured from 21 d after seeding (DAS) to maturity on a weekly basis. Additionally, canopy heights were measured from 16 DAS to maturity in a 32 soybean cultivar trial grown in irrigated and non-irrigated treatments with 4 replications (256 plots) in 4 row plots measuring 1.6 × 3 m. The 2020 growing season in Ottawa was characterized by a wet spring followed by a drought in June and July. Coupled with a late seeding as a result of COVID-19 pandemic protocols, this resulted in shorter than normal spring wheat plots. The soybean trial was also affected by the drought in June and July, although irrigation was supplied to the plots as one of the treatments.
Manual canopy height was measured with a single point LiDAR (SPL, Garman, LiDAR Lite V3) mounted at a stationary height in a nadir position at the top of a pole. The SPL was aimed downwards at a 0.10 × 0.10 m paddle that was in line with the LiDAR and was moved up and down the pole manually, The operator adjusted the paddle to the top of the canopy by eye and the SPL signal was transmitted via Bluetooth to a custom Android application on a tablet and stored. The accuracy of the SPL was verified in the field by comparing heights measured with the SPL to heights measured on a 2 m ruler at heights between 0.01 and 1.10 m, for a total of 110 measurements. Regression analysis revealed that the SPL accurately represented actual measurements with an R 2 of 0.99 and a RMSE of 0.004 m (data not shown).
The on-board processor used the left IR image as the reference for stereo matching, creating a non-overlapping empty region of data on the left edge of the image, which was removed from the array (Carfagni et al., 2019). Processing steps for the depth profiles were done in Python (3.8.5) using the Numpy and Pandas libraries (Van Rossum & Drake, 2009). The set distance was the distance to the ground from the camera, established over a flat surface prior to data collection. The depth array was screened and extreme values over 2.5 m were changed to the set distance. A linear correction was done on both the X and Y axes of the depth array to correct for the angle of the depth camera on the PlotCam. After corrections, depth arrays were converted to height arrays by subtracting the distance to the ground. To improve the accuracy of the distance to the ground an adaptive method was used. A histogram was made with the distances, and the most frequent distance was used as the distance to the ground. When this height was more than 0.10 m greater than the set distance of the depth camera, the set distance was used as the distance to the ground.
Canopy height was calculated for each height array as the difference between the distance to the ground and the distance to the top of the canopy. The average of the top 1 and 5% of heights were selected to compare with the SPL heights based on the work of Wang et al. (2018)  Soybean and wheat trials were analysed separately. In the trials each plot was treated as an individual measurement disregarding cultivar or irrigation treatment. Canopy heights from the D415 camera from each measurement were compared to heights collected with the SPL using SAS (Proc Corr and Proc Reg, SAS Institute, Cary, NC, USA). Linear regression was used to examine the relationship between SPL heights and D415 heights and the coefficient of determination (R 2 ), the intercept, slope, and normalised root mean square error (NRMSE) were determined. NRMSE was calculated as the root mean square error divided by the mean height. Pearson correlation (r) was used to compare SPL and D415 heights at each sampling date and combined across the growing season.

RESULTS AND DISCUSSION
Over the growing season, we determined that the canopy heights from average of the top 5% of D415 soybean depth arrays were most similar to the SPL heights ( Figure 1a, Table 1) . The D415 heights were significantly (p < .001) correlated to SPL heights (r =.95) with an NRMSE of 0.17 m. The D415 canopy heights were similar to the SPL height as indicated by the intercept of −0.01 m and a slope of 1.04. Over the growing season the average of the top 1% of the D415 wheat canopy heights were most similar to the SPL heights ( Figure 1b, Table 1). The D415 heights were significantly correlated (p < .001) with the SPL heights (r = .77) with an NRMSE of 0.23 m. In wheat, D415 height measurements tended to underestimate taller plants as indicated by a slope less than one (0.84) and an intercept of 0. Similarly, Holman et al. (2016) found that wheat canopy height measurement accuracy was lower in incompletely closed canopies using 3D surface models from UAVs. The top 5% of D415 heights and SPL heights on individual sampling dates in the soybean plots were significantly correlated for all dates ( Table 1). The correlation coefficients were larger when the canopies were measured later in the growing season than at the beginning when the plants were small. Early in the growing season, the D415 overestimated SPL heights in short plots and underestimated SPL heights in tall plots as indicated by the positive intercept and slope less than one on these dates. As the growing season progressed, the intercept got smaller and the slope of the line closer to one, indicating that the D415 heights were closer to the SPL heights.
The top 1 % of D415 heights and SPL heights were significantly correlated at all dates for the wheat plots. As the plants grew in size the correlations increased in magnitude, the NRMSE decreased, the intercept approached 0 and the slope of the line became closer to 1. As with the soybean plots, the D415 tended to overestimate short plots heights and underestimate tall plot heights.
Using one measurement from the top of the canopy assumes that it is uniform, which is rarely the case as demonstrated by Figure 2. The manual measurement of the canopy with the SPL assumes that the top of the canopy was chosen correctly by the operator and that the area of the plot where height was measured represented the maximum canopy height. As noted by Holman et al. (2016) manual height measurements are often inefficient and introduce bias because of improper selection of the canopy height. In our case, this may have been particularly true during early crop development when the crop was low to the ground and it was more difficult to measure heights manually. If more manual measurements were done per plot it is likely that the of correlation would have been stronger and the NRMSE reduced. With the high F I G U R E 1 Comparison of the canopy height measured manually with a single point LiDAR (SPL) to heights measured by the RealSense D415 camera (Intel Corporation) over the growing season in the top 5% of heights for soybean (a, R 2 = .90, NRMSE = 0.17 m) and top 1% of heights for wheat (b, R 2 = .59, NRMSE = 0.23 m). Colors indicate days after seeding (DAS) T A B L E 1 Comparison of canopy heights measured manually with the single point LiDAR (SPL) to those measured by the RealSense D415 depth camera (Intel Corporation) at the top 1% (T1) and 5% (T5) of heights at different days after seeding (DAS) and combined over the growing season (GS). Intercept and slope from the regression, NRMSE (normalized root mean square error), and r (coefficient of correlation). For wheat plots n = 112 at each sample date and 672 for the GS, and for soybean plots n = 256 for all dates except 37 where n = 255 , and n = 2559 for the GS. All correlations were significant at p < .001 unless otherwise indicated. ns, not significant

F I G U R E 2
Three dimensional point clouds of field plots of soybean (a) and wheat (b) made from depth arrays. X and Y represent the length and width of the array with a resolution of 640 × 480 depth points degree of height variability within a plot the D415 was most likely a better representation of canopy height than the SPL. It is important to compare the performance of the D415 with similar depth camera systems. Jiang et al. (2016) used the Kinect v2 camera under a shade to measure single cotton (Gossypium hirsutum) plant heights and concluded that there was a significant correlation between data from the camera and manually measured heights with as high as 92% agreement between the two systems of measurement. From a static platform Hämmerle and Höfle (2016) found that the Kinect v2 camera heights correlated with a 3D laser scanner, but full grown maize (Zea mays) canopy heights were underestimated. Further study with a simulated field instrument showed that the Kinect v2 camera heights were correlated with a 3D laser scanner (r = .79 and percent RMSE = 7.0) (Hämmerle & Höfle, 2018). In a field study of sorghum plants Wang et al. (2018) found that, when shaded, the Kinect v2 camera maximum heights were significantly correlated (r = .92) to manually measured heights although the values were slightly overestimated.
Shortly after we began using the D415 camera (40 DAS, soybean and 46 DAS wheat) we noticed some abnormal distance measurements in a few of the plots. This was traced to movement of the D415 prior to completing the height array capture. This may have contributed to the lower correlation values between the D415 and SPL heights in wheat and soybean at the beginning of the growing season. To address this problem, the control of the D415 camera subroutine was moved to a different thread on the processor so data capture occurred simultaneously with the other instruments. Other challenges to obtaining accurate height measurements were seeding furrows, and large rocks and debris that created false height points. When furrows were taller than the plants being measured it was impossible to distinguish plant heights from soil heights using height data alone. Sensor fusion and segmentation with the RGB images captured by the D415 may be a way to increase the accuracy of early growing season height measurement by eliminating non-crop objects. Jimenez-Berni et al. (2018) found, when using scanning LiDAR in wheat, that heights were more unreliable in open canopies or when the plots had debris, rocks or the remainders of furrows in them.
Future studies will include partitioning the height array into smaller areas to measure height differences among different regions of the plot. This will be useful when examining stand establishment, disease susceptibility and lodging. Three dimensional surface plots such as Figure 2 will be used to estimate biomass production (Deery et al., 2020) and determine traits such as crop growth rate.

CONCLUSIONS
The Intel RealSense D415 shows promise as a tool for rapidly and accurately measuring canopy heights in field conditions. While suitable for ground-based platforms, the rolling shutter and maximum operating height of 10 m may limit its potential for integration into unmanned aerial vehicle platforms, or tractor-based platforms with more intrinsic vibrations. The RealSense D415 produced reliable and accurate heights in the field, whereas, the Kinect v2, has limited applicability under high light conditions (Jiang et al., 2016;Wang et al., 2018). The relatively low cost and open software development kit of the D415 makes it a tool that can be incorporated easily for high throughput phenotyping applications. The determination of canopy height was more accurately done when the canopy was full and further work needs to be done to improve the accuracy of height estimations at early stages of crop growth.

A C K N O W L E D G M E N T S
The authors wish to thank Thomas Hotte for experimental site preparation and Mathew Kenny and Alain Saumure for construction of the PlotCam and the single point LiDAR height pole.