Forests cover 30 percent of the Earth’s landmass, but that number is on the decline. Despite forests’ crucial role in conserving wildlife and processing carbon dioxide, many are threatened by deforestation and wildfires.
Complicating these threats is the lack of quantitative information that foresters and environmental researchers need for making important decisions to preserve forests.
Steven Chen, co-founder and CEO of Treeswift and doctoral student in Computer and Information Science (CIS) at Penn Engineering, wants to change that.
Chen founded Treeswift as a spin-off company from Penn Engineering’s GRASP Lab. The idea behind it is simple: use robotic tools to automate forestry and reduce risk for human workers. Treeswift uses swarms of autonomous, flying robots equipped with LiDAR sensors to monitor, inventory, and map timberland. The drones collect images of the land and render them into 3D maps that can be analyzed for precise, quantifiable measurements of a given forest’s biomass.
Trained on NVIDIA GPUs, the deep learning algorithms detect trees from point clouds. Treeswift has collected and labeled all its own training data to ensure high quality and to maintain control over the properties being labeled — such as whether the algorithms should classify a tree and its branches as two separate elements or just one.
Some processing is done at the edge, helping autonomously fly the drone through forests. But the data collected for mapping is processed offline on NVIDIA hardware, including TITAN GPUs and RTX GPUs on desktop systems – plus the NVIDIA DGX Station and DGX-1 server for heavier computing workloads. Its algorithms are developed using the TensorFlow deep learning framework. While the drone platform currently captures images at 1-megapixel resolution, Treeswift is looking at 4K cameras for the deployed product.
Of the variety of applications this data has, Treeswift is focused on three main targets: calculating inventory for the timber industry, mapping forests for preservation, and measuring forest biomass and fuel to prevent the spread of wildfires. The collected data can be used by researchers in a variety of industries to assess the health of forests and build predictive models that can aid in climate change action initiatives.
“Treeswift is a balance of priorities,” Chen said. “We are trying to build a general system that solves a lot of problems. Our main customer base right now is industrial forestry, but we are looking into opportunities that would enable us to work in wildfire forest management as well.”
Chen began thinking about the concepts behind Treeswift when he started his doctoral program at Penn in 2016. He joined the GRASP Lab and worked under the direction of Vijay Kumar, Nemirovsky Family Dean of Penn Engineering, to develop autonomous ground and aerial robots with an emphasis on robot swarms and bio-inspired algorithm designs for collective behaviours. Chen also collaborated with other researchers to build flying robots that could capture precise field data for farmers. These robots buzz through fields and process images of crops via computer vision algorithms, effectively counting the fruit on trees and providing precise numbers to growers.
Over time, Chen began to realize the potential broader impact of his research.
“I realized that I liked everything I was researching but there was something about entrepreneurship that spoke to me. I wanted my research to be more than a paper sitting on a shelf. I wanted something that could fit into a bigger community, that would go beyond CIS researchers, and really make something happen in the world.”
It was this urge to use his Penn Engineering experience to effect real-world change that drove Chen to create Treeswift. Chen is joined by co-founders Michael Shomin, Treeswift’s CTO and a Robotics doctoral graduate of Carnegie Mellon University, Vaibhav Arcot, Software Lead with a Master’s in Robotics from Penn Engineering, and Elizabeth Hunter, COO with a doctorate in Mechanical Engineering and Applied Mechanics, also from Penn Engineering.
Treeswift still works closely with researchers at Penn, including current doctoral students Xu Liu and Chao Qu and research staff member Avraham Cohen, as well as with former visiting student Guilherme Vicentim Nardari, now a doctoral student at the Center for Robotics at the University of São Paulo in Brazil. This team of entrepreneurs, researchers, and engineers all met at Penn, largely in the Precision Agriculture and Forestry group within Vijay Kumar’s lab.
Treeswift is a major step towards automating the forestry industry and bringing the tools of robotics and AI to solve environmental issues. While forests are crucial to the health of the planet, forestry as an industry is ready for a technological revolution. The current practices for gathering forest data are largely manual. Foresters trek into the woods, plot out samples of land, and measure trees by hand using a tape measure. The numbers from these samples are used to make reasonable estimates about forest size, biomass, and more. Data science has revolutionized everything from healthcare to online shopping, and the team at Treeswift believes it is time data science does the same for forestry.
“We want to know what’s going on,” said Nardari. “In Brazil, deforestation is a major issue. Images from satellites can’t tell the full story of what’s going on in a given area, and sometimes trees are cut down selectively, meaning only some are cut down while others remain. You can’t tell this is happening from satellite images, but with our drone images, you can.”
“2020 will begin a decade of climate action,” Chen said. “Big companies are looking to reduce their carbon emissions and they’re looking to forests to see how these resources can be used to assist them in this effort. We need to know what’s going on in the forest. Data will help us do that.”
Not only will this data provide much needed insight into the health of forests, but the current lack of data has devastating effects.
Take wildfires as an example. When foresters assess a given area’s risk for fire, an important factor is fuel, i.e. organic matter that could potentially burn. Knowing the fuel load — the available fuel per unit area — is crucial to predicting the effects of a given wildfire and deciding if a prescribed burn should be performed. Since these burns would occur under ideal conditions, they can prevent the major damage that could otherwise occur during severe heat or lightning events, since those have less potential for human control. The challenge is measuring fuel load with precision. Given the current methods, most numbers foresters rely on come from qualitative estimates.
“To measure the fuel,” Chen said, “what foresters have now is a digital photo series that the U.S. Forest Service provides. To create this, foresters go out and measure the fuel in a given area and take a photo of that area. Other foresters then visually compare what their forest looks like to these photos and, from there, determine approximately what their fuel is.
“There just aren’t adequate tools to gather and provide more accurate data to the people who need to make important decisions about the future of our forests and our climate,” he said.
In addition to manually mapping forests, foresters also rely on satellite images to survey the land, but Chen said satellites are limited in scope.
“You can only see the canopy, the tops of the trees,” he said. “You need to be able to see below that, to the surface fuel and growth as well. That’s why I believe robotics can fill the gap in data collection for the forest service.
“With these robots, we can go out and collect and process data into maps and metrics. Then, we can use AI to figure out what’s actually going on in the timberlands. We can make predictive models that would tell you if you took X action, you would likely have Y result.”
“What Google can predict about our behaviours using gathered data and AI is uncanny,” he continues, “but you can’t do that same process with forests because we don’t have the data to build those models. If climate change is the problem of our generation, we need to be using the same tools that everyone else is using to solve it. If you look at the big advances of the past decade, it’s all built on data.”
Treeswift co-founder and CEO Steven Chen collecting data
In December 2019, Chen received a National Science Foundation (NSF) Small Business Innovation Research grant to conduct research and development on his flying robot systems. Chen said that this funding has been essential for attracting more scientists and engineers to join the company. So far, Treeswift has begun partnering with various forestry companies to test their robots and learn about the current issues facing forest management. They’ve also collaborated with the New Jersey Forest Service and U.S. Forest Service, and have conducted field tests in the Wharton State Forest.
“Treeswift is a way for me to take action,” Chen said. “Rather than just developing an algorithm and writing a paper on it, I’m using that work in the community to bridge the gap between lab research and the real world.”
Treeswift’s development from a research idea to a start-up company comes at an interesting time. In the wake of the COVID-19 crisis, many are realizing the importance of protecting the environment and taking major steps towards climate action. At the same time, travel restrictions due to the pandemic caused Chen to shift the company’s workflow in March, which has led to more remote collaboration with his team and an emphasis on developing robust simulations in which to test their algorithms.
“It’s sort of like building a video game for our robots,” Chen said. “We are working hard on building our simulations so we can test out Treeswift technologies in a virtual world before testing in a live forest. This helps us with the cost of development, of course, and has also been helpful in terms of international collaborations, especially with Gui working with us from Brazil.”
Treeswift’s autonomous flying robots aim to decrease forest survey times and provide more accurate insights into forest health and management. Furthermore, they reduce the physical harm and discomfort to timber cruisers, the people who actually walk through forests to measure trees.
“It may not seem like a big deal, but physically going into a forest without a pathway can be dangerous. There are snakes and ticks, shrubs taller than your head, and inhospitable temperatures. The reason we want autonomous robots is so the person operating it can stay outside the forest and work from a truck on the road. This way, the operator is more comfortable and can collect data faster.”
Treeswift’s mission falls in line with the new NSF Engineering Research Center (ERC) for the Internet of Things for Precision Agriculture (IoT4AG), which will be headquartered at Penn and led by Cherie Kagan, Stephen J. Angello Professor in Electrical and Systems Engineering and in Materials Science Engineering. This ERC will aim to address food, energy and water security through advanced agriculture technologies, such as networked sensors, AI and robotics. Chen says that he hopes that Treeswift will be able to collaborate with this center in the years to come.
Chen dreams of a world where he can deploy a fully autonomous, multi-robot system that can be sent into the forests to survey land while being managed remotely by a human operator. As of right now, Treeswift’s team is building semi-autonomous robots that are operated by humans and feature built-in crash avoidance algorithms, before creating fully autonomous systems. Chen believes in this technology and the many possibilities it opens up.
For now, Treeswift is focusing on forests, though Chen believes that the technology they’re developing is widely applicable to other environments, such as farms, oceans and even outer space. While Chen looks towards the future of using robots for climate action as a big and positive step in the right direction, he acknowledges the apprehensions many people have about integrating complex robotic systems into the world.
“When I talked to foresters, I learned that the number of young people going into the forestry industry is decreasing. So now we have a situation where companies are looking more and more to forest resources to combat climate change but you don’t have a supply of people who are growing to meet that need. I want to help make each forester do what they do with greater efficiency. These robots will not replace human jobs. Instead, they’re providing new tools to the people who have the insight and the passion to manage our forests.”
“We’re not replacing the person,” Chen concludes, “we’re replacing the tape measure.”
Source: Penn Engineering Report