Hybrid Intelligence is the Future of Supply Chain Management
Supply chain and product fulfillment is one of the most exciting industries in the world right now when it comes to digital innovation. While that might not sound too sexy, the method of delivering a product from point A to point B has completely changed over the last five years, and will likely be completely different in the next five.
There are so many variables involved in automating supply chain processes: Delivery, manufacturing, packing, sorting, and more. There are obvious applications in industries like finance and entertainment where AI will have a major impact, but at this exact moment, AI is revolutionizing how products are sold and shipped.
But not all AI is created equally. Just ask Kindred, an AI robotics firm with offices in the San Fransisco and Toronto. The company manufactures robotic arms to aid in sorting and packing materials in a fulfillment setting—a fancy way of saying their robots pack boxes full of products to ship to customers. As of now, Kindred has publicly revealed one client: Gap, which was their pilot program. As retailers are careful not to share their advances with their competition, Kindred is tight-lipped as well. For Gap, Kindred has several robots packing clothing in two fulfillment locations.
Kindred’s robots are different than predecessors. The robots use a specific kind of AI called reinforcement learning, an often-underlooked segment of AI. Sure, that’s different, but not unheard of. What sets Kindred apart is that their machines also have a fleet of specially-trained pilots sitting in a Toronto office teaching the robots human-assisted best practices. Now that’s a bit different.
Repetition is key
James Bergstra is a co-founder of Kindred and the head of AI research at the company. He is responsible for overseeing the transition of research and innovation in machine and reinforcement learning to the company’s business model, publishing research results, and essentially everything involving how the robots work at Kindred.
Over the last three years, Kindred has become one of the leading companies in the world when it comes to researching and applying reinforcement learning with real-world outcomes. Robots learn from their mistakes, and when a human guides them, they learn even quicker.
“Everything the arm does can be made better or worse,” says Bergstra. “And we use AI to make sure it’s better.”
“We look to the way of how humans can operate our machines as a kind of gold standard.” – James Bergstra, Head of AI at Kindred
From a very basic perspective, Ai at Kindred operates like this goes like this: Write an algorithm that teaches robots to recognize and pick up things then place or sort them. When it messes up, let a human guide it to the right outcome. The machine picks up the human’s patterns and then teaches itself, eventually getting to a point of confidence where the humans can stop interacting and move onto a new robot. For example, one “human-inspired” change involved recognizing that a pilot moves much more gracefully and fluid than the original robots did. Bergstra introduced a closed feedback loop between choreographing movements of the arm and how the gripper worked, almost reminiscent to how a human can twist their arm and grip their hand in real time as they are reaching into the back of a cupboard to grab a can.
This is not about humans sitting in the cockpit of a giant robot and piloting it. Kindred’s pilots sit at a desk (see the image below), a cup of coffee within reach, and jump in to help the fulfillment robots if they get stuck. The kinds of things each robot may get stuck on include trying to grasp a particular article of clothing, or not knowing where to place that product.
“Reinforcement learning is a type of machine learning where AI systems can learn from their own experience,” says Bergstra. “Whenever we talk about the robot’s grasping action, if it reaches out and grabs something and misses, it’s like ‘Uh oh, that’s a negative example of something it should avoid in the future.’ But when it’s successful, that’s great, because that means we’re getting work done for our customers, and we’re getting positive reinforcement for how we interact with the world. We record all that data and use it in our learning algorithm to make it more likely in the future to choose actions that lead to good outcomes.”
The combination of technologies to achieve this kind of process is staggering and built completely in-house. The robot exists inside a warehouse and is roughly the size of a human arm, complete with a rubber gripping ‘hand’. It has a stereo camera so the robot pilots can see everything that is happening. The robot utilizes basic optimization and scanning techniques but also has a specialized proprietary technology called AutoGrasp, which can be seen in part below.
The star here is the robot AI itself, but it’s hard to ignore how Kindred has seemingly created an entirely new job with robot pilots. Trial and error is a large part of how Kindred gets to what Bergstra calls “General common sense human-like intelligence” within their robots. When Kindred first began testing their robots, the AI was not confident in what it was doing most of the time, meaning it often had to call out to the human pilots to intervene.
A pilot’s interaction with a robot is minor, and most of the time last just seconds. They will grasp something, or scan something, or contort the robot arm in a smoother way in order to place or pick something up.
“We started the company with this bold idea of building human-like intelligence within machines, as opposed to just optimizing things or doing ‘great AI.’ We include the idea of human-like because we do look to the way of how humans can operate our machines as a kind of gold standard,” says Bergstra. “And we do see that. Our human pilots are excellent in terms of their efficiency and how they deal with variability and the common sense they bring. Our human pilots can even provide customer service, as they can communicate with people on the floor.”
Here’s a quick breakdown of the robot pilot role: they don’t need engineering backgrounds, and Kindred recruits locally, taking in co-ops and interns. Kindred makes it a point to hire employees with different backgrounds, as robot pilot is a newly-created job and they do not know who may be best suited for the role. Pilots monitor an entire fleet, and when they jump in to help a robot, they do not know the facility or robot they are helping. If they cannot solve a problem, they call for someone on the warehouse floor to intervene.
“We are the first company in the world to bring reinforcement learning algorithms into a commercial product.” – Bergstra
But to even get to the point of using human pilots was a lot of work from Bergstra and his team. They used deep learning and neural networks to enable the robot to make predictions based on two available pieces of information: “What am I looking at now in the current state,” and “What might I do in this current state.” In Kindred’s current case, that means grasping a product to pack and ship. The robot uses deep learning to identify the item in front of the robot, how effective it would be to act on that product, and then what the reward would be.
“The robots will call out to pilots for help when they’re looking at a bin, and they’re guessing things to do, and they’re all not promising,” says Bergstra. “When the neural network has enough data that it’s actually making meaningful predictions, and we’ve done that optimization, and it’s telling us all the actions are promising, that tells us we’re confident that we shouldn’t use our pilots to engage.”
Measuring success
The confidence threshold is what Kindred’s value proposition rests on. They report 85 per cent—meaning Kindred is handling 85 per cent of the automation at a near-perfect level. A key part to growing the company was making sure they could hit this threshold quickly. The team did in-house testing during their early years from 2014-2016, found it to be viable, then found a pilot partner in Gap. Kindred began measuring their success in a fulfillment setting first in an offline way, then gradually ramped up.
Bergstra is candid about the early days.
“It’s a new approach to how to do robotics, and we didn’t know how long it would take.”
That early hesitation was well-founded, considering reinforcement learning is a niche form of AI that has not been widely adopted by the robotics community or as an engineering discipline. There’s not a lot of research to draw conclusions from, so understanding how it would fit into a product roadmap and live fulfillment environments was a challenge.
“If you have a lot of information to take into account, the process is slow,” Bergstra says, explaining why reinforcement learning has not yet been fully embraced.
“If there is a long sequence of actions you have to make between when a robot is smart and does something in a good way and there is a payoff for that action, that’s called a delayed reward. The longer that delay is, the harder it is to know which of those actions you did that contributed to the good reward.”
The work Bergstra has been doing with reinforcement learning is hugely impactful on how AI and robotics will be used in the future, even outside of e-commerce fulfillment centers.
“Machine learning and reinforcement learning is based on the idea that data on what works and doesn’t work could asymptotically approach an optimal solution in a way that hard code rules can’t,” says Bergstra. “They don’t change over time and they don’t get better. Engineers could make them better, but they don’t necessarily know how much headroom there is. In the same way, we see how deep learning has changed the way we do computer vision, we expect techniques like reinforcement learning to make a similar transformation to the way people design control systems of robots.”
Bringing Kindred to the world
Perhaps the most important thing for Kindred is to be able to share and replicate the advances they have learned. It is not viable as a business to create a custom-tailored knowledge for each client, so Kindred is prioritizing the ability to use their learned knowledge and systems and apply it to any environment, even outside of the clothing world. The robots can be trained within weeks to pick up boxes, cartons, bottles, bags, or almost anything else.
This is where Kindred stands out. They are one of the only companies in the world that can scale this type of reinforcement-learning backed AI and bring it to new customers with new problems to solve.
“If you were to stat rank all of the different companies that are trying to do robotics in terms of their ability to capture the data that drives these kinds of processes, as far as I know, we’re light years ahead of any company doing that,” says Bergstra. “We are the first company in the world to bring reinforcement learning algorithms into a commercial product.”
That proclamation is music Kindred CEO Jim Liefer’s ears. As the top exec, Liefer is ecstatic about the massive value this approach brings to potential customers.
“The beauty of having our human in the loop is not only are we training the algorithm, but we’re also allowing the customer to keep moving at their pace, so we’re not slowing them down,” he says. “One of the problems with a straightforward fully algorithmic solution without a human is that it takes a lot of time to get to production-grade speed to solve the customer’s problem.”
“The value we provide is in dealing with less-controlled situations where traditional automation options struggle due to variability and all of the real-world edge cases that arise,” says Bergstra. “That is a sweet spot for this kind of data-driven policy for controlling things.”
Kindred’s business model is based on typical fulfillment center standards: They charge on a cost-per-pick basis, meaning Kindred makes money depending on how many successful items are picked by their robots.
“If you think about what we’re solving for, as in the needs of manipulation and the needs between piece of automation A and piece of automation B, there’s a lot of places for us to deploy in.” – Jim Liefer, CEO of Kindred
Liefer knows the model well considering he has been in the supply chain world for almost two decades. He spent three years as a VP at UPS, then moved on to be the VP of operations at Walmart for six years. His excitement about how Kindred can change the face of the supply chain world is well-founded. When he came to Kindred, first as a COO and then as the CEO, he was tasked with putting all of Bergstra’s technical components together to solve a real-world problem.
“Because of my background, I break things down into simpler-to-understand components,” says Liefer. “It comes down to four things: vision, the actual grasping and manipulation ability, the ability to identify what the robot is holding onto, then the ability to put that thing somewhere. When we develop this broad range of technology, we had these four capabilities in hand. From there, those four components are highly applicable to many businesses in the world.”
Liefer points to the human pilots as the people who bridge the last gap for edge cases in fulfillment—a solution that is not offered anywhere else. This allows Kindred to set their sights high.
“There are adjacent areas, even within the four walls of a fulfillment center, for us to work,” he says. “If you think about what we’re solving for, as in the needs of manipulation and the needs between piece of automation A and piece of automation B, there’s a lot of places for us to deploy in.”
Between Bergstra and Liefer, they list examples in the automotive, manufacturing, and even healthcare industries. Anywhere a large set of small components need to be arranged in a specific way, and they arrive unstructured. Kindred can make sense of a pile of things lying amongst one-another. The deliberate focus will be on the product Kindred has already developed as opposed to focusing resources on creating another robot with another intended function.
Kindred currently operates a “fleet” of robots with several different partners. There growing opportunities as online shopping thrives and companies look to automate aspects of their supply chains. But when people think about extrapolating the possibilities for a human-like robotic arm, they often go right to medicine and surgery.
When quizzed about whether he thinks a Kindred robotic arm will be doing open-heart surgery anytime soon, Bergstra laughs it off.
“Maybe not,” he says. “But there are still opportunities for using learning methods in little spots that aren’t necessarily life or death situations on an operating table. Lots of things happen in a hospital. There is a lot of logistics in terms of moving things around and getting things to the right place.”
And that’s just what Kindred does. They get things to the right place. With a little help from their friends, that is.
