Globalisation and individualisation – these are the mega-trends that have allowed the explosive growth in the global movement of goods to persist for so many years. In addition to transporting goods, their storage also plays a critical role. In production as well as in logistics, the automation technology represents a highly promising factor for reducing error rates and increasing effectiveness.
The Department for Conveyor Technology, Material Flow and Logistics at the Technical University of Munich, in collaboration with industry partners, has investigated the use of cameras on forklift trucks for (partial) automation in warehouse and loading logistics in a research project entitled “Forklift Truck Eye” (www.fml.mw.tum.de/Staplerauge). The area of intralogistics in particular contains efficiency potentials worth mentioning because the degree of automation is currently low in many areas of this discipline; numerous isolated solutions also exist, each of which is of little technical and economic use and requires a large amount of integration. The mounting of video cameras to material handling devices now enables the processes to be monitored and controlled in real time on the basis of image processing and reduces throughput times by executing previously sequential procedures in parallel.
For automated intralogistics use and more efficient transport processes, typical deployment scenarios were implemented and evaluated in the research project, such as fork occupied recognition, the identification of goods by means of NVE code, pallet-accurate localisation, and storage compartment-accurate forklift height recognition. The latter was achieved with the aid of a camera system and lighting control systems from the European image processing expert SMARTEK Vision.
Application scenarios:
Challenges:
Application solutions for various target groups
Industrial users, for example operators of forklift truck fleets or warehouses, need solutions for efficient and safe transport processes. In particular, affordable camera systems can be used in a wide number of areas in conjunction with the developed software framework. The growing range of sensor functions offered by system integrators and material handling device manufacturers takes into account the growing need of affordable assistance systems for industrial users.
Forklift truck and general material handling device manufacturers already rely on a large number of highly integrated sensor systems which offer the functional safety demanded in machinery directives, for example. In both manual and automated systems, the advances achieved in the research project provide a technical approach to integrating innovative status detection tasks in the factory and to accomplishing them on the vehicle in an image-based manner.
As a rule, the task of system integrators is to retrofit forklift trucks with sensor systems, taking into consideration the customer’s individual requirements and circumstances. The software framework developed during the project provides a complete development environment and basic functionality for implementing the other project-specific image processing solutions. The “Forklift Truck Eye” delivers the initial implementations in this area that can be flexibly adapted. Thus, system integrators are given a simple basis on which to develop new, innovative detection solutions that enable the logistics sector - which is heavily characterised by SMEs - to offer new products and develop new markets.
The software framework
On the basis of the convenient Software Development Kits (SDK) from SMARTEK Vision, the engineers at TUM implemented the software framework using established image processing libraries and special detection algorithms. The compatibility of the SMARTEK Vision cameras with the GigE Vision® standard and the Ethernet interface of the lighting controller guarantee quick and easy integration of these critical system components into the software framework. In the future, this will also enable the integration of additional sensor functions into the same sensor technology by means of a pipes-and-filters software architecture with minimal integration effort on the hardware side. It will therefore also simplify the development of sensor functions for forklift trucks for people who are less familiar with using computer vision methods.
This will be achieved through the pre-implementation of basic camera functions (e.g. image recording, camera configuration, etc.) or frequently used image processing and evaluation methods as well as special forklift truck algorithms as directly usable modules. With this so-called Black Box principle, the user only needs to understand which input is transformed by the module into which output. By integrating the freely available OpenCV library, the concept for using the framework allows the modules to be combined in a way that is easy to understand. The library provides many computer vision algorithms and a complete development environment (including compiler, debugger, code editor, etc.) created for further simplification and a decrease in the familiarisation effort required allowing program development to be commenced without costs or installation effort.
Essential for industrial applications is the provision of the software framework developed by the Technical University of Munich as an open source licence. To enable practical tracking, the test forklift at the research facility is available to interested parties for assessment purposes and as a demonstrator.
Outlook
Based on the rapid evolution of camera, computer, and bus systems, it is to be assumed that automation in the area of intralogistics will continue to advance and give manufacturers, system integrators, and industrial users the opportunity to further streamline their processes. The increased performance of the camera systems - with their higher resolutions, higher frame rates, more flexible lighting controllers, and greater light sensitivity - can also achieve more exact outcome measurements and controllers based on them (e.g. through high-speed cameras which already achieve much better identification of goods by means of NVE detection).
Advances in data transmission must also be mentioned. The research project made use of camera systems with Ethernet bus; faster interfaces with USB 3.0 and 10GigE are already available and enable the use of higher resolution images at a higher frame rate in real time through higher image transfer rates, e.g. for barcode recognition. A higher refresh rate is also of benefit in the area of localisation.
These developments generally favour the broader use of camera-based image processing systems as a sensor technology for achieving detection tasks that support status identification at the forklift truck in intralogistical transport processes. As such, camera sensor technology offers a challenging and highly promising area of action for new products and markets with the “Forklift Truck Eye” project providing relevant assistance through the software framework and contributing proof of feasibility for the minimisation of risks in product development.
The Department for Conveyor Technology, Material Flow and Logistics at the Technical University of Munich, in collaboration with industry partners, has investigated the use of cameras on forklift trucks for (partial) automation in warehouse and loading logistics in a research project entitled “Forklift Truck Eye” (www.fml.mw.tum.de/Staplerauge). The area of intralogistics in particular contains efficiency potentials worth mentioning because the degree of automation is currently low in many areas of this discipline; numerous isolated solutions also exist, each of which is of little technical and economic use and requires a large amount of integration. The mounting of video cameras to material handling devices now enables the processes to be monitored and controlled in real time on the basis of image processing and reduces throughput times by executing previously sequential procedures in parallel.
For automated intralogistics use and more efficient transport processes, typical deployment scenarios were implemented and evaluated in the research project, such as fork occupied recognition, the identification of goods by means of NVE code, pallet-accurate localisation, and storage compartment-accurate forklift height recognition. The latter was achieved with the aid of a camera system and lighting control systems from the European image processing expert SMARTEK Vision.
Application scenarios:
- Fork occupation
Of importance here is that the camera is able to precisely distinguish the forks of the truck from the image background. If the forks can be completely detected, the fork is in a “free” state. As the forks are steel grey, they do not show up particularly well against dark backgrounds. Consequently, the cameras must be able to record images with as much contrast as possible, whatever the lighting conditions.
- Lift height
Alternative formulation proposal:
As an example, in order to detect the shelf from which the forklift truck has taken a load, the lift height is determined with sufficient precision for this purpose. In subsequent stages of development, it should be possible for the forklift truck driver to set a lift height which will then be autonomously approached by the truck using a camera-based control loop.
- Position in space
As well as determining the height, the system under development must also be able to continually track the position of the forklift truck in a warehouse using a camera. This involves either using two-dimensional markets to ascertain the position, or using sophisticated algorithms to detect the orientation and motion of the forklift truck.
- Load identification
To enable automatic tracking of the material flow, the load must be identified using one and two-dimensional codes. The challenge here is that the camera must be able to record the codes at sufficient resolution and contrast over a very wide viewing angle.
Challenges:
- Deployment environment with changing lighting conditions (dark warehouses vs. light outdoor areas and continuous movement)
- Cost-efficient minimisation of the number of cameras with resulting lens requirements for synchronous solution of as many image processing tasks in parallel as possible, with a wide range of parameters (e.g. size of object, distance)
- Use in harsh environments with engine vibrations and shocks during operation
Application solutions for various target groups
Industrial users, for example operators of forklift truck fleets or warehouses, need solutions for efficient and safe transport processes. In particular, affordable camera systems can be used in a wide number of areas in conjunction with the developed software framework. The growing range of sensor functions offered by system integrators and material handling device manufacturers takes into account the growing need of affordable assistance systems for industrial users.
Forklift truck and general material handling device manufacturers already rely on a large number of highly integrated sensor systems which offer the functional safety demanded in machinery directives, for example. In both manual and automated systems, the advances achieved in the research project provide a technical approach to integrating innovative status detection tasks in the factory and to accomplishing them on the vehicle in an image-based manner.
As a rule, the task of system integrators is to retrofit forklift trucks with sensor systems, taking into consideration the customer’s individual requirements and circumstances. The software framework developed during the project provides a complete development environment and basic functionality for implementing the other project-specific image processing solutions. The “Forklift Truck Eye” delivers the initial implementations in this area that can be flexibly adapted. Thus, system integrators are given a simple basis on which to develop new, innovative detection solutions that enable the logistics sector - which is heavily characterised by SMEs - to offer new products and develop new markets.
The software framework
On the basis of the convenient Software Development Kits (SDK) from SMARTEK Vision, the engineers at TUM implemented the software framework using established image processing libraries and special detection algorithms. The compatibility of the SMARTEK Vision cameras with the GigE Vision® standard and the Ethernet interface of the lighting controller guarantee quick and easy integration of these critical system components into the software framework. In the future, this will also enable the integration of additional sensor functions into the same sensor technology by means of a pipes-and-filters software architecture with minimal integration effort on the hardware side. It will therefore also simplify the development of sensor functions for forklift trucks for people who are less familiar with using computer vision methods.
This will be achieved through the pre-implementation of basic camera functions (e.g. image recording, camera configuration, etc.) or frequently used image processing and evaluation methods as well as special forklift truck algorithms as directly usable modules. With this so-called Black Box principle, the user only needs to understand which input is transformed by the module into which output. By integrating the freely available OpenCV library, the concept for using the framework allows the modules to be combined in a way that is easy to understand. The library provides many computer vision algorithms and a complete development environment (including compiler, debugger, code editor, etc.) created for further simplification and a decrease in the familiarisation effort required allowing program development to be commenced without costs or installation effort.
Essential for industrial applications is the provision of the software framework developed by the Technical University of Munich as an open source licence. To enable practical tracking, the test forklift at the research facility is available to interested parties for assessment purposes and as a demonstrator.
Outlook
Based on the rapid evolution of camera, computer, and bus systems, it is to be assumed that automation in the area of intralogistics will continue to advance and give manufacturers, system integrators, and industrial users the opportunity to further streamline their processes. The increased performance of the camera systems - with their higher resolutions, higher frame rates, more flexible lighting controllers, and greater light sensitivity - can also achieve more exact outcome measurements and controllers based on them (e.g. through high-speed cameras which already achieve much better identification of goods by means of NVE detection).
Advances in data transmission must also be mentioned. The research project made use of camera systems with Ethernet bus; faster interfaces with USB 3.0 and 10GigE are already available and enable the use of higher resolution images at a higher frame rate in real time through higher image transfer rates, e.g. for barcode recognition. A higher refresh rate is also of benefit in the area of localisation.
These developments generally favour the broader use of camera-based image processing systems as a sensor technology for achieving detection tasks that support status identification at the forklift truck in intralogistical transport processes. As such, camera sensor technology offers a challenging and highly promising area of action for new products and markets with the “Forklift Truck Eye” project providing relevant assistance through the software framework and contributing proof of feasibility for the minimisation of risks in product development.
