Most of the current robotic systems are based on a traditional sense-plan-act cycle, where perception and action are viewed as individual processes. This approach, however, cannot deal with the challenges that the agro-food environment poses. As perception and action are tightly coupled through the interactions of the robot with the environment, the scientific challenge is to develop and study active perception methodologies to enable robots to resolve uncertainty by actively gathering new sensory input by changing perspective or manipulating objects. Building on progress in Deep Learning (DL), new DL architectures will be developed for 3D reconstruction, semantic object/scene segmentation and estimation of 3D geometrical object properties needed for gripping and manipulation.