Tag Archives: uav

News – Flyt

The uses of computer vision technologies in controlling UAV flight have been mentioned in some blogs already and will be discussed in more detail in a future blog. It can provide autonomous means of flight which are difficult to replicate manually allowing the use of a UAV in agriculture, inspections, surveys, delivery or emergency response.

The means to implement computer vision technologies on UAVs has generally meant proprietary technology or a complicated open source route with the setup of hardware and the installation of a number of pieces of software before experimentation can even begin.

But the work of  Indian Navstick Labs has developed a number of products to solve these problems.

FlyTOS

FlyTOS operating system is an application development framework built upon Linux and ROS (Robot Operating System), meaning an integration with ROS modules/libraries and sensors. It also supports the APM and PX4 (Pixhawk) open source autopilot systems.

The systems allows the development of obstacle avoidance, autonomous landing with AR tags and object recognition, tracking and following. It’s object tracking can use simple OpenCV based algorithms to detect objects using color and shape and use a Kalman Filter for tracking. It can also incorporate OpenTLD for selecting objects in a display and then following them (this was originally published in MATLAB by Zdenek Kalal).

FlytConsole and FlytVision are inbuilt on-board web apps that aid on in the creation of applications.

It comes with  a web-based control station called FlyConsole and a 3D simulator called FlytSim.

The software can be downloaded for free and installed on an ODROID XU4 companion computer.

ODROID XU4

ODROID XU4

FlytPOD

The FlytPOD – Advanced Flight Computer is a companion computer system running the FlytOS system which is currently being funded as part of an Indiegogo project.

As well as coming with a suspended IMU for vibration damping and an external magnetometer it also supports RTK (Real Time Kinematics) GPS.

It’s USB3.0,  USB2.0, HDMI and user configurable I/Os connectors support a number of systems out-of-the-box including a Gimbal, PX4Flow (Optical Flow sensor), LiDAR (distance sensor) and USB Cameras. While the hardware interfaces on the FlytPOD support a number of specialized sensors including multi-spectral cameras, stereo cameras and LiDAR.

It is designed to be able to process photographs in the companion computer and stream them to ground.

The system comes in two models:

  1. FlytPOD Kit – uSD storage. It costs $499 ($399 in Indiegogo).
  2. FlytPOD PRO Kit –  This kit has the same features as the basic one but also offers sensor redundancy, with triple 3-axis Accelerometers and 3-axis Gyroscopes as well as double external Magnetometers, Barometers and External GPS. It also comes with the faster eMMC storage. It cost $799 ($699 on Indiegogo).

The Indiegogo funding ends on the 3rd October.

Advertisements

Using existing mapping data to control UAV mapping flights – Part 1 – Preliminary Ideas and Experimentation

An intrinsic problem with photogrammetry is its requirement to keep the camera facing the subject matter. A much higher quality and more accurate 3D model is produced using the method than taking photographs at an oblique angle. This is especially true of buildings with with flat facades, (this has already been discussed in another blog).

Work has been done using computer vision to automate the control of the camera position so that it follows targets selected by the pilot. Although this has potential for some recording methods such as site tours, as discussed in another blog, it doesn’t aid in the recording of complex topography or architecture. Although there is potential for the recording of architectural elements using computer vision  technologies (this will be discussed in a later blog).

Other work is being done in using a low detail 3D model of a building to aid in the control of a UAV flying around it, but these are more aimed at collision avoidance than quality recording.

While in the future i plan to look at the potential pre-scanning a building with an aerial LiDAR scanner mounted on a drone before recording with UAV.

Potential solution

The camera gimbal of a UAV can be controlled both remotely and from the autopilot of the UAV which could be used to always keep the camera facing the subject matter, but without pertinent information this would have to be done manually. With wireless camera technology it is possible to remotely view what the camera is recording and so control the movement of the gimbal when required, but this would require a second person to control the camera while the UAV is being flown and would be difficult to implement effectively and costly in a commercial environment.

But it would seem to be possible to use existing 3D data of an area to control the flight of a UAV; both controlling the altitude and the angle that the camera gimbal is pointing. I have already discussed the use of DroneKit Python to create a UAV mapping flight, thish can also be used to control the angle of the camera gimbal.

Existing Data

There are a number of existing sources of data that can be used to aid in creating a mapping flight.

Within the UK LiDAR data is freely available at different spatial resolutions, much of the country is available down to 1m while other areas are available down to 0.25 m.

This resource through processing in GIS (Geographic Information System) software provides all of the information required to create a flight path over the area under study and to control the angle of the camera gimbal so that it will record it to a higher quality than before.

A digital elevation model (DEM) created using photogrammetry from existing overlapping aerial photographs can also be employed once it is georeferenced to its correct location. This resource may provide a higher spatial resolution than the LiDAR data and so a better resource for the creation of the flight path, but the landscape and structures may have changed since the photographs were taken causing problems (this can of course be a problem with the LiDAR data as well).

Co-ordinate system problems

One complication with using LiDAR data to control the UAV is the fact that it is in a different co-ordinate system than the GPS of the UAV (OSGB and WGS84). This can be solved be translating one set of data to the co-ordinate system of the other. As the number of points for the mission path will be a lot less than that for the LiDAR data it would make sense to convert the GPS data to OSGB, but this also requires that it be converted back after the flight path has been created added a certain amount of inaccuracy into the data as a conversion is never 100% accurate.

required Data

Three different pieces of data need to be derived from the LiDAR data which are required for the UAV mapping flight:

  • Altitude.
  • Slope.
  • Aspect.

The Altitude is contained within each point of the LiDAR data and is used when displaying the data in GIS software.

The Slope of the topography/buildings is measure in increments up to 90 degrees, with o degrees being flat and 90 degrees being a vertical face.

The Aspect is which way any slope is pointing in is measured in increments from 1-360 degrees. (degrees).

 

Slope_angles

Slope angles

Although it would be possible to create software that extracts the data from the LiDAR file while creating a flight path this is not currently an option. The flight path is currently created in a piece of software such as the Open Source ‘Mission Planner’ system. In this an area is chosen together with other variables and an optimal flight path is created. This flight path file can then be saved, it contains the X and Y co-ordinates of each point of the mission.

UAV Control

At its simplest the flight path can be created with the altitude and slope derived from the LiDAR being used to control both the UAV altitude and camera gimbal angle. This would work well for sloping topography but would be more complicated for areas with sharp breaks in slope (such as buildings).

Altitude Control

The altitude will need to be carefully controlled to make sure that the quality of the imaginary is consistent across the whole area under study. At its simplest this is easy to do using the altitude data within the LiDAR data, together with obstacle avoidance sensors to aid with safety.

The problem arises when needing to record something near or completely vertical. Rather than requiring a set altitude the UAV needs to maintain a set distance horizontally. This may be possible by creating a buffer in the data around steeply sloping areas.

Drone_flight_path

Problem with vertical offset

Camera Gimbal Control

Most low cost UAV systems come with a 2-axis gimbals, this means that the camera is stabilized so that it always stays in a horizontal plane but also that its rotation can be controlled downwards.

Gimbal_angle

The angle of the gimbal begins at 0 degrees for a forward pointing position to 90 degrees for a downwards facing position. This is how is its controlled within DroneKit.

As seen earlier the slope is calculated between 0 and 90 degrees for a slope.

There are two intrinsic problems with this method:

  1. The slope only goes between 0 and 90 degrees so there is no aspect data within it. If the drone camera is to be controlled to record the building as it flies over if needs to know which way the building is pointing as the 45 degrees on the left is not the same as the 45 degrees on the right. This could be solved by combining the information from the slope and aspect to give more detailed resulting data.
  2. Most standard gimbals are designed to only point forwards and downwards. This means that the UAV has to turn around to record the back side of the building or it needs to fly the path in reverse. The other solution is to use a UAV with a camera that can point in 360 degrees.

GIS Processing

A certain amount of processing is required within GIS software to get the required data from the LiDAR data and combine it with the required mapping flight path. For this ArcGIS has been used both due to its availability at university and my own familiarity with it.

Lidar

Considering the LiDAR data is for a specific square it makes sense to use raster data rather than the points and lines of vector data as it retains the accuracy of the data. The LiDAR data can be simply loaded into the GIS software as a raster.

Within GIS software the Aspect and Slope can be calculated and a raster created showing the results.

This can be done using the Spatial Analyst or 3D Analyst Toolboxes to provide Slope and Aspect rasters.

The data for these can be incorporated into Attribute Tables which can be exported into text files. It is possible to combine all of the data into one attribute table containing the Altitude, Slope and Aspect.

Although it is possible to export this whole raster file including all of the data it is not currently possible to automatically derive the data in software using the flight path, so a flight path has to be loaded into the GIS software.

Flight Path

The flight path file we created in the Mission Planner software needs to be loaded into a feature class in the GIS software. This can be done by loading the point data for the flight path into the software, this is the beginning and end point for each of the back and forth paths across the area needed to be recorded.

We next need to recreate the flight path using ‘point to line’.

Even though we have recreated the lines, deriving enough data from them is not possible as the flight path is designed to fly back and forth at a set altitude. For this reason we need to create a number of extra points. This can be done using ‘Construct points’ where points can be created at set intervals along a line. This can be linked to the level of data that is being used, so for this LiDAR data the points can be set at intervals of 1m.

Once this his has been done ‘Extract multi-values to points’ can be run on the 3 sets of data to create a table containing all of the required data for each point on the flight path we have created.

UAV Mission Creation

Now that we have all of the required input data for the UAV mapping flight we need to create the mission within Dronekit Python.

For the first level of experimentation we can just load the point data file into python then create a number of points for the UAV to fly to which give the X and Y co-ordinates and the required altitude. At the same time we can also program in the angle for the camera gimbal. It may be best to have the UAV hover at the positions for a second or two so that we know how the recording is going.

As already mentioned if we are only using a 2-axis gimbal we are going to have to have the UAV turn through 180 degrees to record the back sides of buildings and slopes sloping away from the camera. We should be able to do this by altering the UAV Yaw. We will need to have the Python read the aspect angle and change how it creates the flight path depending on the aspect of the slope/building.

Future Directions

ArcGIS allows the use of Python to run tools in its toolbox so it seems possible to create a python script which would automatically create a file with all of the information required from input files of LiDAR data and a flight path.

As QGIS also allows the use of python it would also seem possible to create the required file within this open-source solution.

 

UAV Building Facade Recording – Part 1 – Preliminary Ideas and Experimentation

The recording of buildings is an important area in Cultural Heritage, whether for conditional surveys or to record something that is about to be destroyed.

Traditional methods rely upon survey equipment such as Total Stations to take a number of points on the façade, but this results in only points and lines with no great surface detail.

Other more detailed survey techniques such as laser scanning and photogrammetry have also been employed. But laser scanning is expensive and both the techniques are generally ground based missing detail of the façade that is not visible from this position. Scaffolding or a cherry picker can be used to record the whole of the building but again this can add to the cost the recording.

Photogrammetry is a low cost method of producing high quality results but relies upon having the camera parallel to the building to produce the best results, as capturing photographs from an angle brings inaccuracies into the recording as well as there being more detail at the bottom of the 3D model created than at the top.

The UAV would seem to provide an ideal platform to carry a camera parallel to the building, recording photographs with the required photogrammetry overlap. And with its autopilot it would seem possible to automate the recording process allowing the mapping of the façade in the same way that the UAV  can map the ground.

There are of course a number of problems that need to be overcome.

Building Façade Recording

Manual

Building façade recording can be done manually with a UAV, but the larger and more complicated the building façade the mode difficult it is to do this accurately. As the pilot needs to control the UAV accurately in 3 dimensions as well as controlling its speed.

Although the results for an experimental UAV mission are acceptable the difficulty of maintaining a manual position can be seen in the image below.

Automatic

In order to automate the process you need to determine what parameters are required to record a building façade using photogrammetry.

These can be seen below.

Excel Calculations

Building facade recording parameters

First experimentation was done by taking the co-ordinates of the two ends of an example the wall from Google Earth (The south facing wall of the lay brothers’ quarters at Waverley Abbey in Surrey was used). These co-ordinates can then be used to determine the bearing that the wall lies upon and its width. Using the camera parameters and level of detail the required distance from the wall for the flight can be calculated using trigonometry. Trigonometry is once again used to calculate the offset positions for the left and right extent of the flight.

 

The image overlap can be used to determine the number of photographs required in the horizontal and vertical, and hence the change of altitude that is required for each flight pass of the building.

Calculate altitude changes

Calculate altitude

Although it is planned to have the ability for the UAV to hover and take photographs, it is much easier to have it take photographs as it flies across the building façade. This requires the additional calculations and control of optimum flight speed and shutter speed to take photographs which are not adversely effected by motion blur.

Shutter speed formula

Shutter speed formula

Shutter speed calculions

Shutter speed calculations

These preliminary calculations were done in Microsoft Excel.

DroneKit

The drone manufacturer 3DR provides a series of software development kits (SDKs) for writing applications to control your UAV using one of the open-source autopilot systems they support.

DroneKit Python uses the Python programming language and provides a number of examples to help with programming the flight of a UAV; these include flying from co-ordinate to co-ordinate up to complete missions. Together with this there is an API (application program interface) reference which provides all of the Python commands that can be used to control the UAV.

Python

Python is a fairly easy to learn programming language and as DroneKit already requires it to be installed and setup it makes sense to use the same language to calcuate the required paramaters for the flight path. This was done with the aid of a number of online resources. A graphical user interface (GUI) was created using the Tkinter Python package and was used to enter the data. The python code did the calculations then a file is exported which combines these calculations with the DroneKit code for controlling the autopilot. The final file when run will control the UAV flight.

Python GUI

Python GUI

Virtual Drone

Experimentation doesn’t need to be done with a live UAV, it can actually be done with a virtual one using a number of pieces of open-source software. These include Mission  Planner, ArduCopterMAVProxy and SITL (Software in the loop)

Virtual Drone

Virtual Drone

Next Steps

Experimentation with a UAV using the hardware and software is the next step to test whether a GPS can be used in close proximity to a structure.

Limitations of standard UAV GPS accuracy to within the range of meters also complicates the use of this method of controlling the flight. This either needs to be solved with the use of a more accurate GPS (although the proximity to the building may block the signal), sensors that measure distances or the use of computer vision technologies to control the UAV position. The UAV afterall currently only need to fly between two set points then at set altitudes above the ground.

DJI Phantom 4

The DJI Phantom 4 is the new model in the popular phantom range of quadcopters, it has a number of improvements over previous models.

DJI Phantom 4

DJI Phantom 4

Comparison of DJI Phantom 4 and 3

Model Phantom 4 Aircraft Phantom 3 Professional or Advanced Aircraft
Battery 4S 15.2V 5350mAh Intelligent Flight Battery 4S 15.2V 4480mAh Intelligent Flight Battery
Max Flight Time 28 mins About 23 mins
Vision Positioning System 10m 3m
Obstacle Sensing System Optical Sensor – 0.7 – 15m N/A
Intelligent Flight Modes Follow Me
Point of Interest
Waypoints
Course Lock
Home Lock
ActiveTrack
TapFly
Follow Me
Point of Interest
Waypoints
Course Lock
Home Lock

Using the TapFly mode you can tap on a position of the screen in the app to fly to that location.

One of its main improvements is the introduction of obstacle avoidance technology (Sense and Avoid) using cameras mounted above the legs on the front of the Phantom 4.
DJI Phantom 4 - Obstacle Avoidance

DJI Phantom 4 – Obstacle Avoidance

The system, and the subsequent technologies, rely on a companion computer within the drone attached to the various sensors which uses computer vision algorithms to detect obstacles in the drones path. Once it has detected an obstacle it will either hover or fly around it.

DJI Phantom 4 - Companion Computer

DJI Phantom 4 – Companion Computer

It also comes with an improved Vision Positioning System, for position hold without the aid of GPS, which raises the positioning altitude from 3m to up to 10m.

DJI Phanton 4 - Vision Positioning

DJI Phanton 4 – Vision Positioning

A final important new technology is ActiveTrack where a subject can be selected in the app, and once again using computer vision technologies, the Phantom 4 will follow the subject even when it is turning.

DJI Phantom 4 - Active Track

DJI Phantom 4 – Active Track

The DJI Phantom 4 is available for £1,229.00 and will be on general release from the 23rd of March. As such it will be the first commercially available drone with obstacle avoidance technology.

Benefits

The Phantom 4 provides a number of cutting edge technologies on a low cost platform. The benefit of ActiveTrack has already been discussed in a previous blog – UAVs for site tour recording – Part 1 – Theory while the potential of the sense and avoid and vision positioning system technologies will be discussed in a future blog on building recording.

Drawbacks

The main drawback of the system is the fact that the camera is not of the same quality as the Zenmsue X5 which is available for the DJ Inspire 1 Pro/Raw. But even this camera is not of the same specifications as many standard DSLR or mirrorless cameras, only providing 16MP.

UAVs for site tour recording – Part 1 – Theory

Thanks to UAVs there is a growing potential for the provision of high quality visualizations of sites from the air for public consumption; whether as part of the requirement of many archaeology companies as charities, as part of planning policies to interact with the public, or the growing importance of crowdfunding archaeological excavations (DigVentures) which require interaction with their backers. UAVs can provide a means of providing this sort of imagery as part of an overall recording strategy. This includes the recording of site tours which can provide details of a sites which can easily be disseminated to the public.

At its simplest the UAV can provide an aerial element to the video of the site tour by flying past or through elements of the site or flying past or hovering in front of the site tour guide.

The DJI Inspire 1 is one such aerial video platform which can be purchased with two remote controllers; one for controlling the UAV, while the other is used to control the camera gimbal. This allows a pilot to fly the UAV on a set path while someone experienced in film making has complete control of the camera.

DJI Inspire 1

DJI Inspire 1

Although the UAV can provide an excellent platform for aerial video recording as part of site tours, recently developed technologies can make this much more automated and provide a means for one person to both:

  1. The site tour guide.
  2. The UAV pilot recording the site tour.

There are two ways in which this can be done.

1. GPS ‘Follow Me’ technology

'Follow Me' technology (DroneDog using Pixhawk)

‘Follow Me’ technology (DroneDog using Pixhawk)

This functionality is available on many UAVs, including some of the DJI series and those using the open source PX4 and Pixhawk autopilot technologies.

With the PX4/Pixhawk systems the mode can be controlled from a number of base station software solutions including Tower, which can run on Android mobile devices such as smartphones.

The systems uses the GPS of the mobile device as a target for the UAV.

A number of cinematic controls for the UAV are available in the app:

  • Leash – UAV follows actor.
  • Lead – UAV leads actor pointing back at them.
  • Left/Right – UAV keeps pace with actor to the side.
  • Circle – UAV circles actor at specified radius.
'Follow Me' controls (3DR Tower)

‘Follow Me’ controls (3DR Tower)

The following parameters can also be set:

  • Altitude.
  • Radius.
3DR Tower - Altitude and Radius

3DR Tower – Altitude and Radius

The system also controls the camera gimbal, pointing the camera towards the GPS enabled device.

Together these controls can provide various aerial video elements useful for integration in a site tour video which can be controlled directly from the mobile device in the hand of the site tour guide.

2.Computer vision technologies

Computer Vision technologies are an important developing area in robotics and are beginning to be fitted to UAVs.

Some of these technologies use image recognition algorithms to match the subject matter between consecutive video frames allowing the UAV to follow a person or object even when it is rotating and so changing the way it appears.

They come in three forms:

A. Software

Currently in beta testing the Vertical Studio app (available on iOS and Android) uses existing camera hardware on the DJI Phantom 3 or Inspire to provide the imagery for the image recognition algorithms running in the app. A target is chosen in the app which then controls the flight of the UAV.

Vertical Studio App

Vertical Studio App

You can also draw walls in the app that designate no fly areas for the UAV.

Walls in the Vertical Studio App

Walls in the Vertical Studio App

B. Add-on technology

The second is an add-on technology that is fitted to an existing UAV, which connects to the autopilot and controls the flight of the UAV. In the case of the Percepto (funded on the Indiegogo crowdfunding website) the processing is done in a companion computer while the video is taken from an add-on camera, controls are then sent to the autopilot and gimbal to control the movement of them in relation to the subject matter.

Percepto Tracking

Percepto Tracking

Percepto Kit

Percepto Kit

C. Integrated technology

The third is an an integral part a newly built UAV, but is in effect a very similar technology to B.

This is the case with the soon to be released DJI Phantom 4, which is the first commercially available UAV with the technology integrated into it.


The app connects to a companion computer on the UAV which uses the imagery from the camera as a source for the computer vision algorithms. Once again the subject matter is selected in the app and the UAV will follow it.

Phantom 4 App

Phantom 4 App

 

Sources
https://3dr.com/kb/follow-instructions/

http://www.dji.com/product/phantom-4

http://www.dji.com/product/intelligent-flight-modes

http://vertical.ai/features/

http://www.percepto.co/

3D Printing and the UAV

3D printing provides a cheap method of creating objects from 3D computer model files. This, together with recent development in the field, have great potential for the future of the Unmanned Aerial Vehicle (UAV) industry.

3D Printing Parts
Many ready built UAVs can be purchased off the shelf configured to work with a number of different cameras building, but DIY systems can require parts that are not available from traditional sources. This is where the Maker community can come in, whether providing 3D models on such sites as Thingiverse to be 3D printed yourself or at a number of 3D printing shops; or providing ready 3D printed objects over the internet.

Among the objects that are useful to be 3D printed for UAVs are camera specific mounts and mounts for radio antennas. Complete 3D printed UAV frames are also possible.

Recent Developments
Until recently the materials that could be printed were limited, namely only thermoplastics or UV resins for the UAV body, recent developments have allowed the printing of everything from metal to human tissue and organs and even food; opening up whole new potential areas of use.

One example is the research by Dr. Jennifer A. Lewis, a founder of the Voxel8 company and Harvard University professor, which has led to 3D printers being able to print circuits such as the Voxel8 3D Printer developed by her company.

Conductive Ink Printing

Conductive Ink Printing

The Voxel8 3D Printer ships towards the end of the year.

Future developments planned by Voxel8 include the development of inks that are capable of printing resistors, sensors and even lithium ion battery cells.

In collaboration with Autodesk they have developed the Project Wire software which allows everything from design through to machine control of electrical circuits.

Autodesk Project Wire

Autodesk Project Wire

Potential
3D Printers have already been used for printing UAV components, but these recent developments open up the possibility of 3D printing almost complete UAVs in the near future. This would allow for UAVs specific to a task to be designed and printed on demand without the requirement of expensive manufacturing practices.

It would also link in with the idea of drones owning themselves, discussed in a previous blog, with the drones being able to print replacement or upgrade components straight from a 3D printer.

Drones Owning Themselves

On BBC Radio 4’s program FutureProofing of the 16th September the software developer Mike Hearn discussed the potential of cars and drones owning themselves. His ideas build upon collaborative open ventures such as Apache and Linux.

Cars
The cars could act as autonomous one machine businesses which would charge people for rides, then from the profits they could buy fuel, repair themselves and even buy upgrades. They would begin as a new car from a factory but would then become self sustaining/financing with even the ability to purchase a new upgrade car from the factory. Hearn suggests if the autonomous vehicles owned themselves they would provide cheaper fairs than those owned by major corporations.

This links in with the ambitious plans in the Finnish capital Helsinki to provide a ‘comprehensive, point-to-point “mobility on demand” system’ allowing people to purchase transportation options directly from a phone app linking in with availability of everything from driver-less cars, taxis, buses, bikes and ferries on the required route. This could potentially do away with the requirement of car ownership within the city by 2025 by beating it on cost and convenience.

More details of the car aspect of the idea can be found in an article at the BBC News Website.

Drones
He also discussed the potential for the development of delivery services for packages and parcels by drones; where the drones would sell their services to people or companies and use the money earned to maintain themselves. If demand in area was reduced they could move to an area with more demand.

Amazon are currently developing their Amazon Prime Air drone package delivery service.


Wile the first drone delivery took place in the USA in July.

Potential
With the growing importance of drones within archaeological recording this has great potential to make it easier and cheaper for companies to employ this kind of technology without the significant outlay that is required. Drones with a number of different recording packages could be setup in useful locations around the country. They could then be employed by a company or individual for a purpose and transported to the site by the individual who pays for their services. The drone would be paid and would use the finances to pay for charging, repairs and upgrades.

This could obviously go one step further with the introduction of driver-less vehicles; with the drone being based in a vehicle which deploys it to the required location, contains the software required for the processing of the data recorded and its control interface.

Limitations
The ideas of Mike Hearn are only a “thought experiment” and he is not involved in their development, although he is closely involved with the Bitcoin virtual currency which could be used as a method for the drone to pay for itself.